You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
BGP on VPC private gateways so tenants can peer their own routers with the VR #14687
As an operator I would like VPC owners to be able to run BGP between the VR and their own routers over a private gateway, so they can connect their on-prem networks with dynamic failover. As an example, we have two high-speed peering links into a cloud exchange (Megaport, in our case), and we carve out a VLAN on each for a customer's virtual cross connects. Each VLAN would attach to a private gateway on the customer's VPC, so the customer has two paths to their own routers.
With the current code, each gateway needs static routes, and static routes have no way to detect a failure. When a carrier link fails silently, the gateway NIC stays up and the VR keeps sending traffic into the dead link until someone changes the routes by hand. The customer may pay for a second link that only helps after a support ticket. Every new on-prem prefix also needs a static route change on the VPC. AWS Direct Connect, Azure ExpressRoute and Google Cloud Interconnect all run customer BGP on their gateways for this reason, so customers moving hybrid workloads expect it.
CloudStack's existing BGP (#9470) lets operators peer ROUTED VPCs with their own upstream routers. The idea is to extend it to private gateways so the VPC owner can manage it. A BGP flag on the gateway would take the peer AS, a password, the VPC tiers to announce, a max-prefix limit, timers and BFD, and a priority so links can be ranked or share load. The peer address comes from the gateway itself. The VR accepts the customer's routes behind a protected list, announces only the selected tiers, and keeps tenant peers separate from the operator peers and the upstream route-maps from #9964. It should work for NATTED VPCs as well as ROUTED ones, so I think it relates to #12811, which opens dynamic routing to NATTED offerings for IPv6.
threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.
What happened
The threat detection engine failed to produce results.
The request is for a routing option on VPC private gateways. Static routes would stay as they are, and a BGP option would take the peer AS, a password, declared prefixes, a max-prefix limit and timers or BFD, so tenants can peer their own routers with the VR and fail over between links. The reporter wants this to work for NATTED as well as ROUTED VPCs, and to keep it separate from operator peers.
This will be quite a large feature, so I went through the process of developing my own list of requirements based on my experience, reviewed how that compared to the hyperscale cloud providers' approach, and then independently had Claude Ultracode develop a spec and run its council over the compiled list with voting personas as:
Cloud operator / product owner.
Customer network engineer.
Security and multi-tenant isolation reviewer.
Apache CloudStack maintainer.
VR / systemvm implementation engineer.
Design diagrams, including three UI mock-ups, and an API spec are attached for reference as well.
I've developed a series of decisions to share for this feature. Included below are the main ones, and the full list with the rationale and alternatives for each is in the attached file. I'm happy to discuss/change any of them.
As you can tell there's some heavy Claude usage here to develop this spec - no one is going to write 1100 lines of spec by hand. It has streamlined the design approach for this and helped me to develop some very detailed collateral, along with pre-testing a significant amount of these decisions - ruling out lots of options and saving me a significant amount of time in the process.
I have hand reviewed and worked on these decisions in detail over many days and I'm comfortable that they represent fairly sound logic, however of course I would I'd appreciate a detailed review and feedback from maintainers prior to jumping into any code and testing.
The path to implement this will take some time. Some bugs I've already reproduced go first as small fixes, and then I'd deliver this through manageably sized PRs so they can be thoroughly reviewed and tested.
Delivery:
Small bug fixes first to 4.20/4.22 where they reproduce without BGP. These are listed below. The feature then goes to main for the release after 24.0, as four PRs linked from this issue:
frr-reload, the template package and FRR logs in router diagnostics. Operator peers benefit from this one on its own.
Core: API, schema, management server, VR renderer, firewall and BFD.
Status, the route table and the UI.
Usage, reviewed with the usage maintainers.
Each PR can merge on its own, and they merge in order. The offering flag that turns the feature on lands in the last PR, so a release that holds only part of the series exposes nothing to operators or tenants.
API: a BGP flag on the private gateway, named apart from the offering's routingmode, with the BGP settings stored against the gateway and the peer address defaulting to the gateway's own gateway field, with multihop as an option. Both admin VLAN gateways and gateways on a tenant's own L2 network can use BGP. Dedicated commands update and list them, so operators can control access through roles.
Enablement: a new VPC offering capability that is off by default. It is independent of routingmode=Dynamic and works on NATTED and ROUTED VPCs.
Scope: IPv4 and one VPC first. BGP runs between a VPC and on-prem routers only, and routes learned from one peer are never sent to another. IPv6 needs these first:
IPv6 addressing on private gateways. createPrivateGateway only accepts an IPv4 gateway, netmask and IP address.
IPv6 static routes. createStaticRoute only accepts IPv4 CIDRs.
IPv6 firewall rules on the gateway NIC so the VR accepts BGP.
Isolation: tenant and operator peers sit in separate peer groups in the existing router bgp. Every neighbour has mandatory prefix-lists and route-maps, and no routes pass between the two groups.
Inbound: the VR accepts the routes the customer announces, as AWS and Azure do. It always rejects link-local, its own connected subnets, the VPC's own ranges, the default route and very short prefixes. A max-prefix limit counts accepted routes and trips as soon as it is passed. Its default is 1,000, which a load test on a small VR handled with no session drops. A tripped session restarts on its own after an interval, 5 minutes by default. An optional import filter per gateway is a future capability.
Local AS: every VPC presents one operator-set AS to customer routers, as Azure and AWS do. The VPC owner can override it with a private ASN when it clashes with their network. No public ASN is needed. Dynamic ROUTED VPCs keep their allocated AS for operator peers.
Peer AS: any valid ASN except the VPC's own AS, reserved values and an operator deny list of ASNs and ranges.
Outbound: only the tiers the tenant selects for that peer. A follow-up release adds the remote-access VPN client range and the tiers of peered VPCs to the same list. VPCs connect to each other with static routes over private gateways. Site-to-site VPN stays blocked.
Failover: a priority per gateway from 1 to 4, where equal priorities share load, tenant timers within operator bounds (keepalive 3 s and hold 9 s at minimum), and BFD down to 300 ms x 3. ECMP needs the same peer ASN on both links.
Visibility: a Routing tab on the VPC shows the effective route table, and each gateway shows its accepted and advertised routes, why the session last went down, and what was negotiated.
Gateway ACL: the gateway's ACL filters all traffic on a BGP link. A gateway on default_deny brings its session up and installs routes but passes no traffic until the tenant attaches an ACL that allows it. The status shows this.
Operator controls: an admin can hold a session down, and the operator can disable tenant BGP for an account. Each hold stays until the party that set it clears it, and the status lists the holds that apply. Operator-wide limits are root-admin only.
Events: customers get session up and down events through webhooks. Operators get one alert when more than half of a zone's sessions drop in one check interval, since that points at the platform.
Hypervisors: KVM, VMware, XenServer and Hyper-V, within each hypervisor's existing NIC limit.
Usage: a usage record per BGP gateway, in the same way as VPN users.
Activity
github-actions commentedon Nov 12, 2026
Warning
threat detection engine error
The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.
What happened
The threat detection engine failed to produce results.
Review the workflow run logs for details.
🎯 Triage report
The request is for a routing option on VPC private gateways. Static routes would stay as they are, and a BGP option would take the peer AS, a password, declared prefixes, a max-prefix limit and timers or BFD, so tenants can peer their own routers with the VR and fail over between links. The reporter wants this to work for NATTED as well as ROUTED VPCs, and to keep it separate from operator peers.
📊 Assessment
🔗 Similar issues
routed.network.vpc.enabled, return misleading 'not found' error #13745 (related) — BGP APIs hidden behindrouted.network.vpc.enabled, returning a misleading "not found" error💡 Notes and suggestions
Add this agentic workflow to your repo
To install this agentic workflow, run
MitchDrage commentedon Nov 19, 2026
This will be quite a large feature, so I went through the process of developing my own list of requirements based on my experience, reviewed how that compared to the hyperscale cloud providers' approach, and then independently had Claude Ultracode develop a spec and run its council over the compiled list with voting personas as:
Design diagrams, including three UI mock-ups, and an API spec are attached for reference as well.
I've developed a series of decisions to share for this feature. Included below are the main ones, and the full list with the rationale and alternatives for each is in the attached file. I'm happy to discuss/change any of them.
As you can tell there's some heavy Claude usage here to develop this spec - no one is going to write 1100 lines of spec by hand. It has streamlined the design approach for this and helped me to develop some very detailed collateral, along with pre-testing a significant amount of these decisions - ruling out lots of options and saving me a significant amount of time in the process.
I have hand reviewed and worked on these decisions in detail over many days and I'm comfortable that they represent fairly sound logic, however of course I would I'd appreciate a detailed review and feedback from maintainers prior to jumping into any code and testing.
The path to implement this will take some time. Some bugs I've already reproduced go first as small fixes, and then I'd deliver this through manageably sized PRs so they can be thoroughly reviewed and tested.
Delivery:
Small bug fixes first to 4.20/4.22 where they reproduce without BGP. These are listed below. The feature then goes to main for the release after 24.0, as four PRs linked from this issue:
Each PR can merge on its own, and they merge in order. The offering flag that turns the feature on lands in the last PR, so a release that holds only part of the series exposes nothing to operators or tenants.
Redundant ROUTED VPCs also depend on #14446.
Decision summary:
default_denybrings its session up and installs routes but passes no traffic until the tenant attaches an ACL that allows it. The status shows this.Full list of Design Decisions:
bgp-private-gateway-decisions.md
API Spec:
bgp-private-gateway-api-spec.md
Design diagrams:
UI mockups:
DaanHoogland commentedon Nov 20, 2026
the high level requirement seems sane to me. @wido @weizhouapache @JoaoJandre , can you have a look as well?
wido commentedon Nov 21, 2026
It seems sane to me and would be a great feature. It will be a big one though which will be hard to test.