Skip to content

BGP on VPC private gateways so tenants can peer their own routers with the VR #14687

Open

Description

@MitchDrage

The required feature described as a wish

As an operator I would like VPC owners to be able to run BGP between the VR and their own routers over a private gateway, so they can connect their on-prem networks with dynamic failover. As an example, we have two high-speed peering links into a cloud exchange (Megaport, in our case), and we carve out a VLAN on each for a customer's virtual cross connects. Each VLAN would attach to a private gateway on the customer's VPC, so the customer has two paths to their own routers.

With the current code, each gateway needs static routes, and static routes have no way to detect a failure. When a carrier link fails silently, the gateway NIC stays up and the VR keeps sending traffic into the dead link until someone changes the routes by hand. The customer may pay for a second link that only helps after a support ticket. Every new on-prem prefix also needs a static route change on the VPC. AWS Direct Connect, Azure ExpressRoute and Google Cloud Interconnect all run customer BGP on their gateways for this reason, so customers moving hybrid workloads expect it.

CloudStack's existing BGP (#9470) lets operators peer ROUTED VPCs with their own upstream routers. The idea is to extend it to private gateways so the VPC owner can manage it. A BGP flag on the gateway would take the peer AS, a password, the VPC tiers to announce, a max-prefix limit, timers and BFD, and a priority so links can be ranked or share load. The peer address comes from the gateway itself. The VR accepts the customer's routes behind a protected list, announces only the selected tiers, and keeps tenant peers separate from the operator peers and the upstream route-maps from #9964. It should work for NATTED VPCs as well as ROUTED ones, so I think it relates to #12811, which opens dynamic routing to NATTED offerings for IPv6.

Activity

  1. github-actions commented on Nov 12, 2026

    @github-actions

    Warning

    threat detection engine error
    The threat detection engine encountered an error and could not complete analysis. This is a tooling failure, not a security finding.

    What happened

    The threat detection engine failed to produce results.

    Review the workflow run logs for details.

    🎯 Triage report

    The request is for a routing option on VPC private gateways. Static routes would stay as they are, and a BGP option would take the peer AS, a password, declared prefixes, a max-prefix limit and timers or BFD, so tenants can peer their own routers with the VR and fail over between links. The reporter wants this to work for NATTED as well as ROUTED VPCs, and to keep it separate from operator peers.

    📊 Assessment

    Dimension Value Reasoning
    Type type:new-feature New routing capability on private gateways
    Component component:vpc, component:virtual-router Gateway configuration and VR BGP setup
    Severity n/a Not a bug report
    Labels type:new-feature, component:vpc, component:virtual-router Based only on the issue text
    Coding agent Not suitable Requires design decisions on API shape, NATTED support and isolation from operator peers

    🔗 Similar issues

    💡 Notes and suggestions

    Triage by Copilot (automated)

    Generated by Daily Issue Triage · copilot · haiku55 · ~4 AIC · ⊞ 3.8K · ◷

    Add this agentic workflow to your repo

    To install this agentic workflow, run

    gh aw add githubnext/agentics/workflows/daily-issue-triage.md@139fa09ef17cde4135e696ff8e7af09bc5ddfa6e
    
  2. MitchDrage commented on Nov 19, 2026

    @MitchDrage
    ContributorAuthor

    This will be quite a large feature, so I went through the process of developing my own list of requirements based on my experience, reviewed how that compared to the hyperscale cloud providers' approach, and then independently had Claude Ultracode develop a spec and run its council over the compiled list with voting personas as:

    1. Cloud operator / product owner.
    2. Customer network engineer.
    3. Security and multi-tenant isolation reviewer.
    4. Apache CloudStack maintainer.
    5. VR / systemvm implementation engineer.

    Design diagrams, including three UI mock-ups, and an API spec are attached for reference as well.

    I've developed a series of decisions to share for this feature. Included below are the main ones, and the full list with the rationale and alternatives for each is in the attached file. I'm happy to discuss/change any of them.

    As you can tell there's some heavy Claude usage here to develop this spec - no one is going to write 1100 lines of spec by hand. It has streamlined the design approach for this and helped me to develop some very detailed collateral, along with pre-testing a significant amount of these decisions - ruling out lots of options and saving me a significant amount of time in the process.
    I have hand reviewed and worked on these decisions in detail over many days and I'm comfortable that they represent fairly sound logic, however of course I would I'd appreciate a detailed review and feedback from maintainers prior to jumping into any code and testing.

    The path to implement this will take some time. Some bugs I've already reproduced go first as small fixes, and then I'd deliver this through manageably sized PRs so they can be thoroughly reviewed and tested.

    Delivery:

    Small bug fixes first to 4.20/4.22 where they reproduce without BGP. These are listed below. The feature then goes to main for the release after 24.0, as four PRs linked from this issue:

    1. frr-reload, the template package and FRR logs in router diagnostics. Operator peers benefit from this one on its own.
    2. Core: API, schema, management server, VR renderer, firewall and BFD.
    3. Status, the route table and the UI.
    4. Usage, reviewed with the usage maintainers.

    Each PR can merge on its own, and they merge in order. The offering flag that turns the feature on lands in the last PR, so a release that holds only part of the series exposes nothing to operators or tenants.

    Redundant ROUTED VPCs also depend on #14446.

    Decision summary:

    • API: a BGP flag on the private gateway, named apart from the offering's routingmode, with the BGP settings stored against the gateway and the peer address defaulting to the gateway's own gateway field, with multihop as an option. Both admin VLAN gateways and gateways on a tenant's own L2 network can use BGP. Dedicated commands update and list them, so operators can control access through roles.
    • Enablement: a new VPC offering capability that is off by default. It is independent of routingmode=Dynamic and works on NATTED and ROUTED VPCs.
    • Scope: IPv4 and one VPC first. BGP runs between a VPC and on-prem routers only, and routes learned from one peer are never sent to another. IPv6 needs these first:
    • Isolation: tenant and operator peers sit in separate peer groups in the existing router bgp. Every neighbour has mandatory prefix-lists and route-maps, and no routes pass between the two groups.
    • Inbound: the VR accepts the routes the customer announces, as AWS and Azure do. It always rejects link-local, its own connected subnets, the VPC's own ranges, the default route and very short prefixes. A max-prefix limit counts accepted routes and trips as soon as it is passed. Its default is 1,000, which a load test on a small VR handled with no session drops. A tripped session restarts on its own after an interval, 5 minutes by default. An optional import filter per gateway is a future capability.
    • Local AS: every VPC presents one operator-set AS to customer routers, as Azure and AWS do. The VPC owner can override it with a private ASN when it clashes with their network. No public ASN is needed. Dynamic ROUTED VPCs keep their allocated AS for operator peers.
    • Peer AS: any valid ASN except the VPC's own AS, reserved values and an operator deny list of ASNs and ranges.
    • Outbound: only the tiers the tenant selects for that peer. A follow-up release adds the remote-access VPN client range and the tiers of peered VPCs to the same list. VPCs connect to each other with static routes over private gateways. Site-to-site VPN stays blocked.
    • Failover: a priority per gateway from 1 to 4, where equal priorities share load, tenant timers within operator bounds (keepalive 3 s and hold 9 s at minimum), and BFD down to 300 ms x 3. ECMP needs the same peer ASN on both links.
    • Redundant VRs: sessions follow the primary. A keepalived switchover lab test passed in NATTED mode and in ROUTED mode with Routed VR: accept VRRP and conntrackd sync on redundant routers #14446, which redundant ROUTED VPCs need.
    • Visibility: a Routing tab on the VPC shows the effective route table, and each gateway shows its accepted and advertised routes, why the session last went down, and what was negotiated.
    • Gateway ACL: the gateway's ACL filters all traffic on a BGP link. A gateway on default_deny brings its session up and installs routes but passes no traffic until the tenant attaches an ACL that allows it. The status shows this.
    • Operator controls: an admin can hold a session down, and the operator can disable tenant BGP for an account. Each hold stays until the party that set it clears it, and the status lists the holds that apply. Operator-wide limits are root-admin only.
    • Events: customers get session up and down events through webhooks. Operators get one alert when more than half of a zone's sessions drop in one check interval, since that points at the platform.
    • Hypervisors: KVM, VMware, XenServer and Hyper-V, within each hypervisor's existing NIC limit.
    • Usage: a usage record per BGP gateway, in the same way as VPN users.

    Full list of Design Decisions:

    bgp-private-gateway-decisions.md

    API Spec:

    bgp-private-gateway-api-spec.md

    Design diagrams:

    1. Physical/Logical topology - In this case the example is using VXLAN/EVPN.
    01-topology
    1. Supported gateway types and deployment
    02-gateway-types
    1. Peer group arrangement to separate operator upstream routing and private routing
    03-bgp-process
    1. Route filtering approach
    04-route-filtering
    1. Failover process/timers
    05-failover
    1. VR config flow
    06-config-flow
    1. VR Firewall
    07-vr-firewall
    1. NATted vs Routed VPCs
    08-natted-routed
    1. Redundant VR switchover
    09-redundant-vr
    1. API and Data model
    10-data-model

    UI mockups:

    1. VPC Routing tab
    11a-ui-vpc-routing
    1. Private Gateway BGP - Received Routes tab
    11b-ui-gateway-bgp
    1. Private Gateway BGP - Advertised Routes tab
    11c-ui-gateway-advertised
  3. DaanHoogland commented on Nov 20, 2026

    @DaanHoogland
    Contributor

    the high level requirement seems sane to me. @wido @weizhouapache @JoaoJandre , can you have a look as well?

  4. added this to the 25.0 milestone on Nov 20, 2026
  5. wido commented on Nov 21, 2026

    @wido
    Contributor

    It seems sane to me and would be a great feature. It will be a big one though which will be hard to test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions