Each Edge Bare Metal size server has four 25-Gigabit Ethernet interfaces, with two interfaces on each of two separate NIC cards. By default, all four interfaces are placed into a single link access group (LAG) or bond utilizing LACP for individual link monitoring.
Each server connects to two switches: 2x 25GE to switch A and 2x 25GE to switch B. These switches are in the same physical rack as the server. Each of these switches have multiple 100GE fiber connections to multiple other switches in the data center that create an ethernet switching fabric. This switching fabric can sustain full traffic capacity with any single switch or link failure.
Note: This switch redundancy means that a server can always expect up to 50 Gbps of throughput, even if one of its directly connected switches were to fail. While a server does have 100 Gbps of total bandwidth capacity, half of that is redundant.
Egress from the data center’s switch fabric to either the Internet (AS3356) or the MPLS IP VPN network (AS3549) is provided by two diverse provider edge (PE) routers—one pair for each network. These PE pairs connect to a corresponding pair of switches in the fabric. Traffic can be forwarded through both PEs and will switch over automatically upon failure of a PE, switch, or the link between them. This dual-routing path is important to remember when using traceroutes or other reporting. PE1 may be active today but PE2 becomes the active path tomorrow.