EVPN-VXLAN: A Guide to Scaling Data Center Networks Without Extending Layer 2

EVPN-VXLAN enables the construction of data center networks where the physical infrastructure operates end-to-end over IP, while customer segments, virtual networks, and policies are created on an overlay plane. This approach does not eliminate Layer 2 but avoids physically extending it across all switches, reduces reliance on Spanning Tree Protocol (STP), and provides a control plane capable of distributing MAC addresses, IP addresses, and prefixes via BGP.

The key points of EVPN-VXLAN in 30 seconds

• The underlay is a routed IP network, typically with a leaf-spine topology.
• VXLAN encapsulates Ethernet over UDP and assigns a VNI to each virtual network.
• BGP EVPN distributes endpoint locations, gateways, and prefixes among the VTEPs.
• The fabric can utilize all available links via ECMP, avoiding blocked paths as STP would.
• EVPN-VXLAN improves isolation and scalability but requires careful consideration of MTU, BGP design, broadcast, unknown unicast, multicast (BUM) traffic, and automation.

For years, many data centers expanded by adding VLANs, trunk links, and pairwise switch configurations with multi-chassis aggregation technologies. While still valid for small to medium environments, this model begins to show limitations when connecting hundreds of racks, hosting numerous clients, or maintaining workload mobility without expanding failure domains.

EVPN-VXLAN shifts the question. Instead of asking how to carry each VLAN everywhere, it considers where each service needs to exist and how to announce it over a stable IP network.

The problem of large Layer 2 domains

Traditional Ethernet networks learn device locations by observing MAC addresses in received frames. When the destination is unknown, traffic is flooded out relevant ports. Broadcast and some multicast messages are also propagated within the Layer 2 domain.

While this is manageable in small environments, issues arise when a VLAN spans multiple racks or buildings:

• Broadcast domains grow significantly.
• Misconfigurations can impact larger areas.
• Unknown unicast traffic increases.
• STP may block links to prevent loops, reducing available paths.
• Diagnosing issues involves tracking VLANs, trunks, and MAC tables across multiple devices.
• Changes affect increasingly large operational domains.

The core issue isn’t Ethernet itself but converting the entire data center into a large switched network. Layer 2 remains necessary for server access and specific services but shouldn’t define the transport network between switches.

EVPN-VXLAN enforces this separation. Physical connectivity between nodes is handled via IP routing. Ethernet/IP services for each client run above, encapsulated and isolated.

Underlay and overlay: two distinct networks with different roles

Understanding the architecture is clearer when viewed as two planes.

The underlay IP network

The underlay is the physical network connecting switches. In data centers, it often adopts a leaf-spine topology based on a Clos architecture:

• Leaf switches connect servers, storage, firewalls, or other equipment.
• Each leaf links to all spine switches.
• Spine switches carry traffic between leaves and do not need to know specific VLANs of clients.

This topology offers multiple equal-cost paths. Traffic can be balanced via ECMP, enabling simultaneous use of all links and providing backup routes in case of failure.

The underlay protocol can be eBGP, OSPF, or IS-IS, depending on the platform and design. Its key requirement is stable IP connectivity between VXLAN Tunnel Endpoints (VTEPs), which often use loopback addresses as tunnel endpoints.

The underlay must be operational before creating the overlay. If two VTEPs cannot communicate via IP, VXLAN won’t be able to carry services between them.

The service overlay

The overlay contains the virtual networks used by servers and clients. It can provide Layer 2 connectivity, inter-subnet routing, and segmentation with VRFs.

VTEPs acting as VXLAN encapsulators encrypt the traffic entering the overlay and decapsulate it at the destination. For spine switches, these are just regular IP packets—they don’t need to understand MACs, VLANs, or clients within.

This independence allows services to be modified in the overlay without redesigning the physical topology. It also enables reuse of a single IP fabric across multiple isolated clients.

What VXLAN brings

VXLAN, which stands for Virtual eXtensible Local Area Network, encapsulates Ethernet frames within UDP packets. The outer header contains source and destination IP addresses of the VTEPs, allowing the underlay to route it like any other IP packet.

Each segment uses a VXLAN Network Identifier (VNI) of 24 bits, providing approximately 16.7 million unique identifiers, compared to the 12-bit VLAN space in IEEE 802.1Q, which is practically limited to 4,094 usable values.

A simple mapping might be:

VXLAN is the data plane. It defines how to carry traffic but doesn’t inherently determine the positioning of endpoints. The original RFC 7348 supports mechanisms like flood and learn, where the network floods part of the traffic to discover destination locations.

BGP EVPN adds a control plane that reduces reliance on flooding and learning by distributing endpoint information explicitly.

How BGP EVPN functions

Ethernet VPN (EVPN) utilizes Multiprotocol BGP to exchange overlay service information. VTEPs announce connected endpoints, and the fabric learns how to reach them.

Common route types include:

• Type 2, MAC/IP Advertisement: announces MAC addresses with associated IPs when available.
• Type 3, Inclusive Multicast Ethernet Tag: indicates a VTEP’s membership in a domain and assists in distributing broadcast, unknown unicast, and multicast traffic.
• Type 5, IP Prefix: advertises IP prefixes of a VRF without associating them to host MAC addresses.

Routes carry attributes like Route Distinguisher (RD) and Route Target (RT). The RD ensures uniqueness for potentially identical routes within BGP, while the RT controls the import/export of routes into VRFs or EVPN instances.

In large fabrics, establishing a full BGP session between all leaf switches isn’t always practical. Spine switches can act as route reflectors, redistributing EVPN routes without participating directly as VTEPs in data traffic.

This separation enhances observability. Administrators can query BGP to see where MAC addresses are announced, which VTEPs originate them, and their associated VNIs—beyond what is learned via hop-by-hop MAC tables.

Example: communication between two racks

Suppose server A connects to leaf 1 and server B to leaf 4. Both belong to VNI 10100.

The simplified process is:

1. Leaf 1 learns MAC and IP of server A locally.
2. Leaf 1 advertises this via a Type 2 EVPN route.
3. Leaf 4 does the same for server B.
4. Both VTEPs now know the remote location of each endpoint.
5. When A sends a frame to B, leaf 1 associates it with VNI 10100.
6. It encapsulates the frame in VXLAN, adding the outer IP addresses of the VTEPs.
7. The underlay routes the packet via one of the available ECMP paths.
8. Leaf 4 decapsulates the VXLAN packet and delivers the frame to server B.

From the server perspective, both remain within the same logical segment. However, the links between racks carry only encapsulated IP packets, not VLANs over Ethernet trunks.

Therefore, EVPN-VXLAN effectively avoids physically extending Layer 2 across the entire fabric. It can still offer a logical Layer 2 extension where needed, while keeping the physical infrastructure scalable and manageable.

Less flooding, but not elimination

BGP EVPN reduces flooding but doesn’t eliminate it entirely.

BUM traffic—broadcast, unknown unicast, and multicast—still requires distribution methods. Solutions include multicast in the underlay or ingress replication, where the ingress VTEP sends duplicates to interested VTEPs.

EVPN can also help suppress ARP and Neighbor Discovery by leveraging Type 2 routes, allowing VTEPs to respond locally to certain queries without flooding the segment.

However, effectiveness depends on implementation, configuration, and the quality of learned information. Deploying EVPN-VXLAN isn’t just enabling a protocol; it involves careful planning to ensure flooding is minimized and controlled.

Distributed routing and anycast gateways

One of the most advantageous features is placing the subnet gateway directly on leaf switches.

With an anycast gateway, multiple leaves present the same IP address and virtual MAC as the default gateway for servers. Traffic from hosts within the subnet is routed to the local gateway, enabling subnets to communicate without traversing core routers.

The Integrated Routing and Bridging (IRB) model supports both asymmetric and symmetric configurations. In large multi-tenant fabrics, symmetric IRB is common:

• The ingress VTEP routes the packet into the relevant VRF.
• The overlay traffic uses a Layer 3 VNI for routing between subnets.
• The egress VTEP delivers the packet to the destination segment.

This approach avoids requiring every leaf to host all VLANs of a VRF, promoting more scalable and organized designs. Understanding the mappings between Layer 2 VNIs, Layer 3 VNIs, VRFs, and policies is essential for correct implementation.

Active-active multihoming without STP dependence

EVPN supports redundant connections of servers, firewalls, or switches to multiple leaves. Using an Ethernet Segment Identifier (ESI), VTEPs can signal that they share the same segment.

The all-active mode enables simultaneous link usage and coordination of functions such as:

• Designated Forwarder selection for BUM traffic.
• Duplicate suppression.
• Coordinated MAC learning.
• Quick route withdrawal upon link or leaf failure.

This offers a standardized alternative to proprietary MLAG designs. It doesn’t mean MLAG is obsolete or that all devices support EVPN features uniformly but helps reduce reliance on traditional STP for fabric operation.

STP can still be used at the network edge to prevent loops, but its role as the primary link activation mechanism within the spine-and-leaf fabric diminishes.

MTU considerations and other factors that can disrupt the fabric

VXLAN adds headers to the original packet, creating an overhead typically around 50 bytes in IPv4 transport, though this varies. If the underlay’s MTU is 1500 bytes and servers also send frames of this size, encapsulated packets may exceed the limit.

Since many data center switches don’t fragment VXLAN packets as needed, inconsistent MTU settings may cause hard-to-diagnose losses. Thus, underlays usually configure jumbo frames and are validated end-to-end before deploying VXLAN overlays.

Additional planning should address:

• Loopback addresses and point-to-point links.
• ASN values used in underlay and overlay.
• Route reflectors and control plane redundancy.
• Hardware limits on MAC addresses, routes, VNI, and VRF instances.
• Handling of BUM traffic.
• Failure recovery mechanisms.
• Route filtering and policies between tenants.
• Interoperability among different vendor platforms.
• Automation and configuration consistency.

Although RFCs specify procedures, platform capabilities and defaults vary. Interoperability testing with the specific versions intended for production is critical.

How troubleshooting changes

EVPN-VXLAN structures the network into layers, and troubleshooting follows this hierarchy.

1. Check the underlay

Before analyzing EVPN, verify:

• Routing protocol neighbors.
• Routes to VTEP loopbacks.
• ECMP paths and link availability.
• MTU and packet loss.
• Latency and physical errors.

2. Review the EVPN control plane

Next, check:

• MP-BGP sessions.
• Active EVPN address families.
• Routes of types 2, 3, and 5.
• Imported/exported Route Targets.
• Next hop and origin VTEP.
• Route withdrawal and mobility events.

3. Validate the overlay

Finally, verify:

• Mapping between VLANs, bridge domains, and VNIs.
• Layer 3 VNI and VRF configuration.
• Anycast gateway configuration.
• MAC and IP address tables.
• ARP or ND suppression mechanisms.
• VXLAN encapsulation and decapsulation.
• Participating VTEPs in each segment.

While the complexity remains, layering into defined planes helps isolate issues. Proper telemetry, procedures, and trained staff are essential for effective troubleshooting.

When to consider deploying EVPN-VXLAN

This architecture is particularly suitable when the data center needs to support:

• Numerous segments or isolated tenants.
• Horizontal expansion with additional racks.
• Active utilization of multiple paths.
• Distributed gateways.
• Active-active multihoming.
• Automation via repeatable models.
• Controlled workload mobility.
• Separation of physical infrastructure and services.

It’s not always the right choice. Small environments with few VLANs, stable requirements, and a couple of switches can operate adequately with traditional architectures. Introducing BGP EVPN, VTEPs, VRFs, and policies without real needs could increase operational complexity and costs.

Decisions should consider current scale, expected growth, the skills of the technical team, and hardware support capabilities.

EVPN-VXLAN should be adopted when the separation of underlay and overlay addresses specific scale, redundancy, isolation, or operational challenges—not just because it’s a modern technology.

Frequently Asked Questions

Does EVPN-VXLAN eliminate Layer 2 entirely?

No. Servers can still use Ethernet and be within the same logical segment. The difference is that VLANs no longer need to traverse the entire fabric physically; VXLAN transports them over an IP network.

Are VXLAN and EVPN the same technology?

No. VXLAN defines encapsulation and acts as the data plane. EVPN uses BGP as the control plane to distribute MAC, IP, prefix, and segment membership information.

Does EVPN-VXLAN eliminate STP?

It reduces STP’s role because the underlay uses IP routing and ECMP. However, STP may still be needed in access segments or with external equipment prone to creating loops.

What should be checked first if a VXLAN tunnel fails?

Check IP connectivity between VTEPs, routing to their loopbacks, and the underlay MTU. Next, review BGP EVPN sessions, VNI configurations, and advertised routes.