The requirement of Layer 2 connectivity has driven Ethernet-based fabrics and Layer 2 virtual private networks (VPN) to be adopted by internet service providers, cloud operators, and enterprises in their converged networks. A new set of requirements, not initially considered in the design of existing technologies and protocols, has led to the deployment of solutions with certain shortcomings. To address these issues, most notably scalability and active/active non-blocking forwarding, Ethernet VPN (EVPN) was proposed. It has since been extended as a network virtualization overlay (NVO) for data center and campus networks.
This article aims to introduce the core concepts of EVPN, highlight the benefits of using it over other solutions, explore various use cases, and discuss potential caveats. However, for a better understanding, it is essential first to provide insight into network virtualization and the distinction between underlay and overlay networks.
Network virtualization
Virtualization is the abstraction of resources, allowing for the creation of virtual instances of physical entities such as computing resources, storage devices, operating systems, and network components. The abstraction layer created by virtualization fosters better resource utilization, enabling greater scalability and flexibility. This has transformed the way organizations operate.
Virtualization has allowed organizations to consolidate systems onto fewer physical computing resources. This consolidation has not only made infrastructure more efficient and cost-effective but has also driven the need for network virtualization. As computing became more consolidated, the demand for a more flexible and scalable network infrastructure grew in parallel.
One of the first attempts at network virtualization, and the most basic form of VPN, is a virtual LAN (VLAN). VLANs partition networks by abstracting LAN segments. Each VLAN is its own broadcast domain, usually corresponding to a separate IP subnet. In legacy data center and service provider network designs, a VLAN logically represents a separate tenant or service.
Considering the limitations of scale (a maximum of 4094 VLANs), rigid provisioning, and inefficient resource utilization due to loop avoidance mechanisms like the Spanning Tree Protocol (STP) blocking redundant paths, the requirements for network consolidation outgrew this approach. With the advent of large scale internet service providers (ISPs), cloud operators providing services for a substantial number of clients, or even in the case of consolidated enterprise data centers, providing network connectivity became a bottleneck in the process of provisioning new tenants or services.
If you want to learn more about the differences between VLAN and VXLAN, we highly recommend our previous article.
Underlay and overlay networks
The concept of underlay and overlay networks is fundamental to understanding virtualized network architecture, protocols and data flows. It is based on the premise of decoupling the offered connectivity service from packet transportation which releases transit devices from the liability to maintain state which is only a duty of overlay tunnel endpoint devices (servers, leaf switches, PE devices).
Underlay network
An underlay network is the foundational infrastructure on which overlay networks function. It consists of network elements and logical configuration allowing for encapsulated data to be exchanged between network nodes. An underlay network is not aware of the services it is carrying, it simply provides transportation for the upper layer, the overlay network. Its architecture and configuration determines important characteristics, including total available bandwidth, path redundancy, loop detection and load balancing.
Depending on the overall protocol stack, including the overlay network, the underlay can be implemented as Ethernet, Multiprotocol Label Switching, segment routing or IP fabrics.
Overlay network
Overlay networks utilize the underlay network for packet transportation while simultaneously creating logical, virtual networks that segregate traffic from each other. This structure creates isolated connections or tunnels between different tenants and services, effectively separating the transportation of packets from the service layer. As a result, the provisioning of services is simplified, as it primarily concerns the overlay network and not the mission-critical underlay infrastructure. This means that transit devices are relieved from the burden of maintaining service state, further streamlining the process.
Services are differentiated from one another through encapsulation, most often utilizing protocols like SR, MPLS, or virtual extensible LAN (VXLAN). SR/MPLS encapsulation can be employed when an SR/MPLS-aware underlay is deployed, or it can be tunneled over Generic Routing Encapsulation (GRE) or User Datagram Protocol (UDP) encapsulation. Alternatively, a UDP-based VXLAN can be used in pure IP fabrics.
This separation of overlay and underlay provides a modular approach to network design, enabling easier adaptation to changing needs and requirements. By encapsulating services and leveraging either SR/MPLS or VXLANs, network operators can ensure robust, scalable, and efficient connectivity tailored to specific use cases and environments.
How underlay and overlay networks comes together
The following diagram presents a sample topology, illustrating the concept of an underlay network infrastructure with overlay network services deployed on top of it:
The decoupling of the services layer from the transportation layer brings several notable benefits, including:
- the underlay network, if implemented using an IP fabric or MPLS/SR approach, is easily scalable,
- the underlay configuration rarely changes,
- limited scope of configuration changes: the addition of a new service requires changes in the overlay configuration at the tunnel endpoints only,
- the transit network is free of service state.
In this model, transit devices in the underlay network forward packets based on the outer transportation headers. Meanwhile, tunnel endpoints make forwarding decisions based on tunnel encapsulation data, directing traffic to the appropriate VPN. This process effectively separates the concerns of transportation and service provision, as illustrated in the drawing below:
Read more about our network professional services and check how we can improve your network.
Understanding Ethernet VPN and its benefits
EVPN is a next-generation VPN solution designed to address limitations observed in traditional Layer 2 VPN technologies, especially virtual private LAN service (VPLS). EVPN integrates a series of features that enhance traffic handling, scalability, and resilience. Its architecture and capabilities offer significant improvements in network virtualization and provisioning of Layer 2/3 services.
In the below diagram, key EVPN concepts are outlined. Following that, these concepts will be detailed, providing understanding of EVPN mechanisms and their benefits.
The benefits of EVPN can be evaluated from various angles. Frequently, these assessments focus on how EVPN compares to other Layer 2 VPN solutions, particularly virtual private LAN service (VPLS). This comparison is appropriate considering EVPN was specifically designed to overcome the limitations of VPLS. With this context in mind, highlighted below are the key features of EVPN and the advantages it offers over VPLS, followed by the general advantages of using EVPN in virtualized network environments.
Active/Active multihoming
By configuring the Ethernet segment identifiers (ESI) in an all-active mode on provider edge (PE) devices, EVPN facilitates active/active multihoming for customer edge (CE) devices. In this setup, every PE device associated with a specific ESI has the capability to forward traffic. This ensures the CE devices benefit from a load-balanced and non-blocking connectivity.
Control plane learning
Distinguishing feature of EVPN is its use of MP-BGP in the control plane for MAC learning between PEs, rather than traditional data plane methods. This approach decreases flooding from unknown unicast traffic. Many hosts, when connected, send gratuitous ARP (GARP) requests. EVPN PEs, recognizing these, can then distribute MAC and IP addresses to other PEs through MP-BGP.
Furthermore, EVPN offers ARP flooding suppression via Proxy ARP. When an end-host sends an ARP request to a remote host, EVPN PE intercepts and responds if a match exists, reducing unnecessary ARP traffic.
EVPN's control plane learning doesn't utilize the ternary content-addressable memory (TCAM), allowing it to accommodate more MAC/IP mappings without having to consider constrained TCAM size.
Learning controlled by policies
Given that EVPN utilizes MP-BGP within its control plane, it facilitates the implementation of routing policies to regulate the routing information that a device imports into, and exports from, its routing and forwarding tables. This mechanism provides the flexibility to introduce specific routing information selectively and to suppress unnecessary or undesirable updates.
MAC mobility
EVPN supports host mobility across Ethernet segments. If a MAC address becomes reachable in multiple segments, EVPN tags the MAC/IP advertisement routes with the MAC mobility extended community, which includes information necessary to decide which of the competing MAC advertisements to use. This feature is particularly beneficial during seamless host migrations such as vMotion.
Multitenancy
With the use of route target (RT) and route distinguisher (RD) values in BGP as well as possible separation of bridging tables by utilizing separate MAC VRFs, EVPN efficiently handles multiple tenants or VRFs, allowing for isolation and segmentation in a shared network infrastructure.
Provides Layer 2 and Layer 3 services
EVPN delivers both Layer 2 and Layer 3 services. For Layer 2, EVPN facilitates connectivity across segments that are otherwise isolated by a Layer 3 network. This is achieved by implementation of Layer 2 VPN, which acts as a virtual overlay on top of the Layer 3 infrastructure. Additionally, EVPN offers a Layer 3 gateway capability, enabling routing between networks.
Distributed anycast gateway
EVPN provides distributed anycast gateway functionality which allows for end devices connected to services stretched across multiple PEs to utilize a local gateway to send traffic outside their subnet. This approach, often employed to implement a distributed default gateway, optimizes traffic flow by reducing the need to traverse the network to reach a distant gateway.
Flexibility of adding new services
Once the backbone network enabling EVPN is established, the introduction of new layer 2 or layer 3 EVPN-based services is easy and limited in scope. Changes in the infrastructure are required only on the PE/leaf devices circumventing any need to modify intermediate nodes. This approach not only expedites service rollout but also sidesteps potential alterations to the backbone's critical infrastructure.
Loop-free topology without STP
Traditional Ethernet deployments rely upon STP to create loop-free topologies. This comes at a cost of blocked redundant links, high network convergence time and complex configuration which is prone to human error. Misconfiguration can be catastrophic in its effects causing broadcast storms and often leading to network instability and congestion.
EVPN, thanks to usage of BGP and the control plane learning of MAC addresses, alleviates these issues by creating an all-active, non-blocking topology without having to employ STP.
Applicability of EVPN
Thanks to a number of benefits over Ethernet-based fabrics and other types of Layer 2 VPNs, EVPN has emerged as a solution to address certain challenges. It can be considered in several scenarios, including the following:
- Data center interconnect: EVPN was initially designed as a means to stretch Layer 2 and Layer 3 services between sites over the WAN.
- Layer 2 VPNs: building E-Line, E-LAN and E-Tree services: EVPN serves as a next-generation replacement for VPLS and Layer 2 pseudowires.
- Layer 3 VPNs: EVPN provides Layer 3 gateway capabilities and might be used to replace traditional Layer 3 VPNs.
- Control plane for IP fabrics: Relying on a loop-free and non-blocking IP fabric, EVPN can provide Layer 2 and Layer 3 connectivity in data center and campus networks across single or multiple sites.
NVO and EVPN tradeoffs
While network virtualization and EVPN brings a host of benefits to network infrastructures, it is important to acknowledge the trade-offs, most of which stem from the additional complexity inherent in implementation. Bear in mind that the tradeoffs mentioned below are in the context of a virtualized network environment using EVPN as a control plane in comparison to traditional, non-virtualized networks, as opposed to comparing EVPN to other overlay types.
Additional layer of abstraction
EVPN overlay acts as an additional abstraction layer in the network infrastructure. While beneficial for flexibility and scalability, the overlay and underlay design might present challenges for visibility and performance management.
Complex troubleshooting
The addition of an overlay network complicates the troubleshooting process. There might be difficulties in correlating issues between the underlay and overlay networks, especially when this process is conducted manually. On the other hand it might prove to be difficult to find a proper monitoring solution that will do this automatically.
Advanced configuration required
The configuration of EVPN is inherently complex. It necessitates the use of MP-BGP and a tunneling protocol, most often MPLS or VXLAN. Often, vendor advice is to deploy EVPN with EBGP as an underlay protocol, despite supporting simpler IGP protocols, which introduces another layer of complexity. This is especially true for enterprises where staff are not used to operating EBGP internally on a daily basis. Moreover, it necessitates the management of underlay ASNs.
Tunneling and throughput
Overlay networking requires the use of tunneling, which results in additional headers in IP packets, which reduces overall throughput. This can be partially alleviated by usage of jumbo frames to increase the data-to-header ratio.
Additional computing power
If EVPN is deployed on servers, careful planning is required for additional computing power to accommodate tunneling offloading. Alternatively, network interface cards (NICs) with hardware offloading can be utilized.
Network scaling complexity
During the design stage, scaling the network becomes more complex due to the need to manage metrics specific to EVPN and the tunneling protocol of choice. These may include the number of LSPs, VNIs, IRB interfaces, and VRF instances, among others. These complexities are often augmented by the variability of metrics based on deployment models, requiring a certain degree of trust in the vendor. Despite the challenges that scaling of virtualized networks brings, it's crucial to remember the advantages, such as the mitigation of BUM (broadcast, unknown unicast, multicast) traffic flooding, overcoming STP limitations, and eliminating the need for state in the transit network.
Other caveats
In addition to the tradeoffs of EVPN and network virtualization in general listed above there are certain caveats that must be taken into consideration during the design stage.
Feature discrepancies
Given different tunneling protocols compatible with EVPN, there can be variations in feature availability. This requires careful consideration when choosing between protocols like VXLAN and MPLS to ensure the necessary features are supported. It is also crucial to ensure proper encapsulation protocol support across all devices in the backbone network.
Layer 2 loops
Although EVPN provides a loop-free overlay, it remains vulnerable to Layer 2 loop issues arising from incorrect wiring of end systems to provider edge (PE) or leaf equipment. This issue can occur if different Ethernet segment identifiers (ESIs) are bridged together with an additional backdoor link. Some vendors implement solutions based on BFD which fixes Layer 2 loop issues on the EVPN domain border, however the current implementation and capabilities of off-the-shelf chips might render this solution limited. If BFD cannot be used then implementation of STP mechanisms is required to mitigate the problem.
Vendor interoperability
EVPN is an open standard, but vendors often implement a subset of its specifications, sometimes incorporating draft RFCs. Due to this, planning for an inter-vendor environment should be approached with caution and avoided if possible. Early testing of specific use cases is recommended if interoperability is a strict requirement. This caveat is specifically for devices participating in the EVPN and tunneling protocol domains, not for intermediate devices designated to forward tunneled traffic.
Conclusion
EVPN is a prominent networking solution that addresses the limitations of traditional Ethernet-based networks and Layer 2 VPNs. It provides both Layer 2 and Layer 3 services, reduces the impact of network loops, and offers other features like active/active multihoming, control plane MAC learning and distributed anycast gateway. This makes EVPN a reliable choice for modern network infrastructures.
However, network design with EVPN as NVO also introduces challenges. The introduction of an additional abstraction layer in the form of an overlay can complicate network operations, from troubleshooting to configuration. The necessity to adopt new protocols and manage them, coupled with potential hardware limitations, further accentuates the need for meticulous planning.
While EVPN offers many advantages, its successful implementation requires a clear understanding of its features, a careful approach to design, and awareness of potential issues. Important considerations include ensuring compatibility across vendors, capacity planning, and addressing potential Layer 2 loop concerns on the EVPN domain edge.
Glossary
AS Autonomous System
ARP Address Resolution Protocol
ASN Autonomous System Number
BFD Bidirectional Forwarding Detection
BGP Border Gateway Protocol
BUM Broadcast, Unknown unicast, Multicast
ECMP Equal-Cost Multi-Path
ESI Ethernet Segment Identifier
EVPN Ethernet Virtual Private Network
GRE Generic Routing Encapsulation
IP Internet Protocol
IRB Integrated Routing and Bridging
LSP Label-Switched Path
MAC Media Access Control
MP-BGP Multi-Protocol Border Gateway Protocol
MPLS Multi-Protocol Label Switching
MSP Managed Service Provider
PE Provider Edge
RD Route Distinguisher
RFC Request For Comments
RT Route Target
SR Segment Routing
STP Spanning Tree Protocol
TCAM Ternary Content-Addressable Memory
UDP User Datagram Protocol
VLAN Virtual Local Area Network
VPLS Virtual Private LAN Service
VXLAN Virtual eXtensible Local Area Network
VNI Virtual Network Identifier
VRF Virtual Routing and Forwarding
WAN Wide Area Network
