This guide provides step-by-step instructions to configure Spine Leaf Topology using Symmetric IRB EVPN in SONiC, covering VLAN setup, IP address configuration, VRF configuration, VxLAN creation, creating L3VNI, and BGP establishment for EVPN.
Spine-Leaf Architecture
The spine-leaf, or leaf-spine, architecture is a two-layer network topology consisting of spine and leaf switches. This design enhances data center networks by minimizing latency and hop count while boosting overall network efficiency.
Unlike traditional three-tier architectures, the spine-leaf topology employs a full-mesh network structure, making it particularly well-suited for contemporary data centers that predominantly handle east-west traffic. East-west traffic refers to the flow of data packets between servers within the data center, as opposed to north-south traffic, which typically involves communication between servers and external networks.
By IETF, In a Spine-Leaf topology, normally a leaf node connects to a number of spine nodes. Data traffic going from one leaf node to another leaf node needs to pass through one of the spine nodes. Also, the decision to choose one of the spine nodes is usually part of equal cost multi-path (ECMP) load sharing. The spine nodes can be considered as gateway devices to reach destinations on other leaf nodes. In this type of topology, the spine nodes have to know the topology and routing information of the entire network, but the leaf nodes only need to know how to reach the gateway devices to which are the spine nodes they are uplinked to.
Three-tier architectures typically rely on the Spanning Tree Protocol (STP).
In contrast, the spine-leaf architecture features only two layers—spine and leaf switches—forming a simplified two-layer topology. This design facilitates direct connections through the spine layer, ensuring single-hop connectivity between leaf switches and thereby reducing latency and bottlenecks.
The spine-leaf architecture is optimized for east-west traffic and does not utilize STP. While STP provides redundant paths between switches, it allows only one path to be active at a time, potentially leading to oversubscription. By eliminating the use of STP, the spine-leaf architecture further reduces latency and avoids bottlenecks.
Symmetric IRB EVPN
In symmetric IRB routing, both ingress and egress VTEPs perform IRB routing and bridging. The ingress VTEP routes packets to an egress VTEP MAC address in an intermediate virtual-network VNI. In this approach, all routed VXLAN traffic is directed through a dedicated transit VNI known as the L3VNI. This allows bidirectional traffic to traverse on the same VNI in both directions.
Using the L3 VNI associated with each tenant VRF, an ingress VTEP routes all traffic for the prefix to an egress VTEP on the L3 VNI. The egress VTEP routes from the L3 VNI to the destination virtual network or bridge domain. The L3 VNI does not have to be associated with an IP address; routing is set up in the data plane using the egress VTEP’s MAC address. This behavior is known as IP-VRF to IP-VRF interface-less routing. The ingress VTEP does not have to be configured with every destination virtual network; it must have the ARP and MAC addresses only to the egress VTEP, not to each host connected to the VTEP. For this reason, symmetric IRB routing allows the overlay network to scale larger than Asymmetric Routing.
Testbed & Image Version
- GNS3
- SONiC ( 202305 )
Topology
Steps to Configure Symmetric IRB EVPN
- Setup VLAN Environment
- Configure IP addresses
- Configure VRF Setting
- Create VxLAN
- Configure L3VNI on a Switch
- Establish BGP environment for EVPN
Note: The above steps are only for Leaf switches. For Spine switches there is no need to create VLANs, VRF and VxLAN.
Configurations on LEAF-1:
Step 1. Setup VLAN environment as per topology.
admin@sonic:~$ sudo config vlan add 10 admin@sonic:~$ sudo config vlan add 20 admin@sonic:~$ sudo config vlan add 100 admin@sonic:~$ sudo config vlan add 101 admin@sonic:~$ sudo config vlan member add -u 10 Ethernet8 admin@sonic:~$ sudo config vlan member add -u 20 Ethernet12
Step 2. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.11.1/24 admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.12.1/24 admin@sonic:~$ sudo config interface ip add Loopback10 1.1.1.1/32
Step 3. Configure VRF Setting.
admin@sonic:~$ sudo config vrf add Vrf-red admin@sonic:~$ sudo config vrf add Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan10 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan20 Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan100 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan101 Vrf-green admin@sonic:~$ sudo config interface ip add Vlan10 192.168.10.254/24 admin@sonic:~$ sudo config interface ip add Vlan20 192.168.17.254/24
Below is the status of the VLAN table.
Below is the status of the VRF table.
The figure below depicts that VxLAN tunnel is successfully created.
Step 5. Set the layer3 VNI on a switch.
admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-red 100 admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-green 101
Step 6. Establish a BGP environment for EVPN.
sonic# configure sonic(config)# no router bgp 65100 sonic(config)# router bgp 65000 sonic(config-router)# bgp bestpath as-path multipath-relax sonic(config-router)# neighbor 192.168.11.2 remote-as 65000 sonic(config-router)# neighbor 192.168.12.2 remote-as 65000 sonic(config-router)# neighbor 192.168.13.1 remote-as 65000 sonic(config-router)# neighbor 192.168.15.1 remote-as 65000 sonic(config-router)# neighbor 192.168.14.1 remote-as 65000 sonic(config-router)# neighbor 192.168.16.1 remote-as 65000 sonic(config-router)# neighbor 192.168.13.1 update-source 192.168.11.1 sonic(config-router)# neighbor 192.168.15.1 update-source 192.168.12.1 sonic(config-router)# neighbor 192.168.14.1 update-source 192.168.11.1 sonic(config-router)# neighbor 192.168.16.1 update-source 192.168.12.1 sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# neighbor 192.168.13.1 activate sonic(config-router-af)# neighbor 192.168.15.1 activate sonic(config-router-af)# neighbor 192.168.14.1 activate sonic(config-router-af)# neighbor 192.168.16.1 activate sonic(config-router-af)# advertise-all-vni sonic(config-router-af)# exit sonic(config-router)# exit sonic(config)# router bgp 65000 sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# network 1.1.1.1/32 sonic(config-router-af)# network 192.168.11.0/24 sonic(config-router-af)# network 192.168.12.0/24 sonic(config)# vrf Vrf-red sonic(config-vrf)# vni 100 sonic(config-vrf)# exit sonic(config)# vrf Vrf-green sonic(config-vrf)# vni 101 sonic(config-vrf)# end sonic# configure terminal sonic(config)# router bgp 65000 vrf Vrf-red sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# end sonic# configure sonic(config)# router bgp 65000 vrf Vrf-green sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# end sonic# wr
Configurations on LEAF-2:
Step 1. Setup VLAN environment as per topology.
admin@sonic:~$ sudo config vlan add 30 admin@sonic:~$ sudo config vlan add 40 admin@sonic:~$ sudo config vlan add 100 admin@sonic:~$ sudo config vlan add 101 admin@sonic:~$ sudo config vlan member add -u 30 Ethernet8 admin@sonic:~$ sudo config vlan member add -u 40 Ethernet12
Step 2. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.13.1/24 admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.15.1/24 admin@sonic:~$ sudo config interface ip add Loopback10 2.2.2.2/32
Step 3. Configure VRF Setting.
admin@sonic:~$ sudo config vrf add Vrf-red admin@sonic:~$ sudo config vrf add Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan30 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan40 Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan100 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan101 Vrf-green admin@sonic:~$ sudo config interface ip add Vlan30 192.168.18.254/24 admin@sonic:~$ sudo config interface ip add Vlan40 192.168.19.254/24
Step 4. Create VxLAN.
admin@sonic:~$ sudo config vxlan add vtep 2.2.2.2 admin@sonic:~$ sudo config vxlan evpn_nvo add nvo vtep admin@sonic:~$ sudo config vxlan map add vtep 30 30 admin@sonic:~$ sudo config vxlan map add vtep 40 40 admin@sonic:~$ sudo config vxlan map add vtep 100 100 admin@sonic:~$ sudo config vxlan map add vtep 101 101
Step 5. Set the layer3 VNI on a switch.
admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-red 100 admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-green 101
Step 6. Establish a BGP environment for EVPN.
sonic# configure sonic(config)# no router bgp 65100 sonic(config)# router bgp 65000 sonic(config-router)# bgp bestpath as-path multipath-relax sonic(config-router)# neighbor 192.168.13.2 remote-as 65000 sonic(config-router)# neighbor 192.168.15.2 remote-as 65000 sonic(config-router)# neighbor 192.168.11.1 remote-as 65000 sonic(config-router)# neighbor 192.168.12.1 remote-as 65000 sonic(config-router)# neighbor 192.168.14.1 remote-as 65000 sonic(config-router)# neighbor 192.168.16.1 remote-as 65000 sonic(config-router)# neighbor 192.168.11.1 update-source 192.168.13.1 sonic(config-router)# neighbor 192.168.12.1 update-source 192.168.15.1 sonic(config-router)# neighbor 192.168.14.1 update-source 192.168.13.1 sonic(config-router)# neighbor 192.168.16.1 update-source 192.168.15.1 sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# neighbor 192.168.11.1 activate sonic(config-router-af)# neighbor 192.168.12.1 activate sonic(config-router-af)# neighbor 192.168.14.1 activate sonic(config-router-af)# neighbor 192.168.16.1 activate sonic(config-router-af)# advertise-all-vni sonic(config-router-af)# end sonic(config)# router bgp 65000 sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# network 2.2.2.2/32 sonic(config-router-af)# network 192.168.13.0/24 sonic(config-router-af)# network 192.168.15.0/24 sonic(config-router-af)# end sonic(config)# vrf Vrf-red sonic(config-vrf)# vni 100 sonic(config-vrf)# exit sonic(config)# vrf Vrf-green sonic(config-vrf)# vni 101 sonic# configure sonic(config)# router bgp 65000 vrf Vrf-red sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# exit sonic(config-router)# exit sonic(config)# router bgp 65000 vrf Vrf-green sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# end sonic# wr
Configurations on LEAF-3:
Step 1. Setup VLAN environment as per topology.
admin@sonic:~$ sudo config vlan add 50 admin@sonic:~$ sudo config vlan add 60 admin@sonic:~$ sudo config vlan add 100 admin@sonic:~$ sudo config vlan add 101 admin@sonic:~$ sudo config vlan member add -u 50 Ethernet8 admin@sonic:~$ sudo config vlan member add -u 60 Ethernet12
Step 2. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.14.1/24 admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.16.1/24 admin@sonic:~$ sudo config interface ip add Loopback10 3.3.3.3/32
Step 3. Configure VRF Setting.
admin@sonic:~$ sudo config vrf add Vrf-red admin@sonic:~$ sudo config vrf add Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan50 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan60 Vrf-green admin@sonic:~$ sudo config interface vrf bind Vlan100 Vrf-red admin@sonic:~$ sudo config interface vrf bind Vlan101 Vrf-green admin@sonic:~$ sudo config interface ip add Vlan50 192.168.20.254/24 admin@sonic:~$ sudo config interface ip add Vlan60 192.168.21.254/24
Step 4. Create VxLAN.
admin@sonic:~$ sudo config vxlan add vtep 3.3.3.3 admin@sonic:~$ sudo config vxlan evpn_nvo add nvo vtep admin@sonic:~$ sudo config vxlan map add vtep 50 50 admin@sonic:~$ sudo config vxlan map add vtep 60 60 admin@sonic:~$ sudo config vxlan map add vtep 100 100 admin@sonic:~$ sudo config vxlan map add vtep 101 101
Step 5. Set the layer3 VNI on a switch.
admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-red 100 admin@sonic:~$ sudo config vrf add_vrf_vni_map Vrf-green 101
Step 6. Establish a BGP environment for EVPN.
sonic# configure sonic(config)# no router bgp 65100 sonic(config)# router bgp 65000 sonic(config-router)# bgp bestpath as-path multipath-relax sonic(config-router)# neighbor 192.168.14.2 remote-as 65000 sonic(config-router)# neighbor 192.168.16.2 remote-as 65000 sonic(config-router)# neighbor 192.168.11.1 remote-as 65000 sonic(config-router)# neighbor 192.168.12.1 remote-as 65000 sonic(config-router)# neighbor 192.168.13.1 remote-as 65000 sonic(config-router)# neighbor 192.168.15.1 remote-as 65000 sonic(config-router)# neighbor 192.168.11.1 update-source 192.168.14.1 sonic(config-router)# neighbor 192.168.12.1 update-source 192.168.16.1 sonic(config-router)# neighbor 192.168.13.1 update-source 192.168.14.1 sonic(config-router)# neighbor 192.168.15.1 update-source 192.168.16.1 sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# network 3.3.3.3/32 sonic(config-router-af)# network 192.168.14.0/24 sonic(config-router-af)# network 192.168.16.0/24 sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# neighbor 192.168.11.1 activate sonic(config-router-af)# neighbor 192.168.12.1 activate sonic(config-router-af)# neighbor 192.168.13.1 activate sonic(config-router-af)# neighbor 192.168.15.1 activate sonic(config-router-af)# end sonic# configure sonic(config)# vrf Vrf-red sonic(config-vrf)# vni 100 sonic(config-vrf)# exit sonic(config)# vrf Vrf-green sonic(config-vrf)# vni 101 sonic(config-vrf)# end sonic# configure sonic(config)# router bgp 65000 vrf Vrf-red sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# exit sonic(config-router)# exit sonic(config)# router bgp 65000 vrf Vrf-green sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# redistribute connected sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# advertise ipv4 unicast sonic(config-router-af)# end sonic# wr
Configurations on SPINE-1:
The primary role of spine switches is to serve as route reflectors (server) for the leaf switches, which act as their clients in this network architecture.
Step 1. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.11.2/24 admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.13.2/24 admin@sonic:~$ sudo config interface ip add Ethernet8 192.168.14.2/24
Step 2. Establish a BGP environment for EVPN.
sonic# configure sonic(config)# no router bgp 65100 sonic(config)# router bgp 65000 sonic(config-router)# bgp bestpath as-path multipath-relax sonic(config-router)# neighbor 192.168.11.1 remote-as 65000 sonic(config-router)# neighbor 192.168.13.1 remote-as 65000 sonic(config-router)# neighbor 192.168.14.1 remote-as 65000 sonic(config-router)# neighbor 192.168.11.1 route-reflector-client sonic(config-router)# neighbor 192.168.13.1 route-reflector-client sonic(config-router)# neighbor 192.168.14.1 route-reflector-client sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# network 192.168.11.0/24 sonic(config-router-af)# network 192.168.13.0/24 sonic(config-router-af)# network 192.168.14.0/24 sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# neighbor 192.168.11.1 route-reflector-client sonic(config-router-af)# neighbor 192.168.13.1 route-reflector-client sonic(config-router-af)# neighbor 192.168.14.1 route-reflector-client sonic(config-router-af)# end sonic# wr
The figures below show the bgp summary and routing table of Spine-1.
Configurations on SPINE-2:
Step 1. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.12.2/24 admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.15.2/24 admin@sonic:~$ sudo config interface ip add Ethernet8 192.168.16.2/24
Step 2. Establish a BGP environment for EVPN.
sonic# configure sonic(config)# no router bgp 65100 sonic(config)# router bgp 65000 sonic(config-router)# bgp bestpath as-path multipath-relax sonic(config-router)# neighbor 192.168.12.1 remote-as 65000 sonic(config-router)# neighbor 192.168.15.1 remote-as 65000 sonic(config-router)# neighbor 192.168.16.1 remote-as 65000 sonic(config-router)# neighbor 192.168.12.1 route-reflector-client sonic(config-router)# neighbor 192.168.15.1 route-reflector-client sonic(config-router)# neighbor 192.168.16.1 route-reflector-client sonic(config-router)# address-family ipv4 unicast sonic(config-router-af)# network 192.168.12.0/24 sonic(config-router-af)# network 192.168.15.0/24 sonic(config-router-af)# network 192.168.16.0/24 sonic(config-router-af)# exit sonic(config-router)# address-family l2vpn evpn sonic(config-router-af)# neighbor 192.168.12.1 route-reflector-client sonic(config-router-af)# neighbor 192.168.15.1 route-reflector-client sonic(config-router-af)# neighbor 192.168.16.1 route-reflector-client sonic(config-router-af)# end sonic# wr
Results
Below is the result of LEAF-1 switch which shows that remotevtep like “2.2.2.2” and “3.3.3.3” has been learned through EVPN and the source IP is “1.1.1.1”.
The figure below shows the BGP summary of LEAF-1.
The figure below shows that it is a L3 VxLAN with Symmetric IRB because an L3 VNI is used in this topology.
The figures below show the vrf routes.
The result below shows that host PC1 successfully received responses from PC3 and PC5 since they are in same VRF but it is unable to receive response from PC4 because they both are in different VRF.
The result below shows that host PC6 successfully received response from PC2 since they are in same VRF but it is unable to receive response from PC1 because they both are in different VRF.
References
About Hardware Nation:
Hardware Nation is a professional services company that accelerates network transformation through an open networking approach, enabling freedom of choice, flexibility, and cost efficiency. Our seasoned experts have worked on projects for some of the world’s leading organizations, leveraging a hybrid cloud-first and AI-enabled approach. We help our customers navigate the ecosystem, drawing on decades of experience. Our deployments are powered by leading white box and OEM network, compute, and storage vendors. Our expertise encompasses a wide range of industries and use cases, including enterprise, cloud, data center, AI, 5G/ISP infrastructure, and edge IT.
Humza Atlaf
Network Engineer
Humza is a network engineer at Hardware Nation Labs, where his passion for Open Networking drives his work. With a blend of deep expertise and innovative approaches, he designs robust, scalable networks of the future. His practical experience includes configuring and deploying a range of protocols such as LACP, VLANs, MPLS, and VRRP. At his previous role, he was part of a SONiC testing team, further honing his skills in network setup and troubleshooting. Humza is also adept at network analysis with tools like Wireshark, enhancing his ability to manage complex network environments.
Alex Cronin
Co-Founder and Solutions Architect
Alex Cronin is a seasoned infrastructure architect with over 17 years of experience in networking and disaggregated infrastructure. His career is defined by aligning enterprise technology with business needs across diverse market segments, from emerging startups to Fortune 500 companies. He has worked on digital infrastructure projects covering network design and software solutions for data center operators, service providers, and enterprises. He is continuously collaborating with Hardware Nation Labs R&D to explore and pioneer the latest advancements in open networking and is assessing the applicability of AI/ML technology across enterprise, data center, and service provider infrastructures.
