This guide offers detailed instructions for configuring interoperability testing of a Spine-Leaf topology using Asymmetric Integrated Routing and Bridging (IRB) Ethernet VPN (EVPN) between SONiC and Cisco. It covers VLAN setup, IP address configuration, VxLAN creation, and BGP establishment for EVPN.
In our previous article, “Spine-Leaf Topology with Asymmetric IRB EVPN in SONiC,” we thoroughly explained the Spine-Leaf architecture and its significance in modern data center environments.
Testbed & NOS Version
- GNS3
- SONiC ( 202305 )
- Cisco Nexus 9K
The figures below display the versions of SONiC and Cisco, respectively.
Topology
Steps to Configure Asymmetric IRB EVPN Fabric
- Setup VLAN Environment
- Configure IP addresses
- Create VxLAN
- Establish BGP environment for EVPN
Note: The above steps are only for Leaf switches. For Spine switches there is no need to create VLANs and VxLAN.
SONiC Switches
The leaf layer is composed of SONiC switches.
Configurations on LEAF-1:
Step 1. Setup VLAN environment as per topology.
admin@sonic:~$ sudo config vlan add 10
admin@sonic:~$ sudo config vlan add 20
admin@sonic:~$ sudo config vlan member add -u 10 Ethernet8
admin@sonic:~$ sudo config vlan member add -u 20 Ethernet12
Step 2. Configure IP addresses.
admin@sonic:~$ sudo config interface ip add Ethernet0 192.168.11.1/24
admin@sonic:~$ sudo config interface ip add Ethernet4 192.168.12.1/24
admin@sonic:~$ sudo config interface ip add Loopback10 1.1.1.1/32
admin@sonic:~$ sudo config interface ip add Vlan10 192.168.10.254/24
admin@sonic:~$ sudo config interface ip add Vlan20 192.168.17.254/24
Below is the status of the VLAN table.
Step 3. Create VxLAN
admin@sonic:~$ sudo config vxlan add vtep 1.1.1.1
admin@sonic:~$ sudo config vxlan evpn_nvo add nvo vtep
admin@sonic:~$ sudo config vxlan map add vtep 10 10
admin@sonic:~$ sudo config vxlan map add vtep 20 20
Below is the status of the VxLAN Tunnel.
Below is the status of the VxLAN Tunnel.
Step 4. Establish a BGP environment for EVPN
sonic# configure
sonic(config)# no router bgp 65100
sonic(config)# router bgp 65000
sonic(config-router)# bgp router-id 1.1.1.1
sonic(config-router)# bgp bestpath as-path multipath-relax
sonic(config-router)# neighbor 192.168.11.2 remote-as 65000
sonic(config-router)# neighbor 192.168.12.2 remote-as 65000
sonic(config-router)# neighbor 192.168.13.1 remote-as 65000
sonic(config-router)# neighbor 192.168.15.1 remote-as 65000
sonic(config-router)# neighbor 192.168.14.1 remote-as 65000
sonic(config-router)# neighbor 192.168.16.1 remote-as 65000
sonic(config-router)# neighbor 192.168.13.1 update-source 192.168.11.1
sonic(config-router)# neighbor 192.168.15.1 update-source 192.168.12.1
sonic(config-router)# neighbor 192.168.14.1 update-source 192.168.11.1
sonic(config-router)# neighbor 192.168.16.1 update-source 192.168.12.1
sonic(config-router)# address-family ipv4
sonic(config-router-af)# network 1.1.1.1/32
sonic(config-router-af)# exit
sonic(config-router)# address-family l2vpn evpn
sonic(config-router-af)# neighbor 192.168.13.1 activate
sonic(config-router-af)# neighbor 192.168.15.1 activate
sonic(config-router-af)# neighbor 192.168.14.1 activate
sonic(config-router-af)# neighbor 192.168.16.1 activate
sonic(config-router-af)# advertise-all-vni
sonic(config-router-af)# end
sonic# wr
Configurations on LEAF-2:
Step 4. Establish a BGP environment for EVPN.
sonic# configure
sonic(config)# no router bgp 65100
sonic(config)# router bgp 65000
sonic(config-router)# bgp router-id 2.2.2.2
sonic(config-router)# bgp bestpath as-path multipath-relax
sonic(config-router)# neighbor 192.168.13.2 remote-as 65000
sonic(config-router)# neighbor 192.168.15.2 remote-as 65000
sonic(config-router)# neighbor 192.168.11.1 remote-as 65000
sonic(config-router)# neighbor 192.168.12.1 remote-as 65000
sonic(config-router)# neighbor 192.168.14.1 remote-as 65000
sonic(config-router)# neighbor 192.168.16.1 remote-as 65000
sonic(config-router)# neighbor 192.168.11.1 update-source 192.168.13.1
sonic(config-router)# neighbor 192.168.12.1 update-source 192.168.15.1
sonic(config-router)# neighbor 192.168.14.1 update-source 192.168.13.1
sonic(config-router)# neighbor 192.168.16.1 update-source 192.168.15.1
sonic(config-router)# address-family ipv4
sonic(config-router-af)# network 2.2.2.2/32
sonic(config-router-af)# exit
sonic(config-router)# address-family l2vpn evpn
sonic(config-router-af)# neighbor 192.168.11.1 activate
sonic(config-router-af)# neighbor 192.168.12.1 activate
sonic(config-router-af)# neighbor 192.168.14.1 activate
sonic(config-router-af)# neighbor 192.168.16.1 activate
sonic(config-router-af)# advertise-all-vni
sonic(config-router-af)# end
sonic# wr
Configurations on LEAF-3:
Step 4. Establish a BGP environment for EVPN.
sonic# configure
sonic(config)# no router bgp 65100
sonic(config)# router bgp 65000
sonic(config-router)# bgp router-id 3.3.3.3
sonic(config-router)# bgp bestpath as-path multipath-relax
sonic(config-router)# neighbor 192.168.14.2 remote-as 65000
sonic(config-router)# neighbor 192.168.16.2 remote-as 65000
sonic(config-router)# neighbor 192.168.11.1 remote-as 65000
sonic(config-router)# neighbor 192.168.12.1 remote-as 65000
sonic(config-router)# neighbor 192.168.13.1 remote-as 65000
sonic(config-router)# neighbor 192.168.15.1 remote-as 65000
sonic(config-router)# neighbor 192.168.11.1 update-source 192.168.14.1
sonic(config-router)# neighbor 192.168.12.1 update-source 192.168.16.1
sonic(config-router)# neighbor 192.168.13.1 update-source 192.168.14.1
sonic(config-router)# neighbor 192.168.15.1 update-source 192.168.16.1
sonic(config-router)# address-family ipv4
sonic(config-router-af)# network 3.3.3.3/32
sonic(config-router-af)# exit
sonic(config-router)# address-family l2vpn evpn
sonic(config-router-af)# neighbor 192.168.11.1 activate
sonic(config-router-af)# neighbor 192.168.12.1 activate
sonic(config-router-af)# neighbor 192.168.13.1 activate
sonic(config-router-af)# neighbor 192.168.15.1 activate
sonic(config-router-af)# advertise-all-vni
sonic(config-router-af)# end
sonic# wr
Cisco Switches
The Spine Layer is composed of Cisco switches.
Configurations on SPINE-1:
The primary role of spine switches is to serve as route reflectors (server) for the leaf switches, which act as their clients in this network architecture.
Step 1. Configure IP addresses.
switch(config-if)# interface e1/1
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.11.2 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# interface e1/2
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.13.2 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# interface e1/3
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.14.2 255.255.255.0
switch(config-if)# no shutdown
Step 2. Establish a BGP environment for EVPN.
In Cisco Nexus 9k switches, it is necessary to enable all features related to VxLAN BGP EVPN Fabric before utilizing them.
switch(config)# feature bgp
switch(config)# feature vn-segment-vlan-based
switch(config)# feature vn-segment
switch(config)# feature nv overlay
switch(config)# feature interface-vlan
switch(config)# nv overlay evpn
switch(config)# router bgp 65000
switch(config-router)# bestpath as-path multipath-relax
switch(config-router)# neighbor 192.168.11.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# neighbor 192.168.13.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# neighbor 192.168.14.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# address-family ipv4 unicast
switch(config-router-af)# network 192.168.11.0 mask 255.255.255.0
switch(config-router-af)# network 192.168.13.0 mask 255.255.255.0
switch(config-router-af)# network 192.168.14.0 mask 255.255.255.0
switch(config-router-af)# exit
switch(config-router)# exit
switch(config)# exit
switch# wr
Configurations on SPINE-2:
Step 1. Configure IP addresses.
switch(config-if)# interface e1/1
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.12.2 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# interface e1/2
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.15.2 255.255.255.0
switch(config-if)# no shutdown
switch(config-if)# interface e1/3
switch(config-if)# no switchport
switch(config-if)# ip address 192.168.16.2 255.255.255.0
switch(config-if)# no shutdown
Step 2. Establish a BGP environment for EVPN.
switch(config)# feature bgp
switch(config)# feature vn-segment-vlan-based
switch(config)# feature vn-segment
switch(config)# feature nv overlay
switch(config)# feature interface-vlan
switch(config)# nv overlay evpn
switch(config)# router bgp 65000
switch(config-router)# bestpath as-path multipath-relax
switch(config-router)# neighbor 192.168.12.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# neighbor 192.168.15.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# neighbor 192.168.16.1 remote-as 65000
switch(config-router-neighbor)# address-family ipv4 unicast
switch(config-router-neighbor-af)# route-reflector-client
switch(config-router-neighbor-af)# exit
switch(config-router-neighbor)# exit
switch(config-router)# address-family ipv4 unicast
switch(config-router-af)# network 192.168.12.0 mask 255.255.255.0
switch(config-router-af)# network 192.168.15.0 mask 255.255.255.0
switch(config-router-af)# network 192.168.16.0 mask 255.255.255.0
switch(config-router-af)# exit
switch(config-router)# exit
Results
Below is the result of LEAF-1 switch which shows that remotevtep like “2.2.2.2” and “3.3.3.3” has been learned through EVPN and the source IP is “1.1.1.1”.
The figure below shows the MAC address of host PC6 and PC3. The switch LEAF-1 will learn this MAC address as it is Asymmetric IRB EVPN.
Four Hosts are attached on LEAF-2 and LEAF-3 switches. In Asymmetric IRB EVPN, the MAC address of the destination host is learned. The figure below shows that destination MAC addresses of four Hosts have been learned on LEAF-1.
The figure below shows that Asymmetric IRB is succesfully configured on LEAF-1 beacuse all the VNIs are type L2 and no L3 VNI is used.
The figure below shows the BGP summary of LEAF-1.
The figure below displays the list of routes learned through BGP by LEAF-1.
The figure below is the result of LEAF-2.
The figure below is the result of LEAF-3.
The figure below shows the neighbors of SPINE-1.
Below is the result of routes learned by SPINE-1 through BGP.
The figure displays the result of SPINE-1.
To verify the use of two VNIs in the network, packets were captured using Wireshark. The packet capture confirmed that both VNIs 10 and 20 are utilized, depending on which host is generating a ping request.
PC1
PC1 receives a successful reply from PC6 and TTL is decremented by one because both are in different subnets.
PC1 receives a successful reply from PC5 and TTL remains the same because both are in the same subnet.
PC6
PC6 receives a successful reply from PC4 and TTL remains the same because both are in the same subnet.
PC6 receives a successful reply from PC1 and TTL is decremented by one because both are in different subnets.
References
About Hardware Nation:
Hardware Nation is a professional services company that accelerates network transformation through an open networking approach, enabling freedom of choice, flexibility, and cost efficiency. Our seasoned experts have worked on projects for some of the world’s leading organizations, leveraging a hybrid cloud-first and AI-enabled approach. We help our customers navigate the ecosystem, drawing on decades of experience. Our deployments are powered by leading white box and OEM network, compute, and storage vendors. Our expertise encompasses a wide range of industries and use cases, including enterprise, cloud, data center, AI, 5G/ISP infrastructure, and edge IT.
Humza Atlaf
Network Engineer
Humza is a network engineer at Hardware Nation Labs, where his passion for Open Networking drives his work. With a blend of deep expertise and innovative approaches, he designs robust, scalable networks of the future. His practical experience includes configuring and deploying a range of protocols such as LACP, VLANs, MPLS, and VRRP. At his previous role, he was part of a SONiC testing team, further honing his skills in network setup and troubleshooting. Humza is also adept at network analysis with tools like Wireshark, enhancing his ability to manage complex network environments.
Alex Cronin
Co-Founder and Solutions Architect
Alex Cronin is a seasoned Solutions Architect with over 15 years of experience in networking and disaggregated infrastructure. His career is defined by aligning enterprise technology with business needs across diverse market segments, from emerging startups to Fortune 500 companies. He has worked on digital infrastructure projects covering network design and software solutions for data center operators, service providers, and enterprises. He is continuously collaborating with Hardware Nation Labs R&D to explore and pioneer the latest advancements in open networking and is assessing the applicability of AI/ML technology across enterprise, data center, and service provider infrastructures.
