AI training depends on GPUs exchanging data quickly and consistently. Congestion and network failures can slow that communication, leaving accelerators waiting for the next stage of a job.
Several announcements around October's OCP Global Summit focused on the software needed to operate these networks. Microsoft and Nokia reported making SONiC's route programming 6 to 12 times faster. Cisco expanded its validated SONiC offering for Nexus switches, while Meta, Microsoft and Alibaba introduced a specification for switch platform management.
SONiC is already in production at about 2,500 organizations, according to Gartner estimates reported by Network World. Dell'Oro Group forecasts compound annual growth above 50% from 2025 through 2030 for SONiC switch sales in AI scale-out networks.
How SONiC Is Built
SONiC, or Software for Open Networking in the Cloud, is an open-source network operating system that runs on switches from different manufacturers. It uses the OCP Switch Abstraction Interface (SAI) to provide a common interface for programming supported functions across different switch ASICs.
The ASIC handles packet forwarding, while SONiC runs routing protocols, applies configuration and monitors the system. Its major services run in containers. These include the FRRouting (FRR) stack, the switch state service, the sync daemon that programs the ASIC through SAI, and platform monitoring for fans, power supplies and transceivers.
Much of the state shared between these services is stored in Redis databases. CONFIG_DB holds configuration, APPL_DB holds state produced by applications such as the routing stack, and ASIC_DB represents ASIC objects. STATE_DB records operational status. Components also communicate through interfaces such as Netlink and sockets.
This architecture allows SONiC to support different hardware platforms with a common software base. Hardware-specific work is still required: SONiC needs the corresponding vendor SAI implementation and SDK support to use a feature in the ASIC.
Cisco Brings Validated SONiC to Nexus
Cisco is expanding the commercial support available for SONiC. Its October 6 announcement covers N9000 switches with Cisco Cloud Scale silicon for 100G and 400G leaf-spine networks, along with N9100 switches using NVIDIA Spectrum-X for backend AI fabrics. Cisco describes support for Silicon One-based N9000 platforms as the next step. It also publishes SONiC releases for its 8000 Series.
The validated builds include Nexus Dashboard management and Cisco TAC support for both hardware and software. Cisco also plans to extend Live Protect runtime vulnerability shielding to validated SONiC in the second quarter of 2027. NOS choices across supported Nexus models include NX-OS, ACI and SONiC, with availability depending on the platform.
Congestion Control Across Switches and NICs
RoCE, or RDMA over Converged Ethernet, allows compatible NICs to transfer data directly between application memory regions with little involvement from the host CPU. Reliable RoCE transports recover lost packets, but recovery can reduce throughput and increase latency. Conventional RoCEv2 fabrics therefore use congestion management to limit packet loss.
Many deployments combine Explicit Congestion Notification (ECN) with Priority Flow Control (PFC). With ECN, a switch marks packets as a queue fills. The receiving NIC responds with a Congestion Notification Packet (CNP), allowing the sender to reduce its transmission rate. DCQCN, a congestion control algorithm used by many RoCE NICs, uses this feedback to adjust the sending rate.
PFC acts on the link between neighboring devices. When a buffer approaches its limit, the downstream device sends a PFC frame asking its neighbor to pause transmission for the affected priority. Other priorities can continue transmitting on that link.
ECN feedback slows the sender, while PFC pauses one priority on one link
ECN feedback should reduce the sending rate before sustained PFC pauses develop. Coordinating the two requires an understanding of the switch's buffer architecture. On many ASICs, ECN marking uses egress queue occupancy, while PFC uses ingress priority-group occupancy. Their thresholds therefore cannot be treated as directly comparable values. If PFC pauses spread through the fabric, they can also slow unrelated flows sharing the paused priority.
SONiC configures these mechanisms through buffer pools, buffer profiles and queue settings. Lossless priority groups need enough headroom to receive data that remains in flight after a pause is sent. The required headroom depends on port speed, cable length, frame size and device response time.
WRED profiles with ECN enabled control marking thresholds for each queue. SONiC's PFC watchdog can detect sustained pauses and apply a recovery action, such as dropping traffic or ignoring pause frames. This can restore traffic flow, but it sacrifices losslessness while the action is in effect. Platform profiles provide starting settings; validation still needs to account for the NICs and traffic in the cluster.
From Routing Decision to Forwarding State
A routing decision has to reach the switch hardware before the ASIC can use it. Microsoft and Nokia's OCP session examined the software involved in that process and reported a 6 to 12 times improvement in route-programming speed.
In the conventional Redis-based SONiC path, FRR's bgpd passes selected routes to Zebra. Zebra exports forwarding updates through FPM to fpmsyncd, which writes the routes to APPL_DB. Orchagent processes that state, resolves next hops and issues SAI operations through sairedis into ASIC_DB. Syncd then invokes the vendor SAI implementation to program the chip.
A new route crosses a chain of software handoffs before it reaches the ASIC
Processing, serialization and communication between components all contribute to the time this takes. The Microsoft and Nokia work focused on profiling and optimizing the path. The gain on a particular deployment will depend on the switch platform, software release and route scale.
Faster programming is especially useful when a switch needs to install many routes, such as during startup or after a large routing change. Its contribution to failure recovery depends on the failure. Existing ECMP members or hardware protection may restore forwarding before control-plane convergence finishes.
Higher port speeds do not automatically make these software operations faster. Route-programming performance also depends on the switch CPU, the software implementation and the vendor SDK.
Standardizing the Platform Layer
Supporting an ASIC through SAI is one part of bringing SONiC to a switch. The operating system also needs working interfaces to the fans, temperature sensors, power supplies, transceivers and firmware. Those interfaces require integration and validation for each switch model.
The Open Switch Platform Architecture (OSPA), initiated by Meta, Microsoft and Alibaba, aims to make this work more consistent. It defines a vendor-neutral reference architecture in which platform services run on the host CPU and the baseboard management controller retains supervisory and recovery functions. It also standardizes debugging information, including event logs and hardware fault records.
The draft is under OCP review and does not cover routing or forwarding. Adoption could reduce the integration work required to support new switch models across network operating systems.
Supported Distributions
Operating community SONiC requires expertise in the software and the hardware it runs on. Commercial distributions provide another option for teams that need vendor support. Arista, Cisco, Dell and NVIDIA support SONiC on their hardware, with the supported models, features and service arrangements varying by vendor.
The community is also adding capabilities for AI networks. The SONiC Foundation describes work on SRv6, packet trimming and high-frequency telemetry. Availability on a particular switch depends on its hardware and software support.
SONiC in Co-Designed Systems
SONiC also runs in systems whose switches and NICs are designed to work together. NVIDIA offers SONiC within Spectrum-X, which combines switch adaptive routing with out-of-order handling at the receiving SuperNIC. NVIDIA described that coordination in its account of June's MLPerf Training v6.0 results.
OpenAI and Microsoft use MRC, a multipath RDMA transport that operates without PFC, in production, including on NVIDIA ConnectX-8 NICs. Support for its static SRv6 source routing was implemented on switches running SONiC, Cumulus and Arista EOS. The MRC specification was contributed through OCP.
In a May 2026 Tolly evaluation commissioned by Dell, Dell SONiC on NVIDIA Spectrum switches with BlueField-3 SuperNICs achieved 97.5% NCCL AllReduce bus bandwidth efficiency across 32 H200 GPUs at 400GbE. That result applies to the tested configuration. Performance on other fabrics depends on their hardware, software and workloads.
What This Means for AI Networks
At Hardware Nation, we help customers deploy SONiC on disaggregated switching hardware. Preparing a system for production requires evaluating the switch platform, SONiC release and NIC configuration together, including congestion settings and how the network behaves during failures.
SONiC gives operators a common software base across supported switches and the ability to customize network behavior. The engineering required to operate it depends on the distribution, platform integration and management tools.
For teams evaluating an AI fabric, three questions are worth asking the supplier. Who validates the SAI and SDK combination for the switch model? How were the QoS profiles tested against the NICs in use? What NCCL performance has been demonstrated at the intended cluster scale?
Alex Cronin
Alex is a seasoned business leader and AI innovator with over 18 years of experience aligning enterprise IT solutions with business outcomes. A Georgia Tech alumnus, he has led open networking transformations across data centers, service providers, and hyperscalers. As the founder of Hardware Nation and now at the helm of Nanites AI, he’s partnering with major enterprises to build the world’s first closed-loop network autopilot powered by agentic AI. A featured speaker at CableLabs and a Top 12 finalist in the 2025 T-Mobile T-Challenge, Alex is a strong advocate for open infrastructure and autonomous systems.
He has partnered with industry-leading organizations including Deutsche Telekom, Accton, Accenture, Stanford, Georgia Tech, NASA, and Lockheed Martin, and has led hundreds of projects spanning startups to Fortune 500 enterprises.
