SDN dataplane offloading for telco clouds: 18 cores down to 2

The challenge

In a telco cloud the Tungsten Fabric vRouter processes packets in software on the host, and that took 18 CPU cores that workloads need, while throughput stayed under line rate. Moving it onto FPGA SmartNICs meant a custom data plane for VXLAN and MPLSoUDP, integration with the existing vRouter alongside SR-IOV and vDPA, orchestration for OpenStack and Kubernetes with minimal manual steps, and a CI/CD and QA pipeline for every new FPGA firmware image and driver release.

What was at risk:

• Host CPU cores spent on packet processing instead of workloads, throughput below line rate, and an offload that still had to prove its reliability and target performance to telcos

What we did

• Extended the Tungsten Fabric vRouter for full data plane offload, with SmartNIC-aware pipeline management and SR-IOV integration.
• Created a vendor-specific DPDK PMD driver supporting SmartNIC offload for IPv4/IPv6, VXLAN, MPLSoUDP and port mirroring.
• Built a P4-based virtual router on FPGA/ARM SoC SmartNICs, with a kernel module that mirrors Linux routing tables into P4 logic.
• Automated provisioning in OpenStack through customized TripleO Heat templates: SmartNIC bring-up, FPGA bitstream flashing and SR-IOV setup.
• Built a custom Kubernetes CNI plugin based on the Tungsten Fabric CNI, chained through Multus with the SR-IOV device plugin, deployed with Helm and managed by Kubernetes controllers.
• Delivered an automated RFC 2544 test framework with Python, TRex and DPDK tools, Jenkins-based CI/CD pipelines and a reservation system for developers.
• Supported the client in specifying the overall system requirements and verification process for its discussions with telcos.

How it works

The vRouter keeps its role in the SDN platform, and the SmartNIC takes over its packet processing. Five parts make that work and keep it deployable.

1. Data plane offload The custom DPDK poll mode driver translates the vRouter’s internal flow representation into rte_flow rules and then into the SmartNIC’s own data structures. It creates port representors, manages flows in-band and asynchronously, and ages hardware flows. The full feature set runs on the card: VXLAN and MPLS encapsulation, NAT and mirroring.
2. Routing in P4 A P4-based virtual router runs on the FPGA/ARM SoC SmartNIC, with offloaded MPLS midhaul and fronthaul. A kernel module mirrors Linux routing tables into the P4 tables through a bridge from netlink, and routes are offloaded lazily, by heuristic.
3. OpenStack Customized TripleO Heat templates add roles for card configuration and vhost interface bring-up. An initialization container configures the SmartNIC at startup, and container entry points flash and switch the FPGA bitstream, set up SR-IOV and pass the offload arguments to the DPDK vRouter. Nova was tuned for virtual machines that use SmartNIC virtual functions.
4. Kubernetes A custom CNI plugin based on the Tungsten Fabric CNI manages the port representors the SmartNIC provides, chained through Multus with the SR-IOV device plugin. A Helm chart deploys the routing daemon with its offload containers, and Kubernetes controllers handle SmartNIC initialization and lifecycle. KubeVirt virtual machines get the offload through a modified DPDK vhost library.
5. Verification An automated RFC 2544 framework built on Python, TRex and DPDK tools compares pass-through and vDPA performance and tracks regressions. Unit, integration and system tests (regression, functional and performance) are fully automated, the test environment configures virtual networks and the underlay on its own, and Jenkins pipelines run them for every firmware image and driver release.

The whole infrastructure is defined as code, so CI/CD ran from the start of the project.