Offloading VNFs to P4-programmable SmartNICs with an ONOS SDN controller

The challenge

Delays from virtual network functions held back the performance of the whole data center network, and the distribution of workloads across the data center kept changing. A SmartNIC could take network functions off the servers, but it had to be reconfigurable as workloads moved and fit the existing SDN operations, so the same controller and the same intra-DC policies governed it alongside the rest of the network.

What was at risk:

• Delays across the data center network, and a SmartNIC managed apart from the policies the SDN controller enforces

What we did

• Created the solution architecture with network and software engineers experienced in SDN and low-level hardware programming.
• Developed an example firewall in P4 for a Netronome Agilio SmartNIC.
• Selected ONOS, an open-source SDN controller with P4Runtime support, and developed a custom control plane application in it.
• Developed SmartNIC proxy software that converts P4Runtime to Thrift to control the SmartNIC data plane.
• Set up and tested a PoC reusable for load balancers, NAT and other network functions, and a demo of the solution.

How it works

One SDN controller drives a mixed data plane: the SmartNIC through P4Runtime and a switch fabric through OpenFlow. The PoC has five parts.

1. Firewall in P4 The firewall runs on a Netronome Agilio SmartNIC with two 10GbE interfaces, written in P4 for the V1Model architecture. Its logic sits in two ingress tables: a forwarding table that sends packets from an input port to an output port or drops them, and a firewall table that matches fields such as the IPv4 protocol and the Ethernet source address and allows or drops the packet. The build produces the Netronome binary, a design file and the P4Info file that describes the data plane.
2. Control application in ONOS A custom ONOS application has two parts. The pipeconf maps ONOS internal data structures to the fields defined in P4 and, from the P4Info file, maps P4 names to the identifiers P4Runtime uses. The control application holds a firewall service that manages devices and rules, with a web UI and firewall commands added to the ONOS CLI.
3. SmartNIC proxy The SmartNIC’s P4Runtime support was an early version incompatible with the one in ONOS, and it lacked ternary matching and packet-out. CodiLime built a proxy: its gRPC server accepts the P4Runtime writes from ONOS, repacks them and sends them to the SmartNIC over Thrift, the card’s runtime protocol.
4. One controller, two protocols ONOS also controls a leaf-spine fabric of Open vSwitch instances (two spines, four leaves) over OpenFlow, with routing between subnets. Traffic from emulated end users crosses the fabric, passes the SmartNIC firewall and reaches the application servers, so one controller manages both the SmartNIC and the switches.
5. Automated setup A builder machine provisions the servers with Ansible playbooks and a CI/CD pipeline builds the Docker images: the ONOS instance, the SmartNIC proxy, the emulated fabric and the end-user and server hosts. Tests applied firewall rules on Layer 2 and Layer 3 parameters and blocked specific traffic patterns.

Another network function, such as a load balancer or NAT, needs its own P4 implementation and reuses the rest of the PoC.