Benchmarking SmartNICs to RFC 2544 with open-source traffic generation and automated reporting

The challenge

RFC 2544 sets the packet loss threshold at zero, so the test setup itself must never become the bottleneck. The traffic generator has to outpace the device under test, and virtualization layers, kernel interrupts, NUMA placement and CPU contention all disturb latency measurements. A full RFC 2544 run across packet sizes from 64 to 1500 bytes, with eight 60-second binary-search iterations, takes about 56 minutes, which makes frequent manual runs costly.

What was at risk:

• Results distorted by the test environment, from tester limits to time synchronization and unplanned congestion, read as defects in the product
• Performance degradation found late in the development cycle, because full runs are slow and run rarely
• Defects surfacing only once the driver and applications sit on top of the firmware, where fixes cost more

What we did

• Built the test process on RFC 2544 for throughput, latency and frame loss.
• Used the open-source TRex traffic generator on bare metal, with DPDK-bound interfaces, as the source of RFC 2544 traffic.
• Verified the traffic generator in a loopback topology before connecting it to the device under test.
• Shielded measurements from the host with NUMA-aware core and memory assignment, CPU pinning, huge pages and queue mapping for TRex and the TestPMD reflector.
• Tested the SmartNIC layer by layer: firmware first, as a plain traffic reflector, then the driver module and user applications.
• Automated environment setup, test execution, report generation and analysis in a Jenkins CI/CD pipeline with Ansible, Docker and Pytest.
• Generated reports from per-stream TRex statistics, flagging automatically where the RFC 2544 threshold was missed, in versions for technical and for business audiences.

How it works

The framework takes a device from an empty testbed to a finished report in six stages, each automated where repetition pays for it.

1. Topology as code The test topology is described in a YAML file: the servers, the virtual machines and where the traffic generator and the device under test sit. Ansible deploys it, verifies the network interfaces and binds them to DPDK drivers, so topology errors surface before any test runs. TRex and the device can run on separate servers or, to cut costs, on one server with isolated CPU, NUMA and memory resources.
2. Testbed verification Smoke tests confirm that the environment is configured and stable, and that TRex reaches the expected traffic level in a loopback topology, so the generator is proven before it measures anything. To fully load a device with 25 Gbps duplex throughput, the generator has to exceed that rate.
3. Layered device testing The SmartNIC is first tested with firmware only, acting as a simple traffic reflector, to check traffic load, distribution to queues and memory operation. The driver module, user applications and final firmware follow. TRex varies the 5-tuple from packet to packet to confirm that RSS spreads traffic across the device’s queues, and TestPMD statistics show where packets are lost.
4. RFC 2544 runs UDP streams run across packet sizes from 64 to 1500 bytes, with a binary search of eight iterations per size; a size passes when all eight iterations show zero packet loss. Python scripts on the TRex API, or the TRex NDR script for a fast first result, drive the runs from Pytest. For a device still in development, an agreed loss threshold is documented before testing starts.
5. Scheduling in CI/CD Jenkins runs the suites, assigns test environments to users and splits the work into jobs. Tests are grouped by importance and resource cost, so packet-size runs can go daily, others weekly, and selected suites on pull requests.
6. Reporting Per-stream TRex statistics, tagged at transmission, leave out background traffic such as ARP. Results are exported to CSV, analysed with Pandas, compared with the theoretical maximum packets per second and flagged where the RFC 2544 threshold was missed. Technical teams get full results and test configurations; business teams get a summary set against their objectives.

Extra tools where the device needs them. TRex parsers do not support MPLS over UDP, so a router or switch can sit between the generator and the device as a translation layer, and Bird or FRR virtual routers in Docker support LACP and LAG failover tests.