As we can see in Ethernet Alliance’s 2023 Roadmap 
, we can now use link speeds up to 400Gb/s. Speeds 800Gb/s and 1.6Tb/s are “in development” and 3.2Tb/s is marked as a “possible future speed” for the 2030s. Compared to 10Mb/s in the 1980s, now we have devices functioning 40,000 times faster! With such acceleration, Jules Verne’s “Around the World in Eighty Days” would be called “Around the World in 2.88 Minutes”.
The goal you need to achieve during test environment preparation is that the tested network interface card (also called a NIC, network interface controller, or network adapter) will be a bottleneck in the hardware configuration. This is very difficult to achieve these days and involves a number of key issues.
This article will help you better understand how to set up a well-optimized testing environment, and what the crucial aspects are that you should be aware of to succeed.
Hardware for NICs testing
To meet the needs of the market, manufacturers of network devices offer faster, smarter network interface cards known as Smart Network Interface Cards (SmartNICs) - you can check out the differences between them in our article, where we describe different SmartNICs types and features.
There are several important considerations to keep in mind when selecting your hardware.
Vendor documentation
When testing network cards, it is essential to thoroughly familiarize yourself with the manufacturer's documentation to ensure optimal performance and compatibility with other hardware in your environment.
Familiarizing yourself in this way enables you to grasp the full range of functions that can be utilized. Some NICs offer offloading functionality, which refers to the card's ability to handle certain tasks, such as TCP/IP checksum calculation, segmentation offloading, or protocol-specific processing, without burdening the computer's CPU.
In the event of encountering issues with the network card's operation, the manufacturer's documentation often proves to be a valuable resource. It provides a list of potential problems and their solutions, diagnostic recommendations, and technical support that can be consulted when difficulties arise.
When reviewing marketing materials prepared by vendors for your device under test (DUT), it is important to be aware that they often present the best results. However, it is crucial to note that these results can be difficult to replicate in a configuration aligned with your specific needs. This is because the settings applied in these materials are often optimized for achieving better results rather than reflecting real-life configurations and usage scenarios.
Hardware configuration for NICs testing
PCIe size and speed
The first thing you need to do is to check if your motherboard and CPU can handle the full speed of the NIC under test.
Check the processor specifications and make sure it supports the version of Peripheral Component Interconnect Express (PCIe) required by the network card. Also, the motherboard’s PCIe slot must support the NIC’s PCIe size and speed. Below you can see a table with throughput supported by different PCIe versions and sizes.
Transfer rate refers to the encoded serial bit rate; 2.5 GT/s means 2.5 Gbit/s serial data rate.
Throughput indicates the unencoded bandwidth. For example a PCIe 1.0 transfer rate of 2.5 GT/s per lane means a 2.5 Gbit/s serial bit rate corresponding to a throughput of 2.0 Gbit/s or 250 MB/s prior to 8b/10b encoding.
Note that some cards may need more power than a PCIe slot is able to deliver. If the card requires more power for proper operation, it must be supplied from the power supply with an additional cable.
PCIe line sharing between devices
Note that the PCIe line can be shared between devices connected to it, such as storage controllers or graphics cards, which can significantly impact the performance of a network interface card. Therefore, it is crucial to choose a PCIe slot that does not share the PCIe line with another slot where another device is connected.
To select a PCIe slot that avoids sharing the PCIe line with other devices, you can follow the motherboard documentation, looking for any indications or diagrams that show which PCIe slots are physically divided and can be used to avoid sharing the PCIe line. Also there may be labeling on the motherboard.
Some manufacturers label the PCIe slots in a way that allows you to identify which slots are connected and share the same PCIe line. They may use different colors or numbering to indicate groups of slots that are connected. If you don’t have physical access to the motherboard, but have remote access to BIOS, for example through some management platform (most motherboard manufacturers provide such platforms; e.g. Dell’s IDRAC, HP’s iLO, and so on), you can explore the BIOS options to find information about specific PCIe slots and their configuration.
NUMA (non-uniform memory access) in multiprocessor systems
In a multiprocessor system (also known as multiprocessor servers or multi-socket systems), another factor to consider when selecting a PCIe slot for a NIC is the concept of non-uniform memory access (NUMA).
NUMA is a system architecture that is commonly used in multiprocessor systems. It allows each processor to access its own local memory faster than remote memory. In NUMA systems, memory is divided into multiple banks, called ‘NUMA nodes’, with each processor having direct access to a specific set of memory. This architecture optimizes memory access for each CPU, but it can also have an impact on the performance of devices connected to PCIe slots.
To determine the memory node configuration and select the appropriate PCIe slot, you can refer to the motherboard documentation, consult the manufacturer's specifications or use the Portable Hardware Locality (hwloc) software package with lstopo, which can generate nice-looking architecture diagrams. This should provide information on the memory node mapping for each processor and how it relates to the PCIe slots.
By distributing PCIe devices across different memory nodes you can balance the workloads and ensure efficient memory access for the NIC under test. To achieve optimal performance results, we need the NIC to be a bottleneck in the tested topology. That’s why we need a memory setup that is as fast as possible. In conclusion, we need all the test bed components connected to the same NUMA node, e.g. a PCIe slot, an application used for testing, and pinned CPU cores. However, exploring this topic in its entirety would require a more comprehensive and detailed article.
Additionally, it is important to consider configuring the operating system and device drivers to be NUMA-aware. This involves adjusting settings such as processor affinity and memory allocation policies to ensure that network operations are effectively distributed across the available resources in the NUMA system.
By considering both avoidance of sharing the PCIe line with other devices and the NUMA architecture in a multiprocessor system, you can optimize the performance of your network interface cards and achieve reliable and accurate test results.
Storage performance
Storage performance is the next possible bottleneck for satisfactory throughput. Before you start testing data transfer through the network card, test it locally on the server site. If local performance isn’t fast enough to cover the max speed supported by the NIC, remember to update the firmware of your storage controller and place the controller in the proper PCIe slot, according to the PCIe sharing and NUMA considerations from the previous section.
Implementing RAID (redundant array of independent disks) will allow you to obtain a performance several times better than using a single disk and let you customize configuration depending on the data you want to measure that performance with. RAID can be implemented using either hardware or software solutions.
Utilizing a hardware RAID controller can provide performance benefits as some calculations are performed at the hardware level, relieving the system processor. Additionally, hardware RAID controllers often offer advanced caching and disk operation acceleration techniques, which can enhance performance in network card testing.
If you have a test environment with modern processors, it is worth noting that software RAID can potentially be faster than hardware RAID. Modern processors are equipped with advanced computational capabilities and multiple cores, allowing them to efficiently handle software-based RAID calculations. Utilizing CPU resources for RAID management and data processing, can take advantage of these capabilities.
Test environment topology
When choosing a topology for network adapter testing, consider the testing objectives, real-world scenarios, scalability and monitoring requirements. Determine the specific performance metrics you want to evaluate, such as throughput, latency or packet loss.
When looking for the best topology, familiarize yourself with the DPDK performance reports here , but remember that these reference environments may not fit your specific requirements and needs in the real world.
Connecting a network card’s ports directly to the second device’s network card port seems to be the best and the most intuitive option, eliminating possible speed drops. And it is. The more elements and devices between client and server, the more there is to break. The best option would be to have the same model of NIC on both sides.
On the other hand, this topology significantly limits test scope. Network devices provide a lot of functionalities and features which can behave differently on different network cards. For more complex tests it’s necessary to use a network switch between the devices.
When choosing a switch, pay attention to the supported speed on a single port, which should match the speed of the network card under test or be higher. Make sure that the switch has sufficient potential performance to ensure full speed on all ports of the NIC being tested. The switch should also support, for example, link aggregation, load balancing, VLANs or other functionalities you want to test on tested NIC. This topology allows you to test different types of bondings, load balancing from several clients or MPIO using both NIC ports. The optimal choice is to isolate the entire environment from other networks to avoid any interference with your tests. However, if isolating the entire environment is not feasible, it is advisable to isolate the ports on the switch to minimize the impact of other traffic.
Software for NICs testing
When you have the hardware part of the preparation done it’s time to take a look at the software, which is just as important. The first thing you need to do is to check compatibility with your hardware and operating system.
BIOS and firmware update
It is strongly recommended that you update all components of your environment before testing your network card. Where to download the latest version of firmware depends on the manufacturer of your devices. Please follow the vendors’ instructions on how to perform this update. There are probably downloadable tools for your OS which will update your motherboard BIOS, NIC firmware and switch firmware.
Drivers
After updating firmwares it’s time to take a look at the drivers for your device. Compare the installed driver version with the latest one available. If there is a newer driver available, it's also recommended you update it. Remember to check if your tools are compiled for the newest driver version, if not use a compatible driver version. It’s worth reading release notes to look for known issues and fixes in the driver version you have, which can be helpful in possible troubleshooting.
Tools
In network card testing, the use of DPDK (Data Plane Development Kit) tools can greatly enhance the performance and efficiency of the testing process. DPDK is a set of libraries and drivers that enable fast packet processing in the user space, bypassing the operating system kernel. When testing network cards using DPDK, several tools and frameworks can be utilized:
- packet generators like MoonGen or T-Rex,
- performance monitoring such as rte_power and rte_stats,
- packet capture and analysis tools like DPDK-pcap and Wireshark with DPDK support,
- benchmarking tools like Moonbench and pktgen-dpdk,
- the Testpmd tool, which allows packet generation, packet capturing and statistics collection.
By using DPDK tools you can take advantage of high-performance packet processing, control over traffic generation, performance monitoring and analysis.
Testing several network adapters
From my own experience, I strongly recommend using the same test environment if you want to compare test results between different NICs. Otherwise there will be too many variables impacting the results and you won't be able to compare them, or will need to repeat the previous tests in a new environment, which takes time.
Summary
Network interface card testing is a very broad topic and requires a lot of effort. In my opinion, this article focused on the most crucial aspects. As you can see, even preparing a test environment can take some time and requires knowledge of various areas, from networking to system administration. But it’s worth it. The effort put into preparing the environment will pay off in repeatable results, obtaining higher performance and stability, and if the environment is going to be used for other NICs, you will be able to compare results and choose the best network adapter.
As a reference for your setup you can follow the DPDK performance reports , where you can find reports for several NIC models from manufacturers such as Intel, NVIDIA/Mellanox, Broadcom.
