Cutting MQTT Publish distribution time by moving the broker’s heaviest work onto a DPU

The challenge

An MQTT broker’s heaviest work is processing and distributing Publish packets, and its CPU and memory use grow with every extra client, topic and distribution. Moving part of that work onto a SmartNIC means the offload has to act as an active TCP endpoint, generating and processing packets on the broker’s behalf. TCP assumes two endpoints: once the offload sends a single packet, the broker’s and the client’s views of the connection fall out of sync and the connection collapses.

What was at risk:

• Publish messages reaching subscribers too late in time-critical applications, where connections are short-lived or the reaction to telemetry has to be immediate
• Broker host CPU and memory consumed by message fan-out as clients, topics and subscriptions grow
• Dependence on fixed-function TCP offload engines, which the kernel community resists over security, updateability, hardware resource limits and flexibility

What we did

• Built the protocol offload as a DOCA/DPDK application running on the Arm cores of an NVIDIA BlueField-2 DPU.
• Captured MQTT Publish packets from publishers in the offload, matched their topics against a subscriber-topic map and distributed them to subscribers directly, passing all other traffic to the broker.
• Kept the subscriber-topic map current through a gRPC server that the broker notifies of every new subscription.
• Solved TCP SEQ/ACK desynchronization with a translation mechanism that splits each connection into two synchronization domains.
• Integrated the lwIP network stack, so retransmissions, delayed ACKs, Nagle’s algorithm, keep-alive and congestion control run on RFC-compliant mechanisms.
• Bypassed the host TCP/IP stack through socket interception: a Rust libc shim forwards the broker’s socket calls to the DPU over DOCA Comm Channel, with no change to the broker’s source code.
• Measured Publish distribution time and host CPU load against a broker that talks to its clients directly.

How it works

The offload went through three iterations, each removing a limit found in the one before, and was measured at every step.

1. TCP monitoring The offload sits on the path between the broker and the clients and operates on their existing connections. For each connection it keeps two endpoint peer structures that track how each side sees the other’s SEQ and ACK numbers, and rewrites the packets passing between broker and client so both sides accept them. This minimal design already showed a significant reduction in Publish distribution time. It also showed where it broke: under load, coalesced payloads (Nagle’s algorithm) and retransmissions after ~250 ms without an ACK broke the tracking.
2. Network stack integration The offload became an independent TCP endpoint with its own IP address. Clients connect to it, and for each connection it opens a peer connection to the broker on the same port. The lwIP stack, designed for embedded systems and easy to adapt to DPDK APIs, takes care of TCP correctness, so the offload processes a contiguous data stream, message by message. In tests it outperformed TCP monitoring and held up as publishers and subscribers were added, while the reference broker-client setup, left at its default configuration, scaled poorly.
3. Socket interception An extra stack on the path adds latency for the traffic the broker itself handles. To take the host kernel stack out of the path, a libc shim written in Rust intercepts the broker’s socket calls and forwards the relevant ones to the offload over DOCA Comm Channel, with messages encoded in Cap’n Proto. The broker runs unmodified on top of the DPU’s network stack.
4. Message handling on the DPU To keep latency under control, the MQTT processing block continuously polls the lwIP socket that represents the broker. Publish packets are matched and distributed on the DPU. Packets meant for the broker go into a queue, and every recv() call from the host is served from it.
5. Measurement JMeter with its MQTT plugin spawns publishers and subscribers at a set packet rate and measures the time from issuing a Publish packet to its arrival at the last subscriber, compared with a broker and clients communicating directly.

Compatibility is part of the work. An application written against libc expects libc behaviour, and lwIP’s ICMP implementation differs slightly from it, which surfaced when a simple ping was forwarded to the DPU during integration.