Skip to content

Add Multipath TCP (MPTCP) support for listeners, bridges, libmosquitto and clients - #3682

Open
moroznah wants to merge 4 commits into
eclipse-mosquitto:developfrom
moroznah:mptcp-support
Open

Add Multipath TCP (MPTCP) support for listeners, bridges, libmosquitto and clients#3682
moroznah wants to merge 4 commits into
eclipse-mosquitto:developfrom
moroznah:mptcp-support

Conversation

@moroznah

@moroznah moroznah commented Jul 4, 2026

Copy link
Copy Markdown

Add Multipath TCP (MPTCP) support


Summary

This PR adds opt-in Multipath TCP (RFC 8684) support across mosquitto, in four
logical commits:

Commit Area Change
1 lib New MOSQ_OPT_MPTCP for mosquitto_int_option(); shared net__socket_stream() helper
2 broker New per-listener config option mptcp true|false
3 clients New --mptcp flag for mosquitto_pub, mosquitto_sub, mosquitto_rr
4 broker New bridge config option bridge_mptcp true|false

MPTCP lets a single connection use several network paths at once (e.g. WiFi +
cellular, or two NICs), giving seamless failover between paths and potential
throughput aggregation without any change to the MQTT layer. This is
particularly useful for MQTT deployments on mobile/vehicular/IIoT clients with
redundant uplinks, and for broker-to-broker bridges over unreliable WANs.

Everything is opt-in and off by default. No behaviour changes unless the
new options are explicitly enabled. No new build dependencies.

Usage

Broker listener:

listener 1883
mptcp true

Bridge:

connection mybridge
address remote.example.com:1883
bridge_mptcp true

libmosquitto:

mosquitto_int_option(mosq, MOSQ_OPT_MPTCP, 1);

Clients:

mosquitto_sub --mptcp -h remote.example.com -t 'topic/#'

Platform support and fallback semantics

  • Linux-only, kernel 5.6+ (and net.mptcp.enabled=1). All code paths are
    guarded with #if defined(__linux__), so Windows/macOS/BSD builds are
    byte-for-byte unaffected at the socket layer.
  • On non-Linux platforms: MOSQ_OPT_MPTCP returns MOSQ_ERR_NOT_SUPPORTED;
    the mptcp/bridge_mptcp config options are accepted but ignored with a
    logged warning; the client --mptcp flag produces an error.
  • On Linux kernels without MPTCP (or with it disabled): socket creation with
    IPPROTO_MPTCP fails with EINVAL/EPROTONOSUPPORT/ENOPROTOOPT and we
    fall back to plain TCP. The broker detects the fallback (see design
    notes) and logs a warning; the library falls back silently.
  • An mptcp true listener remains fully compatible with plain-TCP clients:
    an MPTCP listening socket accepts both MPTCP and regular TCP connections,
    so enabling the option cannot lock out existing clients.

Design notes

Why substitute the protocol at socket() time instead of in getaddrinfo()
hints.
Passing IPPROTO_MPTCP via ai_protocol is rejected by many libc
versions (getaddrinfo validates the protocol against its own tables). The
established pattern — the same one used by curl and iperf3 — is to resolve
addresses normally and substitute IPPROTO_MPTCP when creating the socket,
falling back to the resolved ai_protocol if the kernel refuses. That is what
the new net__socket_stream(domain, type, protocol, use_mptcp) helper in
lib/net_mosq.c does; it is the single place holding the fallback-errno
logic, and is shared by the client connect path (both the synchronous path and
the WITH_ADNS async path in net__try_connect_step2()) and the broker's
listener setup.

Fallback vs. hard failure. A hard failure on kernels without MPTCP would
make --mptcp/mptcp true undeployable in mixed fleets and would break
brokers on kernel downgrades. Silent fallback alone would mask
misconfiguration on the broker side, so the compromise is: clients fall back
silently (documented), while the broker verifies the created listener socket
with getsockopt(SOL_SOCKET, SO_PROTOCOL) and logs
Warning: MPTCP is not supported by the kernel, falling back to TCP. if the
socket is not actually MPTCP. The check costs one getsockopt per listener at
startup, only when mptcp true is set.

IPPROTO_MPTCP availability at build time. glibc only exposes
IPPROTO_MPTCP from 2.32. Since the value (262) is part of the kernel ABI and
cannot change, net_mosq.h defines it when missing, guarded by
#if defined(__linux__) && !defined(IPPROTO_MPTCP). <netinet/in.h> is
included first so the #ifndef test is evaluated against the system
definition and never conflicts with it. This keeps the feature buildable on
older distros whose kernels do support MPTCP.

Internal API change. net__try_connect() gains a trailing
bool use_mptcp parameter. This is an internal symbol (not exported in
linker.version), so there is no ABI impact on libmosquitto. Its only two
callers are updated: net__socket_connect() passes mosq->mptcp, and the
bridge round-robin primary-retry probe in src/bridge.c passes
context->mptcp. Choosing a parameter over a global keeps the function usable
from both broker and lib contexts.

Bridge wiring. bridge_mptcp is stored on struct mosquitto__bridge and
copied onto the bridge's connection context in bridge__new()
(new_context->mptcp = bridge->mptcp). Because the broker's bridge contexts
reuse the lib's struct mosquitto and connect through net__socket_connect(),
both the normal connect path and the primary-retry path use MPTCP
consistently with no bridge-specific socket code.

Public option numbering/ABI. MOSQ_OPT_MPTCP = 18 extends
enum mosq_opt_t at the end; existing values are untouched. The mptcp flag
is a new bool in struct mosquitto (internal, not part of the public ABI),
placed alongside tcp_nodelay. Old applications running against the new
library see no behavioural difference.

Interaction with existing socket options. TCP_NODELAY,
SO_KEEPALIVE/TCP_KEEPIDLE etc. are supported by the kernel's MPTCP socket
option layer on recent kernels; on older kernels where a particular option is
not yet supported on MPTCP sockets, the existing code paths already only log
a warning on setsockopt() failure, so behaviour degrades gracefully.

Reload semantics. Like socket_domain, the listener mptcp option is
skipped on config reload, because listener sockets are not recreated on
SIGHUP. bridge_mptcp follows normal bridge option handling (bridges are
recreated on reload).

Scope deliberately excluded. No MPTCP-specific tuning is exposed (path
manager, subflow limits, MPTCP_INFO): these are administered system-wide via
ip mptcp / sysctls and would bloat the config surface. WebSockets listeners
are unaffected in this PR (the option applies to the TCP socket under any
listener type on the broker side, since it is applied at socket() creation
for the listener). TLS works unchanged on top of MPTCP since it is purely
above the transport.

Documentation

  • man/mosquitto.conf.5.xml: mptcp (listener section) and bridge_mptcp
    (bridge section) entries.
  • man/common/option-mptcp.xml + references from mosquitto_pub.1.xml,
    mosquitto_sub.1.xml, mosquitto_rr.1.xml.
  • include/mosquitto/libmosquitto_options.h: MOSQ_OPT_MPTCP API docs.
  • mosquitto.conf: commented examples for mptcp and bridge_mptcp.
  • Client --help output updated.

ChangeLog.txt intentionally not modified to avoid conflicts — happy to add
entries if preferred.

Testing performed

On Linux 6.18 (net.mptcp.enabled=1), Makefile build with
-Wall -Wextra -Wconversion, clean:

  1. Listener: broker with mptcp truess -Mln shows the listening
    socket as MPTCP.
  2. Client end-to-end: mosquitto_sub --mptcp + mosquitto_pub --mptcp
    strace confirms socket(AF_INET, SOCK_STREAM, IPPROTO_MPTCP), ss -Mn
    shows the established connection as MPTCP on both ends, message delivered.
  3. Interop: plain-TCP mosquitto_pub against the mptcp true listener —
    works unchanged.
  4. Kernel fallback: with net.mptcp.enabled=0, client --mptcp falls
    back (strace: IPPROTO_MPTCPENOPROTOOPTIPPROTO_TCP, publish
    succeeds); broker logs the fallback warning and serves TCP.
  5. Bridge: two brokers, B with mptcp true listener, A with
    bridge_mptcp truess -Mn shows the bridge connection as MPTCP;
    messages flow across the bridge.
  6. Config validation: mptcp notabool → proper
    Error: Invalid 'mptcp' value at startup; options on non-listener/bridge
    context rejected by the existing REQUIRE_* macros.

Checklist

  • Commits signed off (Signed-off-by, DCO), email matches Eclipse account
  • ECA signed
  • Based on and targeting develop
  • Four small, self-contained commits
  • Linux-only code guarded; other platforms unaffected
  • Man pages, header docs, example config updated

moroznah added 4 commits July 4, 2026 11:12
Add a new MOSQ_OPT_MPTCP integer option to mosquitto_int_option(). When
set to 1, libmosquitto creates its connection to the broker using
IPPROTO_MPTCP instead of IPPROTO_TCP, on Linux with kernel 5.6 or later.
If the running kernel does not support MPTCP, the connection silently
falls back to plain TCP. On non-Linux platforms the option returns
MOSQ_ERR_NOT_SUPPORTED.

This adds a net__socket_stream() helper that encapsulates the
MPTCP-or-fallback socket creation, so the broker can reuse it for
listeners.

Signed-off-by: Igor Sakulin <fxmoroz@gmail.com>
Add a new boolean listener option 'mptcp'. When set to true, the
listener socket is created with IPPROTO_MPTCP so that MPTCP-capable
clients can connect using Multipath TCP. Plain TCP clients are
unaffected and continue to work as normal.

This is only available on Linux with kernel 5.6 or later. If the
running kernel does not support MPTCP, the listener falls back to plain
TCP and a warning is logged. On non-Linux platforms the option is
ignored with a warning.

Like socket_domain, the option is not reloaded on reload signal since
listener sockets are not recreated.

Signed-off-by: Igor Sakulin <fxmoroz@gmail.com>
Add a --mptcp command line option to mosquitto_pub, mosquitto_sub and
mosquitto_rr which makes the clients connect to the broker using
Multipath TCP, via the new MOSQ_OPT_MPTCP library option. This is only
available on Linux with kernel 5.6 or later; if the running kernel does
not support MPTCP the connection falls back to plain TCP. On other
platforms the option produces an error.

Signed-off-by: Igor Sakulin <fxmoroz@gmail.com>
Add a new boolean bridge option 'bridge_mptcp'. When set to true, the
bridge connects to the remote broker using Multipath TCP via the
MOSQ_OPT_MPTCP support in the client code, including for round-robin
primary retry connections.

This is only available on Linux with kernel 5.6 or later. If the
running kernel does not support MPTCP, the connection falls back to
plain TCP. On non-Linux platforms the option is ignored with a
warning.

Signed-off-by: Igor Sakulin <fxmoroz@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant