Flakes in CI: Full Mesh All Externals, "running iperf3 server: Process exited with status 1"

[qmonnet]

While running the VLAB CI from the https://github.com/githedgehog/dataplane repository, I got a failure on vlab-sl-gw-iso-l2vni. I see the following messages in the logs:

2025-09-01T11:20:26.9581280Z 11:20:26 DBG IPerf3 result from=server-05 to=server-07 sendSpeed="22.36 Mbps" receiveSpeed="21.66 Mbps" sent="8.39 MB" received="8.13 MB"
2025-09-01T11:20:26.9581656Z 11:20:26 ERR Error(s) during testing connectivity
2025-09-01T11:20:26.9582072Z 11:20:26 ERR Error key=vpcpeer--server-08--server-07 err="checking iperf from server-08 to server-07: running iperf3: running iperf3 server: Process exited with status 1"
2025-09-01T11:20:26.9586938Z 11:20:26 ERR FAIL test="Full Mesh All Externals" error="testing connectivity: testing connectivity: testing connectivity from \"server-08\" to \"server-07\": checking iperf from server-08 to server-07: running iperf3: running iperf3 server: Process exited with status 1"

and

2025-09-01T11:25:38.6032032Z 11:25:38 INF Test suite results suite="Basic VPC Peering Suite"
2025-09-01T11:25:38.6034039Z 11:25:38 WRN SKIP test="Starter Test" reason="The named external (default) is not present"
2025-09-01T11:25:38.6034465Z 11:25:38 WRN SKIP test="Only Externals" reason="There are no externals"
2025-09-01T11:25:38.6035395Z 11:25:38 ERR FAIL test="Full Mesh All Externals" error="testing connectivity: testing connectivity: testing connectivity from \"server-08\" to \"server-07\": checking iperf from server-08 to server-07: running iperf3: running iperf3 server: Process exited with status 1"
2025-09-01T11:25:38.6036257Z 11:25:38 INF PASS test="Full Loop All Externals"
2025-09-01T11:25:38.6036619Z 11:25:38 WRN SKIP test="Sergei's Special Test" reason="The named external (default) is not present"
2025-09-01T11:25:38.6041057Z 11:25:38 INF Test suite summary tests=5 passed=1 skipped=3 failed=1 duration=16m13s
2025-09-01T11:25:38.6041469Z 11:25:38 INF *** Recap of the test results ***
2025-09-01T11:25:38.6041778Z 11:25:38 INF Test suite results suite="No VPCs Suite"
2025-09-01T11:25:38.6042134Z 11:25:38 WRN SKIP test="Breakout ports" reason="There are virtual switches"
2025-09-01T11:25:38.6042562Z 11:25:38 INF Test suite summary tests=1 passed=0 skipped=1 failed=0 duration=0s
2025-09-01T11:25:38.6042944Z 11:25:38 INF Test suite results suite="Single VPC Suite"
2025-09-01T11:25:38.6043248Z 11:25:38 INF PASS test="No restrictions"
2025-09-01T11:25:38.6044138Z 11:25:38 WRN SKIP test="Single VPC with restrictions" reason="There are virtual switches"
2025-09-01T11:25:38.6044503Z 11:25:38 INF PASS test="DNS/NTP/MTU/DHCP lease"
2025-09-01T11:25:38.6044824Z 11:25:38 WRN SKIP test="MCLAG Failover" reason="There are virtual switches"
2025-09-01T11:25:38.6045198Z 11:25:38 WRN SKIP test="ESLAG Failover" reason="There are virtual switches"
2025-09-01T11:25:38.6045583Z 11:25:38 WRN SKIP test="Bundled Failover" reason="There are virtual switches"
2025-09-01T11:25:38.6045962Z 11:25:38 WRN SKIP test="Spine Failover" reason="There are virtual switches"
2025-09-01T11:25:38.6046370Z 11:25:38 WRN SKIP test="Mesh Failover" reason="There are virtual switches"
2025-09-01T11:25:38.6046906Z 11:25:38 WRN SKIP test="RoCE flag and basic traffic marking" reason="There are no switches that support RoCE"
2025-09-01T11:25:38.6047638Z 11:25:38 INF Test suite summary tests=9 passed=2 skipped=7 failed=0 duration=8m52s
2025-09-01T11:25:38.6048074Z 11:25:38 INF Test suite results suite="Multi-Subnets VPC Suite"
2025-09-01T11:25:38.6048375Z 11:25:38 INF PASS test="Multi-Subnets no restrictions"
2025-09-01T11:25:38.6048771Z 11:25:38 WRN SKIP test="Multi-Subnets isolation" reason="There are virtual switches"
2025-09-01T11:25:38.6049175Z 11:25:38 WRN SKIP test="Multi-Subnets with filtering" reason="There are virtual switches"
2025-09-01T11:25:38.6049514Z 11:25:38 WRN SKIP test=StaticExternal reason="There are virtual switches"
2025-09-01T11:25:38.6050036Z 11:25:38 INF Test suite summary tests=4 passed=1 skipped=3 failed=0 duration=3m5s
2025-09-01T11:25:38.6050433Z 11:25:38 INF Test suite results suite="Basic VPC Peering Suite"
2025-09-01T11:25:38.6050905Z 11:25:38 WRN SKIP test="Starter Test" reason="The named external (default) is not present"
2025-09-01T11:25:38.6051342Z 11:25:38 WRN SKIP test="Only Externals" reason="There are no externals"
2025-09-01T11:25:38.6052383Z 11:25:38 ERR FAIL test="Full Mesh All Externals" error="testing connectivity: testing connectivity: testing connectivity from \"server-08\" to \"server-07\": checking iperf from server-08 to server-07: running iperf3: running iperf3 server: Process exited with status 1"

The test passed on the second attempt, and the failure seems unlikely to be related to the PR from which the workflow ran.

• URL: https://github.com/githedgehog/dataplane/actions/runs/17374698339/job/49319123286 (from bump(cargo)!: 🚀 upgrades available dataplane#807)
• Logs archive: logs_44559228475.zip

[pau-hedgehog]

Another hit, now trying to launch iperf3 client instead and in Multi-Subnets no restrictions test:

22:48:43 ERR FAIL test="Multi-Subnets no restrictions" error="testing connectivity: testing connectivity: testing connectivity from \"server-03\" to \"server-02\": checking iperf from server-03 to server-02: running iperf3: running iperf3 client: Process exited with status 1"

https://github.com/githedgehog/fabricator/actions/runs/17387878887/job/49357393757

[pau-hedgehog]

A hit in master nightly pipeline:
https://github.com/githedgehog/fabricator/actions/runs/17430038308/job/49486607418

[pau-hedgehog]

Seen a strange occurrence of this issue where the speed check actually passed but server exited with 1:

https://github.com/githedgehog/fabricator/actions/runs/17702802470/job/50310671723

22:52:06 ERR Test connectivity failed ping=0 iperf=1 curl=0 took=1m31.013413891s errors="iperf server-01 -> server-02: running iperf3 server: Process exited with status 1 (sent 12.35 Mbps, rcvd 12.35 Mbps, min expected 0.00 Mbps)"
22:52:06 WRN Failed to test connectivity err="iperf server-01 -> server-02: running iperf3 server: Process exited with status 1 (sent 12.35 Mbps, rcvd 12.35 Mbps, min expected 0.00 Mbps)"

This is the one that I was mentioning, @edipascale

[edipascale]

Seen a strange occurrence of this issue where the speed check actually passed but server exited with 1:

https://github.com/githedgehog/fabricator/actions/runs/17702802470/job/50310671723

22:52:06 ERR Test connectivity failed ping=0 iperf=1 curl=0 took=1m31.013413891s errors="iperf server-01 -> server-02: running iperf3 server: Process exited with status 1 (sent 12.35 Mbps, rcvd 12.35 Mbps, min expected 0.00 Mbps)"
22:52:06 WRN Failed to test connectivity err="iperf server-01 -> server-02: running iperf3 server: Process exited with status 1 (sent 12.35 Mbps, rcvd 12.35 Mbps, min expected 0.00 Mbps)"

This is the one that I was mentioning, @edipascale

OK, this is the server erroring out with exit status 1, even though the client completed successfully and updated the speed measurements on the error that we return. It'd be nice to have the error message of the server, maybe it is worth doing some ugly hack to workaround the json decoding issue described here: #912 (comment)

[edipascale]

This particular iperf issue should be resolved now with LD_PRELOAD fix of #971. Unfortunately there seems to be still at least another iperf3 flaky issue (#959) but I'll close this as completed