feat: switch to QUIC multipath #3381

dignifiedquire · 2025-07-07T11:02:25Z

Work on integrating n0-computer/quinn#28 into the iroh magic

TODOs

Merge feat: add n0's version of quic nat hole punching quinn#177
Green CI

github-actions · 2025-07-07T13:29:30Z

Documentation for this PR has been generated and is available at: https://n0-computer.github.io/iroh/pr/3381/docs/iroh/

Last updated: 2025-12-05T14:47:54Z

github-actions · 2025-07-07T13:37:37Z

Netsim report & logs for this PR have been generated and is available at: LOGS
This report will remain available for 3 days.

Last updated for commit: 66a8565

…3664) ## Description This means these tests also work when nextest run locally. ## Breaking Changes  ## Notes & open questions I'm not sure why this wasn't done when the ci profile override was chosen. What am I missing? ## Change checklist  - [x] Self-review.

## Description Bumps netwatch and netdev, to remove duplicate dependency on both [email protected] and [email protected]. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

This is a next step into the world of configurable transports. We now allow disabling the IP based transports entirely. Internally this starts to prepare for a world where the user can configure multiple different transports, IP, relay and others in the future. Closes #2957

## Description Remove the test-only `Endpoint::path_selection` API and instead use `Endpoint::clear_ip_transports` for `PathSelection::RelayOnly `, now that this public API was added in #3651. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

…moteState (#3673) ## Description Renames: * renamed `endpoint_map` -> `remote_map`, `EndpointMap` -> `RemoteMap`, `endpoint_state` -> `remote_state`, `EndpointStateActor` -> `RemoteStateActor` Moved: * moved `path_state` module under `remote_state` (prev `endpoint_state`), its items are used only there and nowhere else ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Floris Bruynooghe <[email protected]>

## Description Merges main and adapts for the changes from #3619 ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Rüdiger Klaehn <[email protected]> Co-authored-by: Friedel Ziegelmayer <[email protected]>

## Description Avoid potentially busy looping in a tokio task. I think this blocking leads to tokio not being able to close the runtime properly.

## Description Fixes #3642 This moves discovery handling fully into the `EndpointStateActor`. The pub(crate) interface to trigger discovery and get a EndpointMappedAddr is now `Magicsock::resolve_remote`, which sends the provided addresses to the EndpointStateActor. The actor starts discovery if it does not have a selected path and if discovery is not running. It returns either immediately if there are any known paths, or waits for discovery to produce at least one result or an error. Once this returns, `resolve_remote` returns either with a EndpointMappedAddr or with the discovery error. This means the current behavior is kept: We only start `quinn::Endpoint::connect` once we have at least one transport address for the remote. If not, we return the discovery error immediately from `iroh::Endpoint::connect`. This opens the door for us to easily tune when to run discovery in other siutations, e.g. when all available paths to a remote are closed. However, for now this PR still only starts discovery when `Endpoint::connect` is called and no path is selected at the moment. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description * fix idle timeout clear condition (previously it would hot loop) * fix hot loop when local_addrs watchable becomes disconnected during shutdown * when sending a datagram fails in the transports sender, include the dst address in the error message * do not break the RemoteStateActor when sending a datagram fails ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: Philipp Krüger <[email protected]>

## Description This reverts a change from this PR: #3384 I originally thought I could make this test more reliable by pausing the tokio time across the `tokio::time::timeout` calls, but it turns out that actually makes the test *more* flaky: - When time is paused, the timeout will immediately fire once the tokio runtime has no more CPU work to do. - It's possible that there's no CPU work to do anymore, while there's something else that is actually still doing work, e.g. networking. - Before the `ActiveRelayActor` finishes its `run_connected` loop, it will call `client_sink.close().await`, which will do actual I/O. When the tokio runtime is paused at that moment, it'll immediately trigger the test's timeout. ## Notes & open questions I couldn't reproduce this problem even across a couple thousand runs of the test locally. I'm not super confident that this fixes things, but I've analyzed the logs and this seems to be the most likely thing that's happening to me. Closes #3613 ## Change checklist  - [x] Self-review.

merge main

## Description This switches from the old DISCO to the so-new-it-doesnt-exit-yet QUIC NAT Traversal. ## Breaking Changes Nothing visible? Maybe? ## Notes & open questions The QUIC NAT Traversal API doesn't exist yet, so this won't even build on any machine that's not mine. I've locally patched in the dummy methods that I use. --------- Co-authored-by: dignifiedquire <[email protected]> Co-authored-by: Frando <[email protected]>

## Description This adds the conecpt of hooks to the iroh endpoint. `Hooks` are structs implementing the `EndpointHooks` trait and are used to intercept the establishment of connections. Multiple hooks can be added to the endpoint, and they will be invoked for each hook in the order they have been added to the endpoint. Currently there's two methods on the `EndpointHooks` trait: * `before_connect` is invoked before an outgoing connection is started. * `after_handshake` is invoked for incoming and outgoing connections once the TLS handshake has completed Both methods return an `Outcome`, which can either be `Reject` or `Accept`. If any hook returns `Reject`, the connection or connection attempt will be rejected. The PR also adds `ConnectionInfo`, which is a struct that has information about a connection, but does not keep the connection itself alive. It allows to inspect stats and paths, and there's a `closed` method that returns a future which completes once the connection closes (without keeping the connection alive). The PR includes two examples: * `auth-hook` implements authentication for iroh protocols through a middleware and a separate authentication protocol. Individual protocols don't need to be aware of authentication at all. * `monitor-connnections` monitors incoming and outgoing connections and prints connection stats once a connection closes. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme) --------- Co-authored-by: dignifiedquire <[email protected]> Co-authored-by: ramfox <[email protected]>

removes the need for two `send` impls

Closes #3641

## Description Since the server was actively closing the connection it is possible that the client would not have read the response yet by the time the connection is closed. ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description Based on #3593 This was always just a placeholder, and can now be collected using `EndpointHooks`. ## Breaking Changes - remove `Endpoint::latency` --------- Co-authored-by: varun-doshi <[email protected]>

## Description Bumps quinn to latest `main-iroh` and netwatch/portmapper to n0-computer/net-tools#72 (the latter is needed because the quinn-udp version changed to 0.6 on `main-iroh`). ## Breaking Changes  ## Notes & open questions  ## Change checklist  - [ ] Self-review. - [ ] Documentation updates following the [style guide](https://rust-lang.github.io/rfcs/1574-more-api-documentation-conventions.html#appendix-a-full-conventions-text), if relevant. - [ ] Tests if relevant. - [ ] All breaking changes documented. - [ ] List all breaking changes in the above "Breaking Changes" section. - [ ] Open an issue or PR on any number0 repos that are affected by this breaking change. Give guidance on how the updates should be handled or do the actual updates themselves. The major ones are: - [ ] [`quic-rpc`](https://github.com/n0-computer/quic-rpc) - [ ] [`iroh-gossip`](https://github.com/n0-computer/iroh-gossip) - [ ] [`iroh-blobs`](https://github.com/n0-computer/iroh-blobs) - [ ] [`dumbpipe`](https://github.com/n0-computer/dumbpipe) - [ ] [`sendme`](https://github.com/n0-computer/sendme)

## Description Whenever we insert a new path, trigger pruning paths. We currently only prune IP paths, and pruning paths only occurs if we have more than 30 IP paths. We will prune any paths that did not successfully holepunch. If there are still over 30 IP paths left, then we order the "inactive" paths (paths that have been closed, but at one point holepunched), and prune the paths that were closed earliest. ## Notes and Questions - Added constants: - `MAX_IP_PATHS` = 30 - maximum IP paths per endpoint - `MAX_INACTIVE_IP_PATHS` = 10 - maximum inactive IP paths to keep - New `PathState` field: - `status` - tracks the `PathStatus` of the path - New `PathStatus` enum: - `PathStatus::Open` - is an open path - `PathStatus::Inactive(Instant)` - was opened once, but currently inactive - `PathStatus::Unusable` - we attempted to use it, but it never connected - `PathStatus::Unknown` - we don't know the status yet - New methods on `RemotePathState`: - `abandoned_path` - marks a path as abandoned with timestamp, triggered when we get the `PathEvent::Abandoned` event - `prune_paths` - triggers path pruning, occurs whenever we insert a path to the `RemotePathState` - changed `insert` to `insert_open_path` - New `prune_ip_paths` function with all the prune logic: - Only prunes if IP paths exceed `MAX_IP_PATHS` - Never prunes active paths or paths of unknown status - Always prunes failed holepunch attempts (PathStatus::Unusable) - Keeps 10 most recently inactive paths that were previously successful - Special case: if all paths failed, keeps `MAX_IP_PATHS` instead of pruning everything - Added tests for edge cases and the typical case

…3724)

…ain minimums when used with multipath (#3721) ## Description This PR encapsulates the `quinn::TransportConfig` in a new struct called `QuicTransportConfig`. It has all of the same methods as the `quinn::TransportConfig`, but the follow methods will log warnings if the user given values make iroh + multipath sub-optimal: - default_path_keep_alive_interval() - should be at most `HEARTBEAT_INTERVAL`ms - default_path_max_idle_timeout() - should be at most `PATH_MAX_IDLE_TIMEOUT`ms - max_concurrent_multipath_paths() - should be at least `MAX_MULTIPATH + 1` - set_max_remote_nat_traversal_addresses() - should be at least `MAX_MULTIPATH` These values are also set properly by default when creating a `QuicTransportConfig`. ## quinn encapsulation/co-location Created a new mod `quic` and co-located all the quinn exports there. This would be the mod where we do any future encapsulation. Note: the `StaticConfig` struct is a private struct that still uses `quinn::TransportConfig` directly. closes #3635 ## Breaking Changes - `iroh` - changes - `QuinnTransportConfig` renamed to `QuicTransportConfig` & is now `Clone` - `ConnectOptions::transport_config: Option<Arc<quinn::TransportConfig>>` -> `ConnectOptions::transport_config: Option<QuicTransportConfig> - `ConnectOptions::with_transport_config(mut self, transport_config: Arc<quinn::TransportConfig>)` -> `ConnectOptions::with_transport_config(mut self, transport_config: QuicTransportConfig) ## Open Questions - [x] Need verification that my assumptions about "at least" and "at most" for the above are the correct assumptions about what we want. **done in #3635**

## Description Adds a missing re-export for `quinn::Side`

n0bot bot added this to iroh Jul 7, 2025

github-project-automation bot moved this to 🏗 In progress in iroh Jul 7, 2025

dignifiedquire changed the title ~~[WIP] feat: use quinn multipath~~ [WIP] feat: use quic multipath Jul 8, 2025

dignifiedquire force-pushed the feat-multipath branch 3 times, most recently from 72cb071 to db712c0 Compare July 18, 2025 14:39

dignifiedquire force-pushed the feat-multipath branch 2 times, most recently from 4827e62 to 946f71c Compare July 28, 2025 20:29

dignifiedquire added 20 commits August 1, 2025 11:13

feat: use quinn multipath

c17a870

update iroh-quinn

7fe570d

start opening paths

2f469ac

add more paths

870716f

set keep alive and idle timeouts for new paths

346a7c2

insert relay path

68b1769

set relay path as backup

0eb3fde

start removing ping logic from the node_map

79ec17f

start tracking path events

c4baca8

start figuring out more details

549adee

wip

9ef5765

get some stuff to work again

75d5525

remove ip_mapped_addresses

4f78898

use correct relay addr on recv

130710f

ensure connection registration

dba89df

remove rtt_actor, this is now done inside quinn on a per path basis

aab083d

open additional paths after the initial connection

d4484da

ensure path open

6cb94e4

some debugging

998e283

update quinn branch

99cee61

flub and others added 30 commits November 14, 2025 13:44

fix test by calling stream.finish() (#3665)

1f2db9f

Merge remote-tracking branch 'origin/main' into feat-multipath

7052392

test(iroh): Fix test_two_devices_setup_teardown hanging (#3675)

59a7e85

## Description Avoid potentially busy looping in a tokio task. I think this blocking leads to tokio not being able to close the runtime properly.

Merge remote-tracking branch 'origin/main' into feat-multipath-merge

c85bbbd

test: mark test_active_relay_inactive as non flaky

e620b5e

Merge pull request #3696 from n0-computer/feat-multipath-merge

d243702

merge main

chore: fixup wasm test

a28d1fb

refactor(iroh): simplify internal transports sending (#3708)

8d56889

removes the need for two `send` impls

refactor(iroh): avoid storing the TransportsSender (#3712)

3f4d365

Closes #3641

chore: update to quinn@main-iroh (#3716)

e22c001

refactor(iroh)!: remove Endpoint::latency (#3717)

783e2ef

## Description Based on #3593 This was always just a placeholder, and can now be collected using `EndpointHooks`. ## Breaking Changes - remove `Endpoint::latency` --------- Co-authored-by: varun-doshi <[email protected]>

chore: update iroh-quinn (#3718)

2cf93a5

feat(iroh): add fast-apple-datapath feature and enable it by default (#…

a5f4a08

…3724)

fix: update to main net-tools (#3726)

1d6e453

refactor: add Side re-export (#3739)

136ff04

## Description Adds a missing re-export for `quinn::Side`

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: switch to QUIC multipath #3381

feat: switch to QUIC multipath #3381

Uh oh!

dignifiedquire commented Jul 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 7, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 7, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

feat: switch to QUIC multipath #3381

Are you sure you want to change the base?

feat: switch to QUIC multipath #3381

Uh oh!

Conversation

dignifiedquire commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODOs

Uh oh!

github-actions bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

dignifiedquire commented Jul 7, 2025 •

edited

Loading

github-actions bot commented Jul 7, 2025 •

edited

Loading

github-actions bot commented Jul 7, 2025 •

edited

Loading