Multi-peer remote-agent dispatch via Tailnet IPs + coordinator proxy (complement to hermes-agent#9295)

[Koraji95-coder]

Summary

PR NousResearch/hermes-agent#9295 lands gateway.tailscale_serve to expose a single machine's API server over a Tailnet — a clean fit for the "one machine, remote access" case. A different deployment shape benefits from a complementary pattern: one coordinator process dispatching to N peers, where each peer runs its own hermes-agent and the WebUI surfaces "this turn is talking to peer X". This issue describes that pattern in case it's useful as design input alongside #9295.

Filing in hermes-webui because the WebUI-visible surface is what differs most. The matching agent-side concerns live in a sibling issue at NousResearch/hermes-agent (link added at the bottom after both posts).

This issue is not a PR proposal — it documents a pattern observed working in production and asks whether any of the supporting concerns (per-message peer indicator, per-profile MCP registry, SSE profile identity) belong on the WebUI roadmap.

Pattern

   client app                    coordinator                    peer machines
   (composer,         POST       (small service)
   consumer apps)  -------->     - peerMap: { name -> (ip, port, basicAuthUser, basicAuthPass) }
                                 - lane routing
                                 - per-message peer override
                                 - response stream         HTTP    +-----------------+
                                       |                  ---->    | peer A          |
                                       |                  Basic    |  auth proxy     |
                                       |                  Auth     |   |             |
                                       |                           |   v             |
                                       |                           | hermes-agent    |
                                       |                           | API @ 127.0.0.1 |
                                       |                           +-----------------+
                                       |
                                       +-------- ...               +-----------------+
                                                ---->               | peer B          |
                                                                    | (same shape)    |
                                                                    +-----------------+

• Each peer's hermes-agent stays at its safe loopback default (no --host 0.0.0.0).
• A small ~300-line stdlib Python reverse proxy on each peer binds to that peer's Tailnet interface IP only (not 0.0.0.0), terminates HTTP Basic from the coordinator, and injects Authorization: Bearer ${API_SERVER_KEY} for /v1/* requests before forwarding to 127.0.0.1:8642. Dashboard routes (/) pass through to 127.0.0.1:9119 without Bearer.
• The coordinator holds a peerMap keyed by logical name, calls each peer with POST /v1/chat/completions over Basic Auth, streams the response back. One client request can fan out to multiple peers in parallel.

How this differs from #9295

Axis	#9295 (tailscale_serve)	This pattern
Peers per deployment	1 (the machine running hermes-agent)	N
Exposure mechanism	Tailscale Serve managed proxy + HTTPS	Direct Tailnet IP bind + small stdlib proxy + Basic Auth
Auth	API_SERVER_KEY Bearer only	Outer Basic per-peer + inner Bearer auto-injected
Routing source	None (single backend)	Coordinator decides which peer per message
WebUI surface	/v1/* becomes reachable	Composer needs a "current peer" indicator and per-message peer override

PR #9295 is the right answer for the "expose this one machine over Tailscale" case. This pattern is the right answer when the same operator wants one composer with a peer picker that can target several machines from a single client.

WebUI-side concerns

A few WebUI rough edges become visible when running this pattern. None are blockers, but they would smooth multi-peer use whether the coordinator is a custom service (like ours) or whatever shape the upstream multi-machine story eventually takes:

1. No "current peer" indicator on the composer. When a session runs through a coordinator that can target different peers, users want a visible chip near the model chip that shows which peer the next turn will go to. Today there is nothing to bind to — the composer treats the request as "local to this WebUI."

2. Profile identity is missing from the session-events SSE payload (Session-events SSE bus has no profile identity (follow-up to #2637) #2660). Each peer's local WebUI publishes sessions_changed to its own SSE bus, but the payload has no profile identity. A coordinator that subscribes to multiple peers' SSE streams can't route the event to the right tab without re-fetching.

3. MCP server registry is process-global per tracking: profile-keyed MCP server registry — agent-side follow-up to #1968 #1977 / agent-side root cause. When a peer runs concurrent profile sessions, MCP servers register by name only. Cross-peer this is fine (separate processes), but cross-profile on a single peer is the existing layer-2 problem.

4. Profile vs. peer-target axis confusion. Per the runtime-model issue at design(profile): align Web UI profile management with Hermes runtime model #749 (Profile is a heavy bundle, not a routing override), our pattern adds a fifth axis: (peer, profile-on-peer). The composer currently has no place to express "Profile A on peer X"; it can express only "this WebUI's currently-active profile."

Of these, #2 (SSE profile identity) is the one most directly addressable in this repo and is already filed at #2660. The others might just be useful design context for the broader multi-machine story.

Verification (honest)

The pattern is wired end-to-end on a coordinator host. As of this filing, both target peers in the local Tailnet are showing offline (Tailscale reports relay "dfw"; offline, last seen 8h ago for one peer and 1d ago for the other), so a fresh live transcript is not available at the moment. A successful round-trip transcript from earlier this week is preserved in the coordinator's tools/remote-agent/README.md:

2026-05-23: POST /api/remote-agent/invoke (target: peer-A) returned 200 with "pong" in 33s

The shape that produced it on the wire (Basic-only from coordinator; proxy injects Bearer for /v1/* upstream):

# from the coordinator host, against the peer's Tailnet IP
curl -u coordinator:<32-char-password> \
     http://<peer-tailscale-ip>:9121/v1/models
# 200 OK -> the proxy is up AND the upstream API server has API_SERVER_KEY set

# the actual round-trip (Chat Completions)
curl -u coordinator:<32-char-password> \
     -X POST http://<peer-tailscale-ip>:9121/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d "{\"model\":\"<profile-default>\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"stream\":false}"
# 200 OK -> assistant reply streams back

Happy to attach a fresh transcript with timing data once a peer is back online — likely within 24h. (Down due to Home renovations)

Why I'm not proposing a PR here

The WebUI changes that would be most useful (per-message peer indicator, peer-aware composer chip, peer override field) are speculative without knowing where the multi-machine story is headed. The maintainer-authored issues (#749, #1977, #2660, plus the agent-side #10567 and #15731) suggest the underlying problem space is well-mapped; a multi-peer coordinator pattern just exercises that space in a particular shape.

Filing as enhancement + ux for visibility. If this isn't a direction the WebUI wants to take, no action needed on this side — the pattern works fine as a coordinator-only convention. If it IS useful design input, the composer-side (machine, capability) picker shape that's currently working as a local patch might be worth sharing as a reference.

If this direction is something the WebUI wants to bring upstream, happy to take it on and work with maintainers on the implementation -- the composer-side (machine, capability) picker, peer-aware chip, and SSE profile identity refs are all in scope. The existing local pattern is a starting point; the upstream shape should match the WebUI's direction (refs #749 / #1977 / #2660), not the coordinator's particular choices.

Related

• feat: Tailscale serve integration for secure remote access NousResearch/hermes-agent#9295 — single-machine Tailscale Serve PR (this issue's companion)
• feat: Tailscale serve integration for secure remote access NousResearch/hermes-agent#9269 — parent issue PR #9295 closes
• Add --host and CORS config for hermes dashboard to enable Tailscale/VPN access NousResearch/hermes-agent#10567 — --host + CORS for dashboard (Tailscale/VPN)
• [Feature]: Make chat tab in web dashboard work with --host set to non-localhost address NousResearch/hermes-agent#15731 — dashboard chat tab + --host non-localhost
• design(profile): align Web UI profile management with Hermes runtime model #749 — Profile runtime model alignment
• tracking: profile-keyed MCP server registry — agent-side follow-up to #1968 #1977 — Profile-keyed MCP server registry (layer-2)
• Session-events SSE bus has no profile identity (follow-up to #2637) #2660 — Session-events SSE bus has no profile identity
• Companion issue: Multi-peer agent dispatch via proxy + Tailnet IPs as a complement to #9295 NousResearch/hermes-agent#32397

AI Usage Disclosure

This issue was drafted with Claude (Opus 4.7) assistance and reviewed by a human contributor before posting. Architecture description, smoke-test transcript, and cross-issue links were verified by the human reviewer. The pattern described is in production use on the contributor's coordinator deployment.

Edits: cross-link to companion issue NousResearch/hermes-agent#32397 added; explicit offer to contribute the implementation upstream if maintainers want this direction added. No claim changes.

[nesquena-hermes]

Summary

Thanks for the careful framing and for keeping this as a design-input issue rather than a PR proposal. The pattern you're describing is real and complements #9295 (Tailscale Serve) rather than competing with it — single-machine exposure vs. coordinator-fanout-to-N-peers are genuinely different shapes.

A few notes on the WebUI-side concerns you listed, in priority order.

Most directly addressable here

(2) Profile identity is missing from session-events SSE payload (#2660). Yes — and this is the one that doesn't depend on any broader multi-machine decision, so it can move independently. The fix is local to the SSE emit path in api/streaming.py and the subscriber code in static/sse.js. If a contributor wanted to take it on, that's the cleanest extract.

Probably worth waiting on broader decisions

(1) Per-message peer indicator on the composer. I see the use case, but the composer is already busy (model chip, profile chip, attached files, slash commands, draft state). Adding a peer chip is fine conceptually, but only if it's tied to a runtime contract the WebUI actually owns. Today the WebUI thinks "this turn goes to my hermes-agent over IPC" — a peer chip implies the runtime layer has been generalized to "this turn goes to runtime X." That generalization is exactly what the runtime-adapter RFC at #1925 (and the recent Slice 4f gate work in PR #2972, v0.51.138) is trying to land. Until the runtime-adapter contract has a concrete shape for "named remote runtime," a peer chip in the composer would be UI scaffolding without a backing contract.

(4) Profile vs. peer-target axis confusion. Similar shape — this is a runtime-layer concern, not a WebUI-composer concern. The composer can express what the runtime layer represents; if the runtime layer doesn't have (peer, profile) as a first-class concept, the composer can't either.

The (3) MCP angle

The process-global MCP registry issue is filed at #1977 and the layer-2 fix is independent of this multi-peer pattern. Cross-peer is already fine (separate processes); cross-profile-same-peer is the existing problem.

Diagnosis of the broader question

The pattern you're describing is solid and the production smoke-test (May 23 transcript, even if peers are currently offline) is a useful design data point. Where I'd push back is on the implementation shape: a coordinator service sitting between client apps and hermes-agent peers is one valid pattern, but it's not the only one — the alternative is having the WebUI itself be the coordinator, with the runtime-adapter layer routing turns to remote agents. Those two pictures look identical to the user but have very different code shapes and reliability properties.

I'd suggest the WebUI roadmap stay focused on:

• Land the runtime-adapter contract so "remote runtime" is a first-class concept (RFC: Make WebUI a thin observability/control client over Hermes Agent runtime #1925 ongoing).
• Fix Session-events SSE bus has no profile identity (follow-up to #2637) #2660 (SSE profile identity) so multi-tab/multi-process subscribers can route events correctly.
• Defer per-message peer chip / peer override field until the runtime-adapter shape names whether routing happens in the WebUI or in a separate coordinator.

If the runtime-adapter direction settles on "WebUI doesn't route — coordinator does," then the WebUI-side surface here is much smaller (basically just (2) plus a generic "current runtime" indicator that could be filled by any backend). If it settles on "WebUI does route," then (1) and (4) become first-class.

Filing as enhancement + gateway

Labeling enhancement + gateway + Backlog - Product Review since this is genuinely design input and not a buildable shape right now. Worth revisiting once #1925 / #2972's runtime-adapter Slice 4f lands, at which point the picture for "where does peer-targeting belong" should be clearer.

Thanks again for the framing and the offer to contribute the implementation if the WebUI takes this direction. The offer is noted and appreciated.