docs: formal spec for remote agents and their management

[tlongwell-block]

What

A formal specification for remote agents and their management — docs/remote-agents.md — in the style of docs/git-on-object-storage.md: stated system model, named invariants, explicit trust boundaries, provider conformance checklist, and an implementation-correspondence table.

Requested by Tyler in the buzz-remote-agents design thread; co-designed with Dawn and Wren (review pending).

Structure

• System model — five principals (Desktop / Provider / Substrate / Agent / Relay) and the design axiom M1: no management channel — everything the desktop knows about a live remote agent flows through the relay.
• Five invariants with enforcement mechanism and stated boundary:
  • I1 identity fail-closed, I2 no secrets in configuration, I3 presence-is-status, I4 at-most-one-live-instance, I5 bounded lifetime.
• Provider protocol — discovery, info/deploy wire contract, untrusted-output rules, the reserved-key rule, and the deploy state machine (Running → no-op).
• Auto-stop — --exit-after-inactivity / BUZZ_ACP_EXIT_AFTER_INACTIVITY, default off, definition of "inactive", and why it must not share a name with the three existing timeout concepts.
• The Kubernetes binding — buzz-backend-kubernetes: kubeconfig-only auth, random-default namespace via schema default, the sprig image, pod shape (bare Pod, terminationGracePeriodSeconds: 60, 32-hex label / full-pubkey annotation), secrets, GC, config budget.
• Known defects at c1bca1b56 (Windows .exe id pollution; provider env inheritance vs kubeconfig exec plugins).
• Open decisions A–E marked inline and consolidated, awaiting owner ruling.

Notes for review

Docs-only. Every code claim was verified against the tree (correspondence table maps each spec concept to its file/function). The spec deliberately documents two desktop bugs as Known Defects rather than fixing them here — fixes are follow-up PRs.

[troyhoffman-oss]

Naming the trust boundary, then
saying which properties hold despite an untrusted provider holding the nsec, is
the right frame, and the M1 argument (the relay was the management plane all
along) is the part that makes the rest small.

For whatever it is worth as a conformance data point: #3449 is a working
provider for a different substrate, buzz-backend-ssh, which deploys agents as
systemd --user units over SSH. It was built before this spec existed, and Jud
ran it end to end this week from macOS against a Debian 12 guest behind a
ProxyJump alias. I went through the invariants against it at 879f57da3. Most
of it lines up, and where it does not, the spec is usually right and we are the
counterexample. Rough tally:

• I1 identity fail-closed: conforms on both sides. The provider re-checks the
  empty-key case itself rather than trusting the desktop
  (buzz-backend-ssh/src/deploy.rs:108-115), which I would suggest making a
  conformance item. It is one if, and it catches a desktop regression at the
  boundary instead of producing an identityless agent.
• I2 no secrets in config: conforms. Five config fields, no credential in any of
  them, passwords structurally impossible because every session runs
  BatchMode=yes. One scar worth a sentence in I2: the anti-secret rule is a
  key-name rule, so it rejects legitimate non-secret names. Our field is
  ssh_identity_file and not ssh_key_path only because the latter word-splits
  to ["ssh","key","path"] and is refused by validate_provider_config
  (backend.rs:461-491). The refusal is explicit and names the key, so it is
  not a silent failure, but it lands at schema-design time on a field that
  holds a path rather than a credential. One sentence in I2 saying the rule
  matches key names, not values, would save the next provider author the
  detour.
• I3 presence-is-status: conforms, and the harness's offline publish is the
  same code path.
• I4 at-most-one: conforms by deterministic pubkey-keyed unit naming. Inline
  comment on step 0 about derive-or-verify and about the rekeying migration.
• I5 bounded lifetime: does not conform, twice over, and the second way is a
  spec-strengthening finding. Inline comment 2.
• M1: partial and interesting. The SSH provider holds an SSH credential to the
  host, but no op queries status, fetches logs, or kills anything
  (main.rs:82-96), so the protocol-surface half holds. Your own kubeconfig
  concession already covers the credential half. The thing SSH raises that
  Kubernetes does not: here the staleness M1 buys is genuinely cheap to avoid
  (one systemctl is-active is a 200ms round trip), so M1 as an absolute ban
  rather than "no required management channel" is a bigger ask on some
  substrates than others. Not asking you to change it, just flagging that the
  price varies.
• Provider protocol: discovery order, id regex, output caps, exit rule,
  redaction, I2 validation and the reserved-key rule all match. launch,
  protocol_version and the staged-binary gate are absent on our branch too,
  same as your Known Defects 3 and 5.

On Known Defects: #1 (Windows .exe id pollution) is already fixed in
#3449 by 285924daf, and slightly more than your one-line description. Strip
the extension before deriving the id, and also move deduplication from the
filename to the id, otherwise a host carrying both buzz-backend-ssh and
buzz-backend-ssh.exe offers two providers the desktop reads as different.
Stripping is unconditional rather than cfg(windows) on the grounds that what
id a filename means should not depend on which machine reads it, and a rule
that only compiles on one platform is one CI never exercises. Happy to split
that commit out as a standalone PR if it helps this land faster.

#4 I confirmed line for line against our tree, and it would have bitten us:
the tick is pool_ready-gated at lib.rs:1743, pool_ready starts false under
lazy pool (:1320) and flips only after a wake (:2570), and wakes need
pending work (:1713-1716). Our env file pins BUZZ_ACP_LAZY_POOL=true for
the same reasons yours does, so a reaper on that tick would have been dead in
exactly the case it exists for.

Two additions to #3's consequence list, both from running this:

• BUZZ_ACP_RELAY_OBSERVER=true is inert without a resolved owner. The frames
  are encrypted and addressed to the owner's pubkey, so with neither auth_tag
  nor an owner the harness logs that it will publish nothing and the setting
  does nothing. Easy to hit rather than theoretical: dropping the NIP-OA
  identity is the current workaround for the cross-machine mention gate (#3277,
  #2987), and such a deploy otherwise works fine: the agent answers mentions,
  pushes branches, and only its telemetry is quietly missing. So your owner
  resolution rule fixes a silent observability hole as well as !shutdown,
  which is a stronger argument for it than the one currently in §Launch data.
• CLAUDE_CODE_EXECUTABLE needs re-resolution, not deletion (inline 4).

One hardening note. We added NoNewPrivileges=true to the unit (7bbb95dd3)
after dschwartzAI raised it from a self-hosted VPS pilot: without it the harness
climbs past the SSH user's privileges through any setuid binary or through
passwordless sudo granted to that user. That pilot exercised the harness, the
CLI, NIP-OA owner-reviewed drafts and repo branch pushes with it set and
nothing broke. It
is deliberately the whole delta, for the same reason you defer
readOnlyRootFilesystem: ProtectSystem/ProtectHome would be guessing at
which of the user's home an agent legitimately needs, and there is no modeled
workspace yet. Which suggests stating the hardening floor substrate-neutrally,
"no privilege escalation beyond the launching principal", with
allowPrivilegeEscalation: false + drop-all and NoNewPrivileges=true as the
two bindings of it.

On DECISION A: the SSH binding has no workspace at all. The unit has no
WorkingDirectory, so an agent starts wherever systemd --user puts it with no
Buzz-native checkout, and in practice every long-running remote agent we ran
operated in a separately provisioned checkout someone made by hand. Recommend
yes on A, and recommend the workspace be a protocol field the desktop
states rather than something the image scaffolds, since the image cannot know
which project.

On DECISION E: we went the other way, restart rather than no-op, because a
systemd restart is about five seconds and immediate config application was
worth more than mid-turn safety at that price. A pod recreate is a cold pull,
which is a completely different tradeoff. Suggest making E a per-binding ruling
rather than a protocol-wide one, with the spec stating the property that must
hold either way (a Start never leaves two live instances, and whichever choice
a binding makes is documented in its section).

Finally, a mechanical thing: this PR adds docs/remote-agents.md as a new file
and so does #3449, so whichever merges second gets an add/add conflict. Yours is
the normative spec and should keep the name. I will rename ours to
docs/remote-agents-ssh.md and retitle it as the SSH binding, so this does not
sit on your critical path. If you would rather absorb a short "§The SSH
Binding" section here (unit shape, lingering, PATH, NoNewPrivileges,
Restart= semantics) with the long-form troubleshooting staying in the
separate file, I will write that section against whatever shape you want.

Happy to take any of the above as issues instead of spec edits if that gets
this merged sooner. The thing I care about is that this ships natively.

[troyhoffman-oss]

The derivation is missing a term. There are two 30s timeouts on that path in
series, not one: buzz-acp/src/lib.rs:2612-2615 drains wake tasks under
timeout(Duration::from_secs(30), ...), and it runs before the prompt drain
you counted at :2629-2636 (also 30s). Then 2s presence (:2702-2712) and 5s
relay close (:2720, budget confirmed in relay.rs:900-912).

I do not think that makes the tail 67s today, and I want to give you the
argument rather than just the number. The two drains look close to mutually
exclusive: pool_ready starts false under lazy pool (:1320) and flips true
only after a wake completes (:2570), and the only non-test producers of
pool.join_set are dispatch_pending (:2968, reached only after a
successful try_claim) and dispatch_heartbeat (:3581, gated on
pool.any_idle()). Before the wake lands, no slot holds an agent, so the
prompt drain has nothing to wait on; once it lands, wake_tasks is empty. One
30s branch plus 2 + 5 is the realistic worst case, so 37s survives.

What I would change is that this is nowhere stated. The mutual exclusion is an
emergent property of the lazy-pool state machine, not an enforced one, and the
spec mandates BUZZ_ACP_LAZY_POOL=true for remote pods (§Launch data), so the
lazy path is the only path here. A later change that lets a wake overlap a live
pool moves the tail to 67s and silently makes 60s too small, with no test and no
sentence anywhere pointing at it.

Two suggestions. Name both timeouts in the derivation and say why only one is
payable, so the invariant is written down where someone changing the pool will
see it. And consider bounding both drains against one shared deadline in the
harness, which is a few lines and makes 37s true by construction rather than
true by coincidence. Worth the precision because your own qualifier ("anyone
tuning a grace period downward from 37s is reading the number without its
qualifier") tells readers to treat the figure as load-bearing, and they will.

Smaller note on the same budget: the per-agent reap is bounded at 5s each
(acp.rs:440, SIGKILL then a bounded wait) and runs serially over slots, so the
segment after the drain scales with BUZZ_ACP_AGENTS rather than being free.

[troyhoffman-oss]

This clause has more teeth than it looks, and I can say so because the first
non-Kubernetes binding violated it and I did not notice until I sent
!shutdown and watched the agent come back.

We run remote agents as systemd --user units in #3449. The unit is
Restart=always / RestartSec=5, which is the obvious default for a
long-running agent and the exact opposite of restartPolicy: Never. So
!shutdown stops the process, systemd restarts it five seconds later, and
both halves of I5 evaporate: the owner stop does not stop, and an inactivity
reaper (once BUZZ_ACP_EXIT_AFTER_INACTIVITY exists) would reap into an
immediate respawn loop. Under systemd, conformance costs a distinguished clean
exit code plus Restart=on-failure, or an explicit self-systemctl stop.
Neither is free and neither is obvious.

Suggestion: promote this from a clause inside I5 to a named conformance item
with the substrate obligation spelled out, something like "the substrate's
restart policy MUST treat harness exit as terminal; a supervisor that restarts
on clean exit voids I5 and the !shutdown contract". Kubernetes gets it by
picking a non-default restartPolicy; systemd gets it by picking a non-default
Restart=. Both are a deliberate choice against the substrate's default, which
is exactly the kind of thing a spec should name.

Related: restartPolicy: Never also gets you the harness's own exit code for
free as a signal. Under systemd we would need that code to distinguish
"reaped for inactivity" (do not restart) from "crashed" (do restart), and
buzz-acp currently exits 0 on the graceful path with no way to tell the two
apart. If the reaper lands with a distinct exit code, both bindings benefit.

[troyhoffman-oss]

This is right, and it contradicts Conformance item 1, which asks for
"meaningful exit codes". A provider author who reads item 1 first will encode
failure classes in the exit status and lose every structured error, because
invoke_provider discards stdout entirely on non-zero exit
(backend.rs:194-208) and reports raw stderr instead.

Our SSH provider resolves it by always exiting 0 and putting every failure
in the JSON body (buzz-backend-ssh/src/main.rs:59-62). That is not a
workaround, it is the only reading of §Invocation that works: exit code carries
exactly one bit, "the process crashed vs it did not", and an operation failure
is a successful process emitting {"ok": false}.

Worth saying that outright in §Invocation and changing item 1 from "meaningful
exit codes" to something like "exit 0 for any operation that produced a
response, including in-band failures; non-zero means the process itself
failed". Otherwise the two sections are a conformance contradiction, and the
half a provider author picks decides whether their error messages ever reach a
user.

One concrete thing that rides on this: we return an optional
recovery: {action, url} alongside {"ok": false} for the one failure that has
a machine-actionable fix (a tailnet ACL demanding a browser re-auth). It is
additive and unnegotiated: a desktop that does not know the key still renders
error, which carries the URL as text. If the spec wants providers to surface
actionable recovery rather than prose, an optional recovery object is a cheap
place to standardize, and it only works if the failure response is trusted,
which means exit 0.

[troyhoffman-oss]

This is the same bug as Known Defect 2 and it is bigger than macOS. I hit it on
the other end of the wire, and there it is not a credential problem, it is a
"the product does not work" problem.

A non-interactive ssh host sh -s reads no profile. On stock Debian that
leaves PATH at /usr/local/bin:/usr/bin:/bin:/usr/games, which does not
contain ~/.local/bin, where every Buzz tool installs, and where pipx and
npm --prefix=~/.local put the ACP adapters. On a host with all twelve
adapters installed and runnable, discover_harnesses reported the entire
catalog as available: false, and the deploy that followed refused the
operator's own pin with "harness not found". The host was fine. Fixed in
e8a8e34fb with one shared preamble prepended to every generated script, and
the unit's own PATH then written from that same shell's $PATH verbatim so the
two cannot drift.

The generalized rule I would put in this bullet: PATH is a property of how a
process was started, at both ends of the protocol. Provider-side (launchd),
and agent-side (non-interactive SSH, systemd --user, cron, anything that is
not a login shell). The Kubernetes binding gets a free pass on the agent side
because the image controls its own PATH, which is precisely why a
Kubernetes-first spec would not surface the second half.

Small detail from getting this wrong once: write the prepend as
export PATH="${HOME:-}/.local/bin${PATH:+:$PATH}", not :$PATH. With PATH
unset the naive form leaves a trailing empty element, which means the current
directory, and that is the one PATH value worth refusing to write.

Also, since §Launch data says CLAUDE_CODE_EXECUTABLE is a host path that must
not be forwarded: dropping it is not sufficient, it has to be re-resolved,
preferring ~/.local/bin/claude (the native launcher, which follows Claude
Code updates) over the adapter's bundled point-in-time binary. Unset it and you
silently get the stale bundled SDK build, which is a version regression rather
than a failure, so nobody notices.

[troyhoffman-oss]

Strongly agree with keying on the pubkey, from having shipped the bug this
prevents. Our first version keyed the systemd unit on the display name. Two
agents both called "Research Bot" on one SSH account therefore shared one unit,
one env file and one backend_agent_id, so the second deploy overwrote the
first agent's minted nsec and starting either record drove whichever identity
was written last. 449a5b9ae. We independently landed on the same 12-hex
fragment you specify, which is a decent cross-binding check on the choice.

Two things I would add here.

First, "MUST derive, never a caller-supplied pubkey" prices in a dependency the
spec does not mention. Our SSH provider has five direct dependencies (serde,
serde_json, zeroize, base64, sha2); deriving the pubkey means linking secp256k1
plus bech32, which is a real addition to that surface for a binary whose whole
job is to shell out to ssh. A middle ground with the same guarantee: the provider
MUST derive or MUST verify a supplied pubkey against the key it derives, and
a mismatch is a fatal in-band error. That names the actual property (the unit
name and the identity running in it agree) rather than the implementation, and
it keeps thin providers thin. Right now we take the payload's pubkey and
validate its shape (64 hex chars) but cannot cross-check it, which is the gap
your rule closes and I would rather close it cheaply than not at all.

Second, deterministic identity-keyed naming has a migration hazard worth one
sentence in this section. When the naming rule changes, the instance name
changes, so a host provisioned under the old rule gets a new instance
alongside the old one on redeploy rather than a replacement. In our case the
orphan kept running under Restart=always with an env file that still held an
nsec, and with no undeploy op there was nothing to clean it up with; we had to
ship manual removal instructions (41b8a628d). Kubernetes has the same shape
(old pod name, old Secret, GC only reaps terminated pods matching the current
label). Since §K8s GC keys on buzz.block.xyz/agent-pubkey, any future change
to the truncation length or the label key strands the previous generation
permanently.

[DanTup]

I'm curious - why does an agent have to be spawned by a desktop app? Why can't I start up a relay and an agent connect to it without ever opening the desktop app?

The first announcements I saw of Buzz said it was different to Slack because Agents were first-class and not owned by users, but Agents appear to be tied directly to a specific human's running desktop (even in this spec about remote agents).

Everything appears to be architected so that communication with an agent is always through the relay, so I don't understand why starting them is tied to the desktop app. Why can't I start an agent from a CLI in a container, and send its public key to an admin to allow to join the server? What happens if I need to restart the host running a relay and agents - can the agents not continue unless I open the desktop app on the machine where I set them up? 🤔

[Shipitrealgood]

At the risk of naivety: was registering the substrate itself on the relay ever considered — VMs, clusters, etc. as keypairs, so deploy targets are relay principals the way agents are? I can see M1 deliberately keeps the provider protocol free of any substrate API, so I'm curious whether a relay-registered environment was considered-and-rejected, or is just beyond v1's scope.

[Jud]

Reviewed from the position of an operator running the #3449 SSH provider in
anger: macOS arm64 desktop, Debian 12 KVM guest reached through an
~/.ssh/config ProxyJump alias, self-hosted relay, and a standing fleet of
long-running agents under systemd --user with lingering enabled.

Everything below was verified against the tree at #3449 head 879f57d rather
than inferred from the prose.

The two documents collide

#3449 and #3748 both add docs/remote-agents.md, 685 lines against 1370, with
different purposes: an implementation contract for the SSH provider, and a
formal spec centred on a Kubernetes binding. Whichever lands second cannot
merge. This needs an explicit ruling, either distinct filenames or one document
with the binding sections separated. #3449 is already CONFLICTING against
main.

Where the spec and the SSH provider disagree

Ordered by how hard they bite.

1. protocol_version is fail-closed, and the SSH provider does not send one

§Info: "A missing protocol_version is an error, not a presumed 1…
Fail closed." §Discovery's pre-secret negotiation gate makes that check
mandatory before the nsec crosses the boundary.

Verified against buzz-backend-ssh's info response:

{"name":"SSH","version":"0.1.0","description":"…","config_schema":{…},"ok":true}

protocol_version appears nowhere in the crate. If the gate ships as specified,
the only existing OSS provider is rejected before deploy. The fix is one field,
but it has to land in #3449 before or with the gate, or the two PRs merge into a
broken remote path.

2. I5 and auto-stop contradict the always-on model the SSH unit is built for

Conformance item 5: the deployed harness "enables an inactivity bound (I5) and
the substrate does not resurrect terminated instances."

The shipped unit asset does the opposite, deliberately:

StartLimitIntervalSec=0
Restart=always
RestartSec=5

and #3449's own doc states it plainly: "Restart=always then restarts the unit
after RestartSec=5; a !shutdown stops the current process, not the unit."
That is resurrection by design, and with lingering enabled it is the whole point
of a VPS fleet.

I5's stated rationale is billable idle capacity, "an idle LLM pool in a cluster
is billable waste". That rationale is real for per-pod Kubernetes billing and
absent on a VPS that is paid for whether or not an agent is idle. As written, an
SSH provider cannot conform to item 5 without abandoning the always-on model,
and reaping our fleet after 2h idle would be a regression rather than a safety
property.

Suggested: scope I5 to metered substrates, or make the lifetime policy a
per-binding declaration (bounded-idle against supervised-persistent) rather
than a global conformance item. BUZZ_ACP_EXIT_AFTER_INACTIVITY is confirmed
absent from the harness, so Known Defect 4 is accurate.

3. The owner-resolution rule would refuse our exact working configuration

§Launch data, normative: "the provider MUST have either a non-null auth_tag
… or a non-null launch.owner_pubkey …; if both are null it MUST refuse the
deploy."

Our fleet ran with no auth_tag and respond_to=allowlist, the current
workaround for the cross-machine mention gate (#3277 / #2987). #3449 permits it,
and the only consequence is that observer frames go unpublished, which #3449 now
documents (879f57d, added after our last field report).

The spec's justification is that without an owner the harness cannot match
!shutdown. On systemd that is not the stop mechanism, systemctl --user stop
is, so the refusal removes a working configuration to protect a mechanism the
binding does not rely on. Suggested: make it a warning, or let
respond_to_allowlist satisfy it, for bindings whose stop path is
substrate-side.

4. M1 "no management channel" against the five ops that make remote setup usable

M1: the protocol "contains no substrate API: no status query, no exec, no
log fetch, no kill", and the spec defines exactly two ops (info, deploy).

#3449 ships five: info, check, discover_harnesses, probe_models,
deploy. check is a status query, and the other two are remote exec.

They are also the ops that make the feature work for us. discover_harnesses is
why the create dialog offered the four harnesses actually installed on the VM
instead of this Mac's catalog, and probe_models is why the model list was the
remote host's. Deleting them to satisfy M1 would mean hand-typing a harness name
and hoping.

M1's real argument is about post-deploy observation, presence-is-status and its
staleness bounds. Config-time discovery is a different thing. Suggested: scope
M1 to the post-deploy lifecycle explicitly, so a binding may offer pre-deploy
inspection ops without being non-conforming.

5. The provider trusts a caller-supplied pubkey instead of deriving it

Not in the spec's Known Defects list, which is all desktop and harness side, so
flagging it as a new one. §Deploy State Machine Step 0 is unambiguous:

The payload carries the nsec, not the pubkey. Before any substrate read or
mutation, the provider MUST parse private_key_nsec and derive the public
key from it… Every selector, name, and comparison below uses the derived
pubkey, never a caller-supplied one.

buzz-backend-ssh does the opposite. It requires a pubkey field in the
payload, validates only its shape (64 hex chars), and keys the unit name, the
env filename and the returned agent_id on it. The crate has no secp256k1
dependency and performs no derivation, verified against Cargo.toml and the
whole src/ tree.

The consequence is the collision that 449a5b9 ("key remote units by agent
identity") set out to prevent. A payload whose pubkey does not match its
private_key_nsec produces a unit and an env file named for identity A while
the agent connects to the relay as identity B, and two records sharing one nsec
but carrying different pubkey values get two live units for one relay
identity, which is I4 violated through the front door. Nothing in the provider
can detect either case.

Deriving it is a few lines and removes a required payload field.

6. Much of the "general" contract is Kubernetes-shaped

The create-intent fingerprint, preconditions.uid/resourceVersion
compare-and-delete, the 409 Status.reason discriminator and the quorum-read
rule are stated normatively in the binding-independent §Deploy State Machine,
but only mean anything against an apiserver. On systemd, "converge to
at-most-one-live-instance" is a deterministic unit name plus restart, which
#3449 already does by keying units on the agent pubkey (449a5b9), satisfying
I4's intent exactly.

Suggested: mark those rules as belonging to the Kubernetes binding, or #3449
gets judged non-conforming against machinery its substrate has no analogue for.

Where the spec is right, and #3449 independently corroborates it

• Known Defect 3 (payload bypasses the launch resolver), verified: no
  launch block is emitted anywhere in agents_deploy.rs or the provider.
  feat: remote agents over SSH — provider contract, buzz-backend-ssh, and where-runs-first #3449 hit the consequences empirically and lists them as its own limitations,
  team instructions lost on remote records and
  runtimeSupportsLlmProviderSelection reduced to a hardcoded id test. Two
  independent routes to the same defect, so it should be fixed once, in the
  shared resolver.
• Known Defect 5 (deploy never checks protocol_version), verified: no
  protocol_version reference in the desktop deploy path at all.
• Known Defect 4 (the I5 reaper does not exist), verified absent.
• Known Defect 1 (Windows .exe id pollution), already fixed on feat: remote agents over SSH — provider contract, buzz-backend-ssh, and where-runs-first #3449
  (285924d), as that PR's author notes.

DECISION A is not open, we answered it in production

The spec asks whether the image should scaffold the agent workspace that
ensure_nest provides locally. #3449 lists the same gap: "No project workspace
is projected onto the host… the unit here has no WorkingDirectory at all."

We hit this and hand-rolled a systemd drop-in:

[Service]
WorkingDirectory=/home/agent/nest

Every long-running agent we run needed it. That is field evidence for yes on
DECISION A, and it should not be scoped to the Kubernetes image, because the SSH
binding has the identical hole. The right shape is the one both docs already
gesture at: a workspace field the desktop states, rather than the provider
guessing.

One thing that changed today

VISION_REMOTE_AGENTS.md landed on main in #3924, and it ratifies this spec's
model in the project's own voice: no substrate control channel after deploy,
staging one exact artifact and refusing an unknown protocol version, converging
to a single live instance, bounding the instance's lifetime, Kubernetes as the
first substrate. It also says outright that "the default state of a remote agent
is 'not running'".

That makes the scoping question above harder to defer, because it is now a
question against merged text rather than an open PR. The vision doc does leave
room ("a cluster today; a VM, a PaaS, or something serverless-shaped tomorrow"),
but nobody has yet written down that a VM binding's lifetime policy should
differ from a cluster's.

What I'm asking for

#3748 is the stronger document as design writing, and its desktop and harness
defect findings are accurate. I verified four of five against the tree. But it
is written around a Kubernetes provider that does not exist yet, and four of its
normative rules (items 1 through 4 above) would reject or degrade the SSH
provider that does, on the substrate we actually run.

None of the four is a deep disagreement. Each is the same scoping question, and
it would help to have it answered on the record: which of these rules are the
provider contract, and which belong to the Kubernetes binding?

[UNST-ai]

+1. would love the same UX as a desktop spawned agent for a remote runtime.

[tlongwell-block]

I'm curious - why does an agent have to be spawned by a desktop app? Why can't I start up a relay and an agent connect to it without ever opening the desktop app?

Creating an agent outside the desktop will work perfectly fine as long as you have your nsec to sign a NIP-OA tag with when you spawn an agent.

The first announcements I saw of Buzz said it was different to Slack because Agents were first-class and not owned by users, but Agents appear to be tied directly to a specific human's running desktop (even in this spec about remote agents).

Agents can run independently or can be managed by a principle like a user or team (given a NIP-OA tag with specific authorization). NIP-OA ("owning an agent") makes it easier to run a relay with an npub allowlist so that the relay owner/admins don't need to add every agent individually. It's also nice to know who is responsible for a given agent.

Everything appears to be architected so that communication with an agent is always through the relay, so I don't understand why starting them is tied to the desktop app. Why can't I start an agent from a CLI in a container, and send its public key to an admin to allow to join the server? What happens if I need to restart the host running a relay and agents - can the agents not continue unless I open the desktop app on the machine where I set them up? 🤔

You can do this! The agent harness will retry with backoff to connect to a failed/restarted relay.

@DanTup great questions. Thanks for participating here!

[tlongwell-block]

At the risk of naivety: was registering the substrate itself on the relay ever considered — VMs, clusters, etc. as keypairs, so deploy targets are relay principals the way agents are? I can see M1 deliberately keeps the provider protocol free of any substrate API, so I'm curious whether a relay-registered environment was considered-and-rejected, or is just beyond v1's scope.

This is an interesting idea! Your substrate would need to support sending events to the relay for this to be useful. ie "I'm about to shut down", "I have a hardware issue", "Restarting proc 1", etc

If you have a subtrate that wants to adopt this, we can look at this idea @Shipitrealgood

For now, we'll target generic ones to get folks going and look at Buzz specific substrates later. We have something in mind already. Keep an eye out ;)

[DanTup]

@tlongwell-block thank you for the replies!

Creating an agent outside the desktop will work perfectly fine as long as you have your nsec to sign a NIP-OA tag with when you spawn an agent.

Are there any examples/docs on this? I had a relay running and I wanted to spawn and connect an agent in a fresh container. I tried a few things like starting https://github.com/block/buzz/tree/main/crates/buzz-acp with codex set up, and it claimed to connect to the relay and found 0 channels, and it was never visible to me in the desktop. I had added its public key to the relay (via the relay admin scripts).

I spent a bunch of time with various AIs and none of them could figure out what was wrong and they all concluded it was not supported.

For now, I don't mind whether the agent is standalone if "owned by me", I just want to test and play around - although eventually it would be nice if the agents could be "owned by me" even when started this way.

This is the thing holding me back from trying this out, I just won't run them on my desktop for anything 😄

I understand you're all very busy, but if there aren't existing docs/examples, an example script (even written by agents) would be really helpful. Maybe even an example agent in the deploy/compose config or something?

Thanks!