Skip to content

fleet release intermittently returns Internal server error for live rostered remote agents #1671

Description

@khaliqgant

Impact

Fleet cleanup is not reliable: exact-name release requests can fail with a generic internal error even while the target is reported as live on an online node. This strands agents and makes clean-room orchestration teardown non-deterministic.

Exact reproduction (2026-09-06)

Environment:

  • caller CLI: agent-relay v11.10.2
  • target node: finn-mini
  • target node reported broker relay-broker/11.8.3, online/live with handlers live

Both targets were listed by agent-relay fleet agent list --all --json as remote live+roster on finn-mini immediately before and after release.

Commands:

agent-relay fleet release relaycast-374-retry-finn-0906 --reason "PR 375 handed off to supervised exact-head review and merge gate"
agent-relay fleet release relaycast-374-sonnet-0906 --reason "PR 375 handed off to supervised exact-head review and merge gate"

Both waited roughly 20 seconds and returned only:

Internal server error

Control: a release of relayfile-465-veto-sonnet-0906 on the same node shortly beforehand dispatched successfully. A second target without a roster entry returned the expected Agent not found, so the CLI can distinguish at least that path.

Definition of done

  • Return the typed downstream failure with invocation/node/request correlation instead of a generic 500.
  • Make release idempotent for a live or already-exited exact-name remote worker.
  • Reconcile roster and node-local presence after release so a successful response proves the process and roster record are gone.
  • Add deterministic tests for success, already released, node timeout, stale roster, and version-skewed node handlers.
  • Add this case to the two-clean-Daytona Fleet RelayFlow and fail qualification if cleanup cannot prove absence.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions