Skip to content

Broker startup does not retry Relaycast workspace_busy admission 429 #1728

Description

@khaliqgant

Problem

Relay broker startup/admission treats Relaycast HTTP 429 workspace_busy as terminal after one attempt. That makes otherwise healthy RelayFlow/Fleet qualification runs fail while the workspace write-admission lane is briefly saturated.

Clean evidence

Relay PR #1665 proof run 34409605164 provisioned a fresh Daytona step sandbox, installed the exact Relay 11.10.4 broker, seeded the workspace, and then failed before the proof arm could start:

mcp-args --register failed (exit 1): registration for base-prover was rate-limited;
retry after 60s: Workspace write capacity is busy; retry with backoff
(code: workspace_busy; attempts: 1)

No product assertion ran, so this is orchestration-infrastructure failure rather than a feature failure.

Expected contract

  • Retry only HTTP 429 carrying the exact workspace_busy code.
  • Keep unrelated 429 responses terminal.
  • Bound the retry count and total delay.
  • Preserve status, code, message, and attempts on exhaustion.
  • Do not replay the complete startup handshake after request-scoped retry exhaustion because it contains unkeyed workspace/agent-registration POSTs.
  • Add an exact base/head RelayFlow proof case.

Scope

This is distinct from #1715, which covers worker /api/spawn preregistration and HTTP 503 database_overloaded. It is also complementary to AgentWorkforce/cloud#3507, which handles the Cloud executor direct mcp-args --register invocation.

Related upstream incident: AgentWorkforce/relaycast#389.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions