Skip to content

bug: fleet restart bypasses start_agent_internal — no agent adopts a rebuilt base image on "Restart All" #1860

Description

@AndriiPasternak31

Summary

POST /api/ops/fleet/restart (routers/ops.py::restart_fleet, ~:222-300) stops and starts each
running agent with raw Docker calls, bypassing start_agent_internal() entirely:

container = get_agent_container(agent_name)
if container:
    await container_stop(container, timeout=30)
    await container_start(container)

grep -n "start_agent_internal" src/backend/routers/ops.py returns zero hits.

Consequence: none of the eight needs_recreation config predicates run, and neither does #1809's
image-drift adoption. No agent in the fleet adopts a rebuilt trinity-agent-base on this path
not just the system agent.

Why it matters

"Restart all agents" is the most intuitive thing an operator does after build-base-image.sh. It is
also the one path guaranteed not to upgrade anything. The operator sees every agent restart, assumes
the new image is live, and it isn't — the same silent-staleness class as #1809 and #1816, but
fleet-wide.

Found during the #1816 plan review while mapping every code path that starts a stopped agent.

Where

Note: architecture.md already documents this bypass for a different reason — the #1089
subscription-token rotation notes that "a fleet restart via routers/ops.py does a raw
container_stop+container_start that bypasses start_agent_internal". The consequence for image
adoption was not drawn.

Design tradeoff for whoever picks this up

Two shapes:

  1. Route ops-restart through start_agent_internal — cleaner, single source of truth, gets all
    eight predicates plus bug(lifecycle): stale transport circuit-breaker Redis key survives agent delete/recreate — a fresh healthy agent fast-fails as "unhealthy" #1560 breaker clearing for free. But it pulls credential injection, skill
    injection and read-only sync into a bulk operation across every agent at once. Real
    blast-radius question; that is why the bypass presumably exists.
  2. Replicate the cold-start gate in the restart loop — narrower, but forks the adoption logic,
    which is exactly what bug: a rebuilt agent base image is never picked up — no image-drift predicate, so the v0.8.5 "stop/start-with-recreate" upgrade step is a no-op #1809's acceptance criteria ruled out.

Acceptance criteria

Related: #1809 (per-agent cold-start adoption), #1816 (system-agent adoption), #1814 (platform images
not rebuilt on in-place upgrade), #1817 (per-agent start lock).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions