Handling Resume from Pause failures

[msau42]

When we pause an actor, the actor is forced to resume on the same node. However, at the time of resume, the node may not have capacity for the actor anymore. Let's discuss how we can recover from this situation in the near term.

My thoughts for the desired near term flow:

1. Resume() returns an out of capacity error
2. User needs to trigger Suspend() in order to offload the snapshot to remote storage.
3. User can then retry Resume()

In the future, we could improve on this by automatically doing steps 2+3 as part of the initial Resume() call.

[ILLUM1N0X]

Implementation Plan: Near-Term Flow for Resume from Pause Failures

To recover from out-of-capacity failures when resuming an Actor from STATUS_PAUSED, we will implement the near-term flow (Pause -> out of capacity -> Suspend -> Resume on any node).

Key Design Decisions

• Explicit Error Signaling: Resume() returns codes.ResourceExhausted when a node lacks capacity, avoiding silent cloud storage uploads or SLA violations.
• Service Separation (NodeStorage): Paused actors have no active ateom pod. Instead of adding a filesystem operation to AteomHerder, we introduce a dedicated NodeStorage gRPC service on atelet for node-level filesystem offloading.

---

Implementation Phases

1. Define NodeStorage Service (internal/proto/ateletpb/atelet.proto)
   • Add service NodeStorage with rpc OffloadLocalCheckpoint. Allows ateapi to upload local checkpoint files directly from disk without an active ateom pod.
2. Implement NodeStorage Server (cmd/atelet/main.go)
   • Implement OffloadLocalCheckpoint to read checkpoint files from ateompath.LocalCheckpointsDir, upload them to remote storage, and delete the local directory.
3. Support STATUS_PAUSED in Suspend Workflow (cmd/ateapi/internal/controlapi/)
   • dialer.go: Add DialForNode(nodeName string) to AteletDialer (extracting pod lookup/connection logic from DialForWorker) since paused actors have no worker pod name.
   • workflow_suspend.go:
     • Allow actors in STATUS_PAUSED. When paused, call dialer.DialForNode(nodeName) and invoke NodeStorage.OffloadLocalCheckpoint.
     • Clear LocalSnapshotInfo upon entering STATUS_SUSPENDED, removing the node scheduling constraint so Resume() can schedule anywhere.
4. Standardize Capacity Errors (cmd/ateapi/internal/controlapi/workflow_resume.go)
   • In AssignWorkerStep, return codes.ResourceExhausted ("no free workers available matching scheduling constraints") when scheduler.Schedule() fails due to capacity.
5. Automated Testing (cmd/ateapi/internal/controlapi/)
   • Add unit tests in workflow_suspend_test.go and an e2e test in functional_test.go verifying: Pause -> capacity failure -> Suspend -> relocation Resume.

[msau42]

Overall approach lgtm. Just a few comments:

• I'm not sure if we need a separate service. There is some overlap with existing Checkpoint/Resume calls wrt managing local snapshots, and we'll probably need to ensure some concurrency control among all 3 methods. The existing service dials into atelet, not the ateom pod.
• We will have some intermediate state where the local snapshot is deleted but the LocalSnapshotInfo hasn't been updated yet. We should think about the failure modes and recovery path.
• Also keep in mind atelet: prune local pause snapshots #705 which is also handling the snapshot cleanup

[msau42]

Related issue: #791

[BenTheElder]

Longer term I think we should consider direct p2p instead of bouncing through GCS, for latency reasons. I also filed #798

[ILLUM1N0X]

The offload mechanism (enabling SuspendActor on a PAUSED actor) is tracked in #791 and currently being implemented by Dmitry Berkovich (@dberkov).

I will follow up by reviewing those PRs. Once they land, I'll work on standardizing the out-of-capacity error in ResumeActor (codes.ResourceExhausted) and adding an end-to-end test for the full pause-to-resume capacity recovery flow.

[ILLUM1N0X]

Phases 1–3 landed under #791, though not as written above (the offload RPC went on the existing AteomHerder rather than a new service). Remaining work is phases 4 and 5, revised below.

Phase 4 — Unambiguous capacity signal

Return codes.ResourceExhausted from assignWorkerAttempt for scheduling.ErrNoCapacity (currently returns FailedPrecondition in workflow_resume.go).

We can't return FailedPrecondition as-is: it is overloaded on this RPC, since it also covers "the actor isn't in a resumable state" (workflow_resume.go) and the two demand opposite client responses.

We considered keeping FailedPrecondition and carrying an AIP-193 ErrorInfo reason to disambiguate, which would have left the router untouched. It doesn't work: ateapi's ServerUnaryInterceptor rebuilds every error as status.Error(code, message) and drops details.

FailedPrecondition is currently the de-facto capacity signal, so three consumers move with it:

1. The router's park predicate (resumer.go): add ResourceExhausted so parking is unchanged.
2. The HTTP mapping (errors.go): fold ResourceExhausted into the FailedPrecondition/Aborted branch (503 + description). Its current "actor %s rate limited" message was written speculatively since nothing in substrate produced the code until now.
3. The no_capacity metric bucket (extproc/metrics.go): move it to the new code.

Accepted risk: we don't exclusively own ResourceExhausted (grpc-go emits it on message-size violations), so a passthrough would be parked once before failing. Not reachable in practice given the payload sizes on this path.

Phase 5 — Relocation test

The test goes in functional_test.go with a synthetic two-node fleet, building on the harness TestSuspendActor_FromPaused already uses.

TestResumeActor_RelocatesAfterSuspendFromPaused:

1. createTemplate + createWorkerPod(..., "worker-1", "node1", "pool1").
2. Create actor A, resume (lands on worker-1), pause → PAUSED, pinned to node1, worker-1 freed.
3. Create actor B and resume it → B occupies worker-1, so node1 has no free worker.
4. createWorkerPod(..., "worker-2", "node2", "pool1") → free capacity exists, but on the wrong node.
5. ResumeActor(A) → assert ResourceExhausted.
6. SuspendActor(A) → assert SUSPENDED and LocalSnapshotInfo == nil.
7. ResumeActor(A) → assert it succeeds and the worker assignment is worker-2 on node2.