Skip to content

Worker: per-type claiming serializes job types, and per-type limits oversubscribe maxConcurrency #153

Description

@NathanTarbert

Two coupled scheduling defects in packages/outpost/queue/src/worker.ts. Distinct from #138, which covers shutdown and health reporting.

1. Per-type batches are awaited sequentially

claimAndProcessJobsPerType awaits each type's batch inside the loop, so the per-type concurrency pools never actually run concurrently. A slow type blocks every type after it in the same poll: AI_RESPONSE has a 120s timeout, so an ESCALATION or SLACK_MIRROR job queued behind it waits for the AI pipeline even though its own pool is idle.

2. The per-type limits sum above the global cap

The registered per-type limits sum to 13 while maxConcurrency is 10. Combined with the sequential claiming above, types declared last in the map are the ones starved — SLACK_MIRROR is currently last.

Either dispatch the per-type batches concurrently (Promise.all over the types, each still respecting its own limit), or make the global cap the authority and document the relationship. Whichever is chosen, the invariant deserves a test: today queue.test.ts:301 computes maxSeen and never asserts it, so a concurrency regression cannot fail the suite.

Found by: CR round 2 on #150 (slots 1-2, 2-1, 5-1).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureWorker, queue, CI, deploy, containers, observabilityroadmapTracked on the Outpost roadmaproadmap: nextRoadmap horizon: after launch path clears

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions