Skip to content

worker: shutdown() does not actually drain in-flight jobs #181

Description

@jerelvelarde

Found during the four-round review of #180. Not fixed there — apps/worker/src/index.ts was deliberately left at main's behaviour.

Worker.start() registers its own SIGTERM/SIGINT handlers (packages/outpost/queue/src/worker.ts) that call this.stop(). Those are registered before index.ts's handlers, so on SIGTERM:

  1. The Worker's own handler runs first. stop() synchronously sets running = false, then begins awaiting the drain — nobody awaits that promise.
  2. index.ts's shutdown() then calls await worker.stop(), which hits if (!this.running) return and returns immediately.
  3. shutdown() proceeds to prisma.$disconnect() and process.exit(0) while jobs are still executing.

So the drain Worker.stop() exists to provide never happens on a real shutdown. In-flight AI_RESPONSE (120s) and HUBSPOT_SYNC/ACCOUNT_SCORING (300s) jobs are killed mid-execution with Prisma disconnected under them.

This compounds with a second finding: nothing in the queue reclaims a job stuck in PROCESSING. worker.ts sets status='PROCESSING', lockedAt=NOW() on claim and only clears lockedAt on completion/failure/retry; there is no stale-lock reaper in job-cleanup.ts or anywhere else. So any hard exit mid-job strands those rows permanently, not until the next boot.

Suggested fix: a registerSignalHandlers: false option on Worker so the app owns shutdown, or make stop() joinable (cache the in-flight stop promise and return it rather than early-returning on !running). A stale-lock reaper is worth its own consideration either way.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area: infrastructureWorker, queue, CI, deploy, containers, observabilityroadmapTracked on the Outpost roadmaproadmap: nowRoadmap horizon: V1 launch critical path / safety

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions