Found during the four-round review of #180. Not fixed there — apps/worker/src/index.ts was deliberately left at main's behaviour.
Worker.start() registers its own SIGTERM/SIGINT handlers (packages/outpost/queue/src/worker.ts) that call this.stop(). Those are registered before index.ts's handlers, so on SIGTERM:
- The Worker's own handler runs first.
stop() synchronously sets running = false, then begins awaiting the drain — nobody awaits that promise.
index.ts's shutdown() then calls await worker.stop(), which hits if (!this.running) return and returns immediately.
shutdown() proceeds to prisma.$disconnect() and process.exit(0) while jobs are still executing.
So the drain Worker.stop() exists to provide never happens on a real shutdown. In-flight AI_RESPONSE (120s) and HUBSPOT_SYNC/ACCOUNT_SCORING (300s) jobs are killed mid-execution with Prisma disconnected under them.
This compounds with a second finding: nothing in the queue reclaims a job stuck in PROCESSING. worker.ts sets status='PROCESSING', lockedAt=NOW() on claim and only clears lockedAt on completion/failure/retry; there is no stale-lock reaper in job-cleanup.ts or anywhere else. So any hard exit mid-job strands those rows permanently, not until the next boot.
Suggested fix: a registerSignalHandlers: false option on Worker so the app owns shutdown, or make stop() joinable (cache the in-flight stop promise and return it rather than early-returning on !running). A stale-lock reaper is worth its own consideration either way.
Found during the four-round review of #180. Not fixed there —
apps/worker/src/index.tswas deliberately left at main's behaviour.Worker.start()registers its own SIGTERM/SIGINT handlers (packages/outpost/queue/src/worker.ts) that callthis.stop(). Those are registered beforeindex.ts's handlers, so on SIGTERM:stop()synchronously setsrunning = false, then begins awaiting the drain — nobody awaits that promise.index.ts'sshutdown()then callsawait worker.stop(), which hitsif (!this.running) returnand returns immediately.shutdown()proceeds toprisma.$disconnect()andprocess.exit(0)while jobs are still executing.So the drain
Worker.stop()exists to provide never happens on a real shutdown. In-flightAI_RESPONSE(120s) andHUBSPOT_SYNC/ACCOUNT_SCORING(300s) jobs are killed mid-execution with Prisma disconnected under them.This compounds with a second finding: nothing in the queue reclaims a job stuck in
PROCESSING.worker.tssetsstatus='PROCESSING', lockedAt=NOW()on claim and only clearslockedAton completion/failure/retry; there is no stale-lock reaper injob-cleanup.tsor anywhere else. So any hard exit mid-job strands those rows permanently, not until the next boot.Suggested fix: a
registerSignalHandlers: falseoption onWorkerso the app owns shutdown, or makestop()joinable (cache the in-flight stop promise and return it rather than early-returning on!running). A stale-lock reaper is worth its own consideration either way.