schedule-maintenance can re-fire every prompt every 5 min when one provider silently fails

[jrhizor]

Problem

schedule-maintenance (runs every 5 min) has an unthrottled expedite path. Combined with an overdue check that trips on any missing model, a single silently-failing provider makes every prompt look perpetually overdue, so maintenance drags each prompt's next job to now on every cycle — firing prompts ~every 5 min instead of on their 24h cadence.

This is what turned the v0.2.15 OpenAI crash into a ~10x runs + API-spend event (with prompts/brands flat). The provider crash itself is fixed (#436) and per-run failures are now reported to Sentry (#438); this issue is the defense-in-depth so the next silent provider failure can't run away.

Where

apps/worker/src/jobs/schedule-maintenance.ts:

• modelNames is all SCRAPE_TARGETS models (~L64-65), not the models a brand actually runs.
• isOverdue trips if any of those models lacks a recent prompt_runs row (~L104-119) — a provider that never saves a row (always fails) is overdue forever.
• The expedite path is a raw UPDATE pgboss.job SET start_after = now() with no throttle (~L140-156)…
• …unlike the create path right below it, which is throttled by singletonKey + singletonSeconds: 3600 (max 1 new job/prompt/hour) (~L164-181).

apps/worker/src/jobs/process-prompt.ts: the job only throws (→ backs off) when all runs fail, so a consistently-failing provider never fails the job — it reports completed while still billing the API and never producing a row.

Impact

Normal: 1 firing/prompt/24h. Broken: up to 1 firing/prompt/5min (~288x ceiling; ~10x observed under localConcurrency: 10). Runs and API spend scale together; no new prompts or brands.

Possible resolutions

1. Throttle the expedite path (primary ask). Give it the same guard the create path has — e.g. a per-prompt minimum re-expedite interval, or skip a prompt whose job start_after was already moved to ~now within the last cycle. Caps the blast radius regardless of why a prompt looks overdue.
2. Scope the overdue check to brand.enabledModels via selectTargetsForBrand(...), mirroring process-prompt, so models a brand doesn't run don't force perpetual overdue. Partial — doesn't cover an enabled model that always fails (the actual incident).
3. Base "overdue" on scheduling state, not data recency. Only expedite when the next run is genuinely late (no pending job, or a created job whose start_after is already in the past) rather than because a model lacks a recent row. Addresses the root cause but is a larger semantic change.
4. Surface consistently-failing providers (alert / back off) instead of silently completing. Partially covered by surface per-run provider failures in the worker to Sentry #438; a threshold could gate expediting.
5. Hard per-cadence floor: never run a prompt more than once per cadence (or a small floor) regardless of maintenance.

Likely combo: (1) + (2) for a small, low-risk cap, with (3) considered if we want to fix the overdue semantics properly.

Refs: #436 (provider fix), #438 (Sentry observability).