[BUG] Webhook Route Can Permanently Drop Deliveries (429/500 = Lost Events) and Grows `webhook_deliveries` Forever

[ionfwsrijan]

Description

src/app/api/webhooks/github/route.ts:

• Lines 52-67: rate-limits all webhook traffic to 100/min per installation (or a global:{eventType} bucket when no install id).
• Lines 69-99: INSERTs a webhook_deliveries row for every delivery received — including event types the app never handles (push, create, delete, star, watch).
• Lines 101-119: forwards github/{eventType} to Inngest for any event type; most have no handler and are simply persisted + dispatched for nothing.
• Lines 65-67, 97, 118: return 429/500 on rate-limit, DB, or Inngest failures.

There is no cleanup for webhook_deliveries anywhere (grep shows only the INSERT in route.ts and the schema at src/lib/db/schema.ts:358); src/inngest/functions/maintenance.ts:160-172 prunes activity_log but not this table. src/inngest/functions/dead-letter.ts:24-33 only catches events that reached Inngest and failed — a route-level 429/500 never reaches Inngest.

GitHub's documented behavior: GitHub does not automatically redeliver failed deliveries — a non-2xx response (400-599) marks the delivery as failure; redelivery is manual and only available for ~3 days.

Expected Behavior

The webhook endpoint returns 2xx fast for anything it can accept, only persists events the app cares about, and never lets a transient rate-limit/DB issue destroy business-critical events.

Actual Behavior

1. Real events get lost permanently. On a busy installation, the shared 100/min per-install bucket can be exhausted by noise events, so a pull_request (closed, merged) delivery gets 429 → GitHub marks it failed → never redelivered → handleMerge in process-pr-event.ts:343-403 never runs → XP never awarded and the recommendation never marked completed. The 14-day autoUnclaimStale job (maintenance.ts:364-433) then silently releases the contributor's claim, deleting their work. No reconciliation job re-fetches merged-PR state, so this XP loss is unrecoverable.
2. Unbounded table growth. Every push/star/comment writes a row to webhook_deliveries, forever, with no TTL/prune job.
3. 503 on missing secret (route.ts:18-20) and 500 on transient DB errors are also hard drops.

Affected Files

• src/app/api/webhooks/github/route.ts (lines 18-20, 52-67, 69-99, 101-119) — Rate limit, persist-all, dispatch-all, hard errors
• src/lib/db/schema.ts (line 358) — webhook_deliveries table
• src/inngest/functions/process-pr-event.ts (lines 343-403) — XP award that never runs on dropped PR events
• src/inngest/functions/maintenance.ts (lines 160-172, 364-433) — Prunes activity_log but not webhook_deliveries
• src/inngest/functions/dead-letter.ts (lines 24-33) — Only catches events that reached Inngest

Proposed Fix

1. Whitelist handled event types (e.g., pull_request, issues, pull_request_review, installation, installation_repositories) before rate limiting or persisting; return 200 immediately for everything else and don't INSERT
2. Split rate-limit buckets per event type (or per repo) so merge/issue events aren't starved by push noise
3. Always return 2xx for accepted-but-deferred work; remove the 500-on-send path in favor of relying on the persisted delivery row + retry
4. Add a maintenance job to prune webhook_deliveries older than N days, and a reconciliation job that periodically re-checks recently-merged PRs against GitHub to award missing XP