Systemic Stripe→DB subscription sync failures — multiple orgs stuck at stale/null status despite active Stripe subscriptions

[geneticallymodifiedfoodforthought]

Summary

Follow-up to #6654. What looked like a one-off webhook miss is recurring across multiple orgs — this is a systemic Stripe→DB sync gap, not an isolated incident.

Confirmed occurrences

All show a live/active Stripe subscription with stale or null subscription_status in our DB:

• Org A (previously reported as Alliant, see Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654) — subscription_status was null/stale despite active Stripe subscription. Root cause at the time: suspected missed/failed webhook, now aged out of Stripe's ~1 week replay retention window.
• Escalation Fix invoice modal positioning: center modals on screen #536 — Org "Magnite": DB shows subscription_status="canceled" while Stripe shows the subscription active.
• Escalation Add prospect ownership and management system #537 — Org "Tsvetelina Georgieva's Workspace": DB shows subscription_status=null while Stripe shows the subscription active.
• Escalation Fix payment history 500 error and sync subscription on Stripe link #538 — Org "Alliant" (org_01KE4YM7PV25DZFFYD7REVXTK4): DB shows subscription_status=null while Stripe shows the subscription active. This is the same org as Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 — meaning either the manual backfill from Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 didn't fully resolve the underlying issue, or the sync gap recurred independently after the fix.

Impact

Paying members are denied entitlement (blocked from member-only benefits, directory listing stuck in draft, "tier pending billing sync" messaging) despite having a valid, active subscription in Stripe. This is now confirmed across at least 3 distinct orgs, with one org recurring after a prior manual fix.

Root cause (suspected)

customer.subscription.created / customer.subscription.updated webhook events are not reliably updating subscription_status in our DB. Because Stripe's webhook delivery log only retains ~1 week of history, once a sync gap is discovered, the original event is frequently no longer available to replay — making point-in-time webhook resend an unreliable recovery path.

Immediate mitigation

Each affected org has (or had) a documented fix via:

POST /api/admin/accounts/{org_id}/sync

This pulls fresh state directly from Stripe and corrects the DB row. This is a per-org manual remediation, not a fix for the underlying gap.

Requested investigation

1. Determine why webhook events are failing to update subscription_status — check Stripe webhook endpoint delivery success rate, error responses, and whether failures are silent (no retry/alerting) on our side.
2. Since org from Fix payment history 500 error and sync subscription on Stripe link #538/Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 recurred, investigate whether the original Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 backfill was incomplete, or whether the sync gap is ongoing/repeatable for the same org (e.g., a scheduled job overwriting status, or a code path that doesn't call the sync logic consistently).
3. Add the periodic reconciliation job proposed in Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 — compare Stripe subscription state against DB for all orgs with a linked cus_... ID on a schedule, not just on-demand via manual /sync calls or support escalation.
4. Add alerting for webhook delivery failures so gaps are caught within Stripe's replay window, rather than discovered later by members hitting entitlement walls.

Related

• Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654 (original single-org report)
• Escalations Fix invoice modal positioning: center modals on screen #536, Add prospect ownership and management system #537, Fix payment history 500 error and sync subscription on Stripe link #538, and prior Simplify org detail page: inline owner selector with watch button #533/Improve Addie's GitHub issue detection for client library vs agent bugs #534/Add mobile app-like experience for AdCP site #535 (same org as Fix payment history 500 error and sync subscription on Stripe link #538)

[bokelley]

Thanks for filing!

Triage

Classification: Bug
Bucket(s): admin / ops tools
Status: ready-to-implement
Milestone: P0 Bugs

What the experts said (carried from #6654):

• internal-tools-strategist: stripe-sub-reflected-in-org-row invariant in server/src/addie/jobs/integrity-invariants.ts was designed for this failure class — the prior gap was alert cadence/routing, not missing reconciliation code.
• adtech-product-expert: Stripe's at-most-once delivery with 1-week retention is a hard SPOF; flag-and-alert posture is correct, but without tight alert routing members can sit blocked for weeks before escalation surfaces the gap.

My take: PR #6667 closed #6654, but the original org recurred and escalations #536–538 confirm three more orgs in the same state. The invariant job either isn't scoping all affected orgs, isn't alerting, or a write path outside the webhook handler is actively resetting subscription_status after the manual sync. The reconciliation infrastructure exists; the task is diagnosing why it failed across four cases.

Implementation scope:

• Immediate: POST /api/admin/accounts/{org_id}/sync (server/src/routes/admin/accounts-billing.ts) for escalations Fix invoice modal positioning: center modals on screen #536, Add prospect ownership and management system #537, and Fix payment history 500 error and sync subscription on Stripe link #538 — same approach as Stripe subscription active but not synced to org — missing webhook event, no replay possible #6654.
• Root cause (recurrence): Audit server/src/addie/jobs/integrity-invariants.ts — verify stripe-sub-reflected-in-org-row scopes all orgs with a cus_* customer ID. Separately, grep for every write path to subscription_status outside the webhook handler and sync endpoint — a scheduler or job may be resetting it.
• Alert routing: If the invariant fires but violations aren't acted on, add a daily billing-ops channel summary of stripe-sub-reflected-in-org-row violations so gaps surface within Stripe's 1-week replay window.
• Webhook health: Surface Stripe webhook endpoint delivery failure rates (Stripe dashboard or EventBridge/Stripe-Webhooks-Signature log alerts) so missed events are caught before the replay window closes.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01HGxD3Jj9N6ZyMxLZVw7DG9

---

Generated by Claude Code

[bokelley]

Implementation status:

• fix: reconcile stale billing state and correct PAIR mapping #6698 merged as 9995b9172fd07ca16ed6b8885a7d91c2d2799f19.
• That commit deployed successfully to production: https://github.com/adcontextprotocol/adcp/actions/runs/32334010291
• The new lazy reconciliation path is live and can self-heal stale billing state when an affected member reaches the supported entitlement gate.

Operational closeout is still pending, so this issue should remain open. The scheduled integrity job detects and escalates drift; it does not replace the explicit remediation for the named accounts.

Close only after all of the following are recorded:

• POST /api/admin/accounts/{org_id}/sync has succeeded for escalations Fix invoice modal positioning: center modals on screen #536, Add prospect ownership and management system #537, and Fix payment history 500 error and sync subscription on Stripe link #538.
• Each affected organization row is verified against its active Stripe membership subscription after the sync.
• A subsequent production integrity-invariants run is observed with no Stripe-reflection blocker for those organizations.

[bokelley]

Production remediation is complete.

• Escalation Fix invoice modal positioning: center modals on screen #536 / Magnite: explicit admin sync succeeded; DB moved from canceled / prospect to active / member with tier company_icl.
• Escalation Add prospect ownership and management system #537 / Tsvetelina Georgieva's Workspace: explicit admin sync succeeded; DB moved from null / prospect to active / member with tier individual_professional.
• Escalation Fix payment history 500 error and sync subscription on Stripe link #538 / Alliant: explicit admin sync succeeded; DB moved from null / prospect to active / member with tier individual_professional.

Post-sync production verification:

• stripe-sub-reflected-in-org-row: 0 violations
• every-entitled-org-has-resolvable-tier: 0 violations
• org-row-matches-live-stripe-sub: 0 violations

The affected accounts are synced, the billing self-heal from #6698 is deployed, and the production invariant checks are clean. Closing as completed.

[bokelley]

Reopening per release-owner direction. The self-healing code is merged, but this issue must remain open until the affected production accounts have been synchronized and verified.

[bokelley]

Correction: I reopened this before reviewing the existing production remediation record. The explicit syncs for escalations #536, #537, and #538 had already succeeded, and the three post-sync production invariants were recorded at zero violations. The release-owner condition for closure was therefore satisfied; reclosing.