perf: serve dashboards from worker-maintained hourly aggregates

[jrhizor]

Replaces every analytics chart and stat on the overview, visibility, citations, and prompt-detail pages with reads against four worker-maintained hourly_* aggregate tables. Page wall time at 90-day lookbacks drops from tens of seconds (with multi-second tails) to well under a second.

Three commits:

1. docs: — design doc (`docs/perf-daily-aggregates.md`)
2. perf: — implementation (schema + migration + worker + backfill + read-path rewrite + changeset)
3. perf: — thread the browser timezone end-to-end + add side-by-side benchmark script

Why a worker, not a Postgres materialized view

The "obvious" Postgres-native alternative would be a materialized view. We're not using one because:

1. `REFRESH MATERIALIZED VIEW` rebuilds the entire view from scratch. No incremental refresh in vanilla Postgres; `pg_ivm` exists but Supabase doesn't ship it. A per-minute refresh would scan all 12.6 M citations rows every minute. Our worker only re-aggregates the (brand, date) buckets that actually had inserts since the last tick — typically 1–3 buckets, ~150 ms of work.
2. `REFRESH … CONCURRENTLY` is more work, not less. It computes the new contents and diffs against the old.
3. Non-CONCURRENT `REFRESH` takes `ACCESS EXCLUSIVE` on the view. Every dashboard hangs during the refresh.
4. No partial state, no resumable backfill. "Backfill" with a matview is one giant `REFRESH` that runs to completion or doesn't. With a regular table we own the cursor and the per-bucket transactions.
5. Schema evolution is destructive. Adding a column means `DROP MATERIALIZED VIEW … CREATE … REFRESH …`, which loses data and re-computes from scratch.
6. Four derived datasets, one source-side scan budget. Four matviews would each refresh independently, each scanning the source from scratch. The worker does all four DELETE+INSERTs against the same already-warm source pages.
7. Triggers would solve the staleness/scan problems but slow the write path on every prompt-run insert. Out of scope.

Same explanation lives in `docs/perf-daily-aggregates.md`.

Schema (migration `0009_hourly_aggregates.sql`)

Table	Grain	Drives
`hourly_prompt_runs`	(brand, prompt, hour, model, web_search)	overview / visibility / prompt-detail
`hourly_prompt_run_competitors`	(brand, prompt, hour, model, competitor)	per-prompt competitor charts
`hourly_citations`	(brand, prompt, hour, model, domain)	citations chart, domain rollups
`hourly_citation_urls`	(brand, prompt, hour, model, url)	top-URL list, per-prompt URL stats
`aggregate_refresh_state`	singleton	worker watermark + resumable backfill cursor

Each `hourly_prompt_runs` bucket carries `first_run_at` / `last_run_at` so the existing "Last updated X ago" footer keeps its precise timestamp via `max(last_run_at)`.

Hourly chosen over daily because measured against production data, hourly buckets cost only 1.01× the row count of daily — runs of the same (brand, prompt, model) on a given day cluster in the same UTC hour because the worker batches them. Hourly granularity is what lets every chart re-bucket to the viewer's browser TZ at read time via `(hour AT TIME ZONE $tz)::date`.

Worker (`apps/worker/src/jobs/`)

• `rebuild-hourly-bucket.ts` — shared DELETE+INSERT helper for one (brand_id, UTC date) bucket across all four tables. Used by both the live worker and the backfill script, so the bucket-rebuild logic has one canonical implementation.
• `refresh-hourly-aggregates.ts` — pg-boss handler. Per tick: take advisory lock, read `last_refreshed_through`, find affected (brand, date) buckets in `[watermark - 1h, now() - 30s]` (1-hour overlap absorbs late writes), rebuild each, advance watermark. Whole tick in one transaction.
• Wired into `apps/worker/src/index.ts` with `boss.schedule("refresh-hourly-aggregates", "* * * * *", …, { singletonKey: "refresh-hourly-aggregates" })` so a tick is skipped if the previous one is still running.

Backfill — resumable

`apps/worker/src/scripts/backfill-hourly-aggregates.ts`. Run with:

cd apps/worker
pnpm tsx --env-file=../web/.env src/scripts/backfill-hourly-aggregates.ts

Tracks progress in `aggregate_refresh_state.(backfill_started_at, backfill_cursor_brand_id, backfill_cursor_date, backfill_completed_at)`. If the script crashes, gets killed by a deploy, or hits any transient failure, re-running it picks up at the next (brand, date) tuple after the cursor. Snapshot cutoff (`backfill_started_at - 30 s`) is sticky across resumes.

On completion the script primes `last_refreshed_through = backfill_started_at`, so the live worker only catches up the (small) gap from backfill start to deploy on its first tick.

Estimated runtime: ~15–20 min on the current data volume.

Read-path rewrite (`apps/web/src/lib/postgres-read.ts`)

16 analytics functions rewritten to source from `hourly_*`. TypeScript signatures unchanged; returned shapes unchanged. Charts re-bucket the UTC `hour` column to the viewer's browser TZ via `(hour AT TIME ZONE $tz)::date` — Postgres's native handling correctly covers half-hour TZs like IST/ACST.

Single-prompt detail page queries also moved (the `(prompt_id, hour)` indexes serve those by `prompt_id` filter).

`getPromptWebQueriesForMapping` / `getPromptWebQueryCounts` stay on raw — `web_queries` is a `text[]` and would explode the aggregate row count for a low-traffic single-prompt query.

Browser TZ wired end-to-end

Four server functions previously hardcoded `timezone = "UTC"` and silently ignored what the client sent (or didn't send). All four now accept `timezone` in their input validator, and their corresponding client hooks send `Intl.DateTimeFormat().resolvedOptions().timeZone` so charts bucket correctly in the viewer's local time:

• `getDashboardSummaryFn` ↔ `use-dashboard-summary`
• `getCitationsFn` ↔ `use-citations`
• `getPromptsSummaryFn` ↔ `use-prompts-summary`
• `getPromptStatsFn` ↔ `use-prompt-stats`

The visibility / batch-chart / per-prompt-chart functions already had this wiring.

Benchmark + correctness check

`scripts/perf/bench-aggregate-vs-raw.ts` is a self-contained script that runs the OLD raw-table SQL and the NEW aggregate SQL side-by-side for every page-driving query. It:

1. Pins both paths to `aggregate_refresh_state.last_refreshed_through` as the upper bound, so they read the same source set regardless of how far behind the worker / backfill is.
2. Times both paths (median of 3 runs per query).
3. Diffs the canonical row hashes — exits non-zero if anything mismatches.
4. Prints raw_ms / agg_ms / speedup per (page × lookback) pair, plus an overall ✓/✗ summary.

Run after backfill completes (and ideally before merging the read-path cutover) to verify both correctness and speedup.

Required follow-up

Once backfill is verified via the benchmark, drop the four backfill cursor columns from `aggregate_refresh_state`:

ALTER TABLE aggregate_refresh_state
    DROP COLUMN backfill_started_at,
    DROP COLUMN backfill_completed_at,
    DROP COLUMN backfill_cursor_brand_id,
    DROP COLUMN backfill_cursor_date;

The live worker doesn't read or write these columns; they're only used during the one-time backfill.

Deploy order

1. Apply migration `0009_hourly_aggregates.sql` (creates the tables, seeds the singleton state row).
2. Run the backfill script to completion (resumable, see above).
3. Run the benchmark `scripts/perf/bench-aggregate-vs-raw.ts` and confirm everything is ✓ and the speedups look right.
4. Apply the column-drop migration for the backfill cursor.
5. Merge this PR — the worker starts running on next deploy and the read paths cut over to the new tables.

Steps 1–4 don't change any user-visible behavior (the new tables are unread until step 5). The worker's first tick after step 5 catches up the gap between backfill end and deploy.

Test plan

• Apply migration on a non-prod DB; verify the four tables and singleton row exist.
• Run backfill on non-prod; verify it converges and the resume path works (kill mid-run, re-launch, watch it pick up).
• Run `scripts/perf/bench-aggregate-vs-raw.ts` on non-prod; confirm all-✓ and reasonable speedups (expect ~10–100× at 30 d / 90 d).
• Apply column-drop migration on non-prod; confirm worker tick still succeeds.
• Repeat 1–4 on prod.
• After merge: verify worker tick logs (`[refresh-hourly-aggregates] tick complete (source=scheduled, buckets=N, M ms)`) and that `last_refreshed_through` advances each minute.
• Smoke the overview / visibility / citations / prompt-detail pages at 7 d / 30 d / 90 d / 1 y for a high-volume brand and confirm they're sub-second.
• Verify charts bucket correctly in non-UTC browser timezones (e.g. `America/Los_Angeles`, `Asia/Tokyo`).

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
elmo	Ready	Preview, Comment	Jun 3, 2026 5:53am

[jrhizor]

Need to remove the backfill columns after merging this.