events: denormalize kind onto event_tags so tag-CTE can pre-filter by kind

[MastaP]

Fixes #23.

Why

Production groupchat relay (`sphere-zooid-relay-eu`) is at 99.7% CPU on db.m6g.large because two NIP-29 group-history queries account for ~65% of all DBLoad. The materialized tag CTE filters by tag only — it can't see the `kind IN (...)` predicate.

Event distribution in production:

kind	count	what
9021	90,437	NIP-29 join request
9000	90,024	NIP-29 add member
9	11,365	chat message
other	~3k	group state, etc.

So for a hot group like `general`: ~97k events tagged `h='general'` (95k membership + 1.4k chat). A subscription read for `{kinds: [9,11,12], #h: ['general']}` materializes 97k tag rows, hash-joins, throws away 95% on the kind filter, sorts, takes 1000. ~1.1s per query (verified by EXPLAIN ANALYZE).

50k users × ~2 membership events per join is what tipped the balance. Same query was fine at 45k users.

What

• `event_tags` gets a nullable `kind` column.
• `saveEventWith` writes kind alongside (event_id, key, value); batch size unchanged at 15000 (60000 params, under 65535 cap).
• `buildSelectQuery` pushes `kind IN (...) OR kind IS NULL` into each per-tag CTE branch when `filter.Kinds` is set. The `IS NULL` branch keeps reads correct on un-backfilled rows during rollout.
• `Init` is split into pre-migrate and post-migrate phases. The new `(key, value, kind, event_id)` covering index lives in post-migrate so it runs only after migration 002 adds the column on existing schemas.
• Migration 002: `ALTER TABLE event_tags ADD COLUMN IF NOT EXISTS kind INTEGER`. Instant on PG 11+.

The new index creation and the historical-row backfill are intentionally not in the auto-migration. The runner enforces a 30s per-statement deadline; `CREATE INDEX CONCURRENTLY` cannot run inside a transaction and leaves an INVALID index on timeout that `IF NOT EXISTS` would silently skip on retry. So those run as one-shot ops via the dbops task — see Rollout below.

Once the CTE returns 1.4k rows instead of 97k for hot groups, expected per-query cost drops from ~1.1s to ~10–50ms.

Verification

`go test -timeout 300s ./...` passes; `gofmt -l -s .` clean. Four new integration tests:

• `TestEventStore_SaveEvent_PopulatesTagKind` — kind column populated on insert.
• `TestEventStore_QueryEvents_TagAndKindCTE` — 3 chat + 5 membership events tagged with the same group; asserts the kind filter excludes membership at the CTE.
• `TestEventStore_Init_UpgradeFromPre002` — the test that earns its keep. Drops the kind column + index + migration marker on a freshly-initialized schema, then re-runs `Init()`. Asserts column and post-migrate index both come back. Fresh-schema tests would not catch the original ordering bug (CREATE TABLE rebuilds the column), so this simulates the production upgrade path explicitly.
• `TestEventStore_QueryEvents_TagKindNullCompat` — clears `kind` to NULL on a saved event's tag rows and asserts the `OR IS NULL` branch keeps it visible.

Rollout (production schema with 438k existing tag rows)

Run these from the dbops task before merging this PR, so the column and index exist before the new code path lands:

```sql
-- 1. Add column (instant, metadata-only on PG 11+).
ALTER TABLE sphere_relay__event_tags ADD COLUMN IF NOT EXISTS kind INTEGER;

-- 2. Backfill historical rows. Idempotent — re-runnable if interrupted.
UPDATE sphere_relay__event_tags et
SET kind = e.kind
FROM sphere_relay__events e
WHERE et.event_id = e.id AND et.kind IS NULL;

-- 3. Build the new covering index (does NOT block writes).
CREATE INDEX CONCURRENTLY IF NOT EXISTS sphere_relay__idx_event_tags_key_value_kind_event_id
ON sphere_relay__event_tags(key, value, kind, event_id);
```

After merge, `Init()` will see the column and index already exist (IF NOT EXISTS is idempotent), the migration marker will be set on first run, and the new CTE shape goes live with full plan support.

Follow-ups (not in this PR)

• Drop the `OR kind IS NULL` branch once production backfill is verified complete and any remaining NULL rows are reconciled.
• Consider whether the eventual storage of all 90k+ membership events as durable history is the right design vs. a snapshot/projection. That's a NIP-29 design conversation, not an immediate fix.

[gemini-code-assist]

Code Review

This pull request optimizes tag-based queries by denormalizing the kind column into the event_tags table and introducing a new covering index. The changes include updates to the schema initialization, event saving logic, and the query builder to improve performance for groups with many membership events. Feedback focuses on improving the database migration safety by using concurrent index creation, removing a now-redundant index, refactoring duplicated kind-conversion logic, and ensuring errors from the SQL builder are properly handled.

[Copilot]

Pull request overview

This PR denormalizes events.kind onto event_tags so the tag-filter MATERIALIZED CTE can pre-filter by kind, significantly reducing work for hot NIP-29 group-history queries dominated by membership-event tag rows (issue #23).

Changes:

• Add nullable event_tags.kind via migration 002 and populate it on the write path (saveEventWith).
• Push kind IN (...) OR kind IS NULL into each per-tag CTE branch when filter.Kinds is set, preserving correctness during backfill rollout.
• Split EventStore.Init() into pre-/post-migrate phases and add integration tests covering insert population, upgrade ordering, and NULL-compat behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
zooid/migrations/002_event_tags_kind.sql	Adds nullable kind column to __event_tags and documents rollout considerations.
zooid/events.go	Writes kind into event_tags, pushes kind predicate into tag CTE branches, and adds a post-migrate covering index creation step.
zooid/events_test.go	Adds integration tests for tag-kind population, kind/NULL compatibility, and pre-002 upgrade path behavior.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

zooid/events.go:299

• buildSelectQuery now returns (squirrel.SelectBuilder, error), but zooid/query_performance_test.go still calls it as if it returned only a builder (e.g., in explainAnalyze). This will break compilation/tests; update that helper (and any other remaining call sites) to handle the new error return.

func (events *EventStore) buildSelectQuery(filter nostr.Filter) (squirrel.SelectBuilder, error) {
	eventsTable := events.Schema.Prefix("events")
	eventTagsTable := events.Schema.Prefix("event_tags")

	// Collect valid single-letter tag filters and sort for deterministic SQL.