perf: ENSIndexer Performance Memory

[shrugs]

This issue is used to track performance numbers for runs of ENSIndexer and correlate between plugins and changes and stuff like that. I'm using it as memory for my Claude

[shrugs]

ensv2-only mainnet baseline (main @ c38ad75)

Local hardware reference for the post-canonicality merge. ENSRainbow stubbed (labelByLabelHash short-circuits to null), LABEL_SET_ID=subgraph, fresh schema ensindexer_perf_ensv2only.

Wall-clock

Started	2026-05-08 16:36:42 -0500
Crashed (ENOSPC, 99.8%)	2026-05-09 08:04:49
Resumed (Ponder crash recovery)	2026-05-09 09:34:19
Realtime reached	2026-05-09 09:42:24
Active backfill (excl. 89m crash gap)	15h 36m
Total wallclock (incl. gap)	17h 5m

The crash was disk-fill on the host, not indexer-related. Ponder's start resumed cleanly from the last checkpoint (Detected crash recovery build_id=89f6dcb0a4 last_active=1h 29m 38s) and finished the remaining 0.2% in 8m 40s.

Throughput

Total events indexed (sum across chains)	158,415,613
Cumulative handler-time	28.76 hours
Active throughput	2,820 events/sec (events / active wall)
Effective throughput	2,575 events/sec (events / total wall)

Per-chain

chain	events	total handler-time	avg ms/event
8453 (Base)	84,286,147	8h 55m	0.38
1 (Mainnet)	62,510,569	13h 39m	0.79
59144 (Linea)	10,885,458	3h 04m	1.01
10 (Optimism)	733,198	3h 03m	14.99
42161 (Arbitrum)	185	199 s	1077.8
534352 (Scroll)	56	70 s	1246.8

The high per-event averages on Arbitrum (n=185) and Scroll (n=56) are distorted by sparse-range batches where the 1000+ms duration reflects RPC fetch wait rather than handler cost. Chain 10's 15ms/event is real — likely large-batch contention on indexer-stage replay. Chain 8453 + chain 1 dominate event volume (92.6% of all events).

Top handlers by total wallclock (resume-only, last 8m 40s)

Note: /metrics resets on process restart, so these are from the resume window only — directional, not cumulative.

handler	n	total	avg ms
ensv2/BaseRegistrar:Transfer	87 k	113 s	1.30
ensv2/ENSv1Registry:NewOwner	164 k	108 s	0.66
ensv2/BaseRegistrar:NameRenewed	7 k	21 s	2.91
ensv2/BaseRegistrar:NameRegisteredWithRecord	24 k	18 s	0.74
protocol-acceleration/StandaloneReverseRegistrar:NameForAddrChanged	24 k	11 s	0.43
protocol-acceleration/Resolver:VersionChanged	23	2 s	71.80

Notes

• ENSv1Registry:NewOwner at 0.66 ms/event is in line with the cc60281 sepolia run (post-canonicality): 0.141 ms/event on sepolia-v2, 0.279 ms/event on mainnet — within expected scaling for the larger mainnet domain set.
• No errors mid-run except the disk crash. After realtime, observed transient Invalid ChainIndexingStatusSnapshot: latestIndexedBlock must be before or same as backfillEndBlock errors — non-fatal and self-resolves as latestIndexedBlock advances past backfillEndBlock.
• DB at end: ensindexer_perf_ensv2only 218 GB; ponder_sync 217 GB.

Baseline established

This 15h 36m active / 2,820 events/sec is the new reference for mainnet ensv2-only on local hardware (M-series Mac, localhost Postgres) at main HEAD.

[shrugs]

materialized-name — feat/materialized-name @ cbc283ac9, plugins=ensv2,protocol-acceleration, namespace=mainnet, LABEL_SET_ID=subgraph v0, ENSRainbow labelByLabelHash stubbed → null, hardware=M-series Mac w/ localhost Postgres.

backfill-only — the run crashed in ponder's post-backfill create_indexes transition with index row requires 23448 bytes, maximum size is 8191 (strongly suggestive of a btree index on the new materialized name column exceeding the 8 KB leaf limit). realtime catch-up never measured.

• wallclock: 12h 15m 46s (2026-05-11 20:58:03 CDT → 2026-05-12 09:13:49 CDT, first progress=100% estimate=0ms)
• avg events/sec: ~3,597 (158,811,496 events / 44,146 s)
• top 5 by total wall-time (prom snapshot at 09:23:00, ts=1778595780; _count was not captured before teardown so no ms/event or p95):
  • ensv2/ENSv1Registry:NewOwner — 8,978,464 ms
  • ensv2/BaseRegistrar:Transfer — 3,361,818 ms
  • ensv2/BaseRegistrar:NameRenewed — 2,381,756 ms
  • ensv2/BaseRegistrar:NameRegistered — 2,009,410 ms
  • ensv2/BaseRegistrar:NameRegisteredWithRecord — 1,873,256 ms
• vs ensv2-only baseline main @ c38ad750b (15h 36m / 2,820 EPS / ~158.4M events):
  • wall −21.4%, EPS +27.6%
  • event volume +0.26% (block-range drift over 3 days; same work shape)
  • handler-mix shape unchanged; ENSv1Registry:NewOwner still dominates

caveats / followups:

• the create_indexes crash blocks the run from completing — index definition on the materialized-name column needs to change before a real a2a run is possible
• secondary noise signature pre-crash: Invalid ChainIndexingStatusSnapshot: latestIndexedBlock must be before or same as backfillEndBlock from apps/ensindexer/src/lib/indexing-status-builder/indexing-status-builder.ts:170, ~1/sec for 10 min before shutdown; did not cause the crash
• next run should snapshot ponder_indexing_function_duration_count + histogram_quantile(0.95, …) alongside _sum so per-event ms and p95 can be reported in this template
• treat the +28% backfill EPS as a strong positive signal, not shippable proof — realtime overhead on this branch is still unknown

[shrugs]

materialized-name — feat/materialized-name @ 38fca71a2, plugins ensv2,protocol-acceleration, mainnet

• wallclock: 13h 57m (2026-05-12 21:34:41 → 2026-05-13 11:31:40 -0500, 50,219s)
• avg events/sec: 3,163 (158,890,225 events / 50,219s)
• ENSRainbow labelByLabelHash stubbed for apples-to-apples vs baseline. label set: subgraph v0. local hardware: M-series Mac, localhost Postgres.

top 5 by wallclock:

• ensv2/ENSv1Registry:NewOwner — 10,154,391 ms (16,886,600 events, 0.601 ms/ea, p95 92 ms)
• ensv2/BaseRegistrar:Transfer — 3,575,436 ms (12,899,581 events, 0.277 ms/ea)
• ensv2/BaseRegistrar:NameRenewed — 2,400,526 ms (1,478,028 events, 1.624 ms/ea, p95 48 ms)
• ensv2/BaseRegistrar:NameRegisteredWithRecord — 2,320,016 ms (3,645,350 events, 0.636 ms/ea, p95 400 ms)
• ensv2/BaseRegistrar:NameRegistered — 2,012,786 ms (4,614,209 events, 0.436 ms/ea)

vs baseline c38ad750b (15h 36m / 2,820 EPS, 158.4M events):

• wallclock: -1h 39m (-10.6%)
• avg EPS: +343 (+12.2%)
• ensv2/ENSv1Registry:NewOwner: 0.66 → 0.601 ms/ea (-9.0%)

notable:

• despite adding materialized-name maintenance (canonicalName / canonicalLabelHashPath / canonicalNode columns on Domain, plus inline canonicality cascade in handlers), backfill-to-realtime is faster than baseline.
• the hash + GIN trgm indexes on long-text columns (Domain.canonicalName, Label.interpreted) — landed via fix: swap btree on label.interpreted for hash + GIN trgm #2096 — prevented the prior btree-size crash that aborted the materialized-name v1 run at the post-backfill create_indexes step (single row index entry of 23,448 bytes exceeded postgres btree leaf limit of 8191 bytes).
• a startup race in LocalPonderClient.metrics / getIndexingMetadataContext crashed the first attempt at this run within ~13s of launch (all 6 chains missing from the in-process Ponder metrics registry). non-deterministic — a clean retry won the race. not specific to this branch.

sepolia-v2 smoke test (same branch / SHA / plugins, namespace sepolia-v2): completed clean in 2m 48s, no create_indexes crash, transitioned cleanly to realtime.

[shrugs]

unigraph+protocol-acceleration mainnet — main @ bb950c798, plugins=unigraph,protocol-acceleration (unigraph is the renamed ensv2 plugin), namespace=mainnet, label set=subgraph/0, schema=ensindexer_perf_unigraph_pa

• wallclock: 14h 42m (2026-05-18 17:58:47 → 2026-05-19 08:41:12 -0500; 52,945s)
• avg events/sec: 3,015 (159,631,942 events across 6 chains: 1, 10, 8453, 42161, 59144, 534352)
• top 5 handlers by total wallclock:
  • unigraph/ENSv1Registry:NewOwner — 10,431,034 ms total (16.94M events, 0.616 ms/ea, p95 8.4 ms)
  • unigraph/BaseRegistrar:Transfer — 3,953,989 ms total (12.93M events, 0.306 ms/ea, p95 0.11 ms)
  • unigraph/BaseRegistrar:NameRenewed — 3,070,246 ms total (1.56M events, 1.972 ms/ea, p95 4.8 ms)
  • unigraph/BaseRegistrar:NameRegistered — 2,293,987 ms total (4.62M events, 0.497 ms/ea, p95 0.975 ms)
  • unigraph/BaseRegistrar:NameRegisteredWithRecord — 2,124,470 ms total (3.67M events, 0.578 ms/ea, p95 15.0 ms)
• vs ensv2-only mainnet baseline (c38ad750b, 15h 36m / 2,820 EPS): 54m faster (5.8%) and +195 EPS (+6.9%) despite adding the protocol-acceleration plugin (which doubles handler invocations on shared ENSv1Registry/Resolver events — see counts for unigraph/ENSv1Registry:NewOwner and protocol-acceleration/ENSv1Registry:NewOwner, both 16.94M). Improvement comes from interim commits between c38ad750b and bb950c798, not from a literally-faster plugin combo. Not strictly apples-to-apples.
• ENSRainbow heal stubbed (labelByLabelHash short-circuit), reverted post-snapshot
• realtime store qps at end-epoch: insert 13.6/s, find 4.2/s, update 1.06/s, delete 0.02/s
• handler outliers worth noting: unigraph/NameWrapper:TransferBatch at 156.8 ms/ea avg (rare event); unigraph/BaseRegistrar:NameRegisteredWithRecord p95 15 ms is the highest among top-5

raw json snapshots in /tmp/perf-test/main-unigraph_pa-*.json

[shrugs]

unigraph_pa — main @ addfba696 (#2155 merge), plugins unigraph,protocol-acceleration, namespace mainnet

run purpose: verify the two #2155 fixes (find-domains query perf + basenames/lineanames canonicality) against a clean reindex. ENSRainbow heal stubbed to null (apples-to-apples), subgraph label set.

indexing perf (on par with the ensv2-only baseline, no regression):

• wallclock: 15h 14m (2026-05-19 17:01 → 2026-05-20 08:15), all 6 chains live, 0 crashes
• total events: 159,777,412
• avg events/sec: 2,913
• top 5 by wallclock:
  • unigraph/ENSv1Registry:NewOwner — 10,336,894ms total (16,957,657 events, 0.610 ms/ea, p95 9.9ms)
  • unigraph/BaseRegistrar:Transfer — 4,023,743ms total (12,938,331 events, 0.311 ms/ea, p95 1.5ms)
  • unigraph/BaseRegistrar:NameRenewed — 3,121,487ms total (1,557,852 events, 2.004 ms/ea, p95 4.8ms)
  • unigraph/BaseRegistrar:NameRegistered — 2,305,026ms total (4,615,770 events, 0.499 ms/ea, p95 4.0ms)
  • unigraph/BaseRegistrar:NameRegisteredWithRecord — 2,079,164ms total (3,679,267 events, 0.565 ms/ea, p95 4.6ms)

find-domains query timings (criterion 2 — RESOLVED). warm, via /api/omnigraph, against the live index:

• domain-subdomains (.eth) — 7ms (pre-fix: 15.26s+)
• domain-by-name (eth) — 13ms (pre-fix: 2.69s)
• hello-world — 6ms
• find-domains (vitalik, NAME) — 18ms
• domain-resolver (vitalik.eth) — 23ms
• namegraph — 38ms
• find-domains (a, NAME) — 697ms (pre-fix ~11s w/ harmful partial idx)
• find-domains (a, DEPTH) — 917ms (known single-keystroke edge case)

all returned real data. new indexes present (domains_registry_id__id_index, domain_resolver_relations(domain_id)).

caveat: timings measured against an index where 8.6M basenames/lineanames are non-canonical (canonical_name NULL → excluded from trigram/hash scans). with canonicality fixed, the broad-prefix typeahead candidate set is ~2.8x larger, so a% could be slower than shown.

canonicality fix (criterion 1 — FAILED). identical to pre-fix state:

• total domains 13,576,997 / canonical 4,887,742 / non-canonical 8,689,255
• canonical .base.eth = 0, .linea.eth = 0
• 0 cross-chain bridge edges anywhere: mainnet base.eth/linea.eth.subregistryId = NULL, target L2 registries' canonicalDomainId = NULL
• base.eth's resolver IS the recognized BasenamesL1Resolver yet pointers null → suspected handleBridgedResolverChange ordering bug
• linea.eth's resolver is 0x1507ce…fbff, NOT the datasource's configured LineanamesL1Resolver 0xde16ee…679a → stale datasource address

schema ensindexer_perf_unigraph_pa left in place for further inspection.

[shrugs]

alpha + mainnet seed-tip checkpoints — 1.16.0 + per-chain end blocks

commit ef5f8818 = v1.16.0 (24e7f34ac) with feat/per-chain-end-blocks cherry-picked on top (the END_BLOCK_<chainId> feature isn't in the 1.16.0 release tag). Bare-metal Cherry Servers Ryzen 9 9900x (US-Chicago). Seed-tip zero-RPC reindex from the R2 ponder_sync cache, with real ENSRainbow heals (not heal-stubbed) — so EPS here is not apples-to-apples with the stubbed perf baselines above.

alpha — NAMESPACE=mainnet, all 8 plugins (subgraph,basenames,lineanames,threedns,unigraph,protocol-acceleration,registrars,tokenscope), labelset searchlight/1

• wallclock: 16h 24m 52s (Completed indexing across all chains)
• total events: 273,354,999 across 6 chains (1, 8453, 10, 42161, 59144, 534352)
• avg events/sec: ~4,630 (273.4M / 16h25m); ponder throughput meter peaked ~5,140
• output schema: 416 GB on-box (~308 GB fresh pg_restore)
• 0 RPC timeouts — fully cache-served
• seed-tip per-chain end blocks: 1=25353521, 8453=47553138, 10=153131791, 42161=474938261, 59144=31065135, 534352=33987536

mainnet — NAMESPACE=mainnet, PLUGINS=subgraph, SUBGRAPH_COMPAT=true, labelset subgraph/0, chain 1 only

• wallclock: 1h 56m 10s (chain-1 cache-served; paused + resumed once)
• total events: 23,856,957
• avg events/sec: ~3,420
• output schema: 39 GB
• END_BLOCK_1=25353521

notes

• EPS ramped from ~380 → ~5,000 as caches warmed (cold start), then held.
• No v1.15.2 → v1.16.0 backfill-throughput regression found in ensindexer. ponder unchanged (0.16.6 / viem 2.50.3 both tags); only 3 commits touched apps/ensindexer/src. The lone suspect (a new gin_trgm index, feat(ensapi): add Account.nameReferences Omnigraph field #2292) has zero backfill cost because Ponder builds secondary indexes after historical backfill, not during.
• These checkpoints were copied to the blue ENSDb + the reader/snapshot instance. Cross-instance gotcha: a checkpoint built from a patched/cherry-picked commit yields a different Ponder build-id than a stock-image indexer, so the consuming indexer's _ponder_meta.build_id must be patched to adopt the schema (also pg_dump hard-qualifies the gin_trgm_ops opclass to its source schema, so pg_trgm must live in the same schema — ensnode — on every instance).

[shrugs]

unigraph_20260717 (partial — killed for debugging) — feat/sov-contracts @ 67ec1349, plugins=unigraph,protocol-acceleration, namespace=mainnet, DISABLE_ENSRAINBOW_HEAL=true

• wallclock: 15h 16m to ~47% backfill (2026-07-17 17:40 CDT → killed 2026-07-18 08:57 CDT, mainnet at ~block 12.9M), 6.85M events. never reached realtime — killed after root-causing why.
• bottleneck: cascadeLabelHeal (canonicality-db-helpers) on RegistrarController:NameRegistered/NameRenewed. the canonical_label_hash_path @> ARRAY[labelHash] predicate has no GIN index during backfill (ponder applies schema indexes at realtime only — the docstring even warns about this) → parallel seq scan over the entire domains table per cascade: measured 193ms at 657k rows, grows linearly with table size. with heal disabled, every controller registration cascades: the same-tx NewOwner creates the label as unknown, then the controller's emitted label string "heals" it.
• lifetime avgs at kill: NameRegistered 232 ms/event (32.2M ms total over 139k events), NameRenewed 125 ms/event (11.0M ms over 88k). those two = ~12h of the 15h wallclock.
• live experiment: CREATE INDEX CONCURRENTLY ... USING gin (canonical_label_hash_path) on the running schema (37s build at 657k rows) → predicate 193ms → 0.16ms; NameRegistered 258 → 44 ms/event (10m windows); total eps 65 → ~300.
• residual ~44ms: context.ensDb.sql with a non-select statement makes ponder flush() + invalidate() + clear() the entire indexing cache per cascade (indexing-store/index.ts). also taxes neighboring handlers — every find falls through to postgres until the cache re-warms (e.g. ENSv1RegistryOld:NewResolver lifetime avg 33 ms/event on pure-PK ops). a read-only raw sql only flushes the referenced tables and keeps the cache.
• production relevance: with heal enabled the cascade fires for every label absent from the pinned label set — i.e. every registration/renewal after the label-set snapshot — so long-running mainnet indexes hit this exact wall in the tail years, worsening as domains grows (at 3M+ rows the seq scan alone is ~1s/event).
• proposed fixes: (1) create the GIN index at backfill time (idempotent CREATE INDEX IF NOT EXISTS in initIndexingOnchainEvents; note ponder replays drizzle index statements verbatim without IF NOT EXISTS at realtime, so the backfill index needs a distinct name or the schema-defined index should move out of drizzle). (2) restructure cascadeLabelHeal to a read-only SELECT (partial flush, no cache clear) that fetches affected domain ids + paths via the GIN index (typically 1 row), then re-render names through cached PK finds/updates.
• raw snapshots in /tmp/perf-test/main-unigraph_20260717-*.json (local); schema ensindexer_perf_unigraph_20260717 left in place for inspection.
• caveats: warm ponder_sync rpc cache (71–97% per chain); DISABLE_ENSRAINBOW_HEAL maximizes cascade frequency vs production; GIN index added mid-run at ~47%, so lifetime avgs mix pre/post-index regimes.