Improve the default configuration for RPC endpoints and rate limits

[tk-o]

Logs

Attaching to ensindexer, ensadmin-1, ensrainbow, postgres
postgres    | The files belonging to this database system will be owned by user "postgres".
postgres    | This user must also own the server process.
postgres    |
postgres    | The database cluster will be initialized with locale "en_US.utf8".
postgres    | The default database encoding has accordingly been set to "UTF8".
postgres    | The default text search configuration will be set to "english".
postgres    |
postgres    | Data page checksums are disabled.
postgres    |
postgres    | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres    | creating subdirectories ... ok
postgres    | selecting dynamic shared memory implementation ... posix
postgres    | selecting default "max_connections" ... 100
postgres    | selecting default "shared_buffers" ... 128MB
postgres    | selecting default time zone ... Etc/UTC
postgres    | creating configuration files ... ok
postgres    | running bootstrap script ... ok
ensrainbow  |
ensrainbow  | > @ensnode/ensrainbow@0.1.0 serve /app/apps/ensrainbow
ensrainbow  | > bun src/cli.ts serve
ensrainbow  |
ensadmin-1  | {"level":"info","ts":1741110729.2306805,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
ensadmin-1  | {"level":"info","ts":1741110729.231133,"msg":"adapted config to JSON","adapter":"caddyfile"}
ensadmin-1  | {"level":"warn","ts":1741110729.231141,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
ensadmin-1  | {"level":"info","ts":1741110729.2315724,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
ensadmin-1  | {"level":"warn","ts":1741110729.2318718,"logger":"http","msg":"HTTP/2 skipped because it requires TLS","network":"tcp","addr":":4173"}
ensadmin-1  | {"level":"warn","ts":1741110729.2318804,"logger":"http","msg":"HTTP/3 skipped because it requires TLS","network":"tcp","addr":":4173"}
ensadmin-1  | {"level":"info","ts":1741110729.231882,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
ensadmin-1  | {"level":"info","ts":1741110729.2321897,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x4000581b00"}
ensadmin-1  | {"level":"info","ts":1741110729.2353382,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
ensadmin-1  | {"level":"info","ts":1741110729.2353513,"msg":"serving initial configuration"}
ensadmin-1  | {"level":"info","ts":1741110729.2393878,"logger":"tls","msg":"cleaning storage unit","storage":"FileStorage:/data/caddy"}
ensadmin-1  | {"level":"info","ts":1741110729.240437,"logger":"tls","msg":"finished cleaning storage units"}
postgres    | performing post-bootstrap initialization ... ok
postgres    | initdb: warning: enabling "trust" authentication for local connections
postgres    | initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
postgres    | syncing data to disk ... ok
postgres    |
postgres    |
postgres    | Success. You can now start the database server using:
postgres    |
postgres    |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres    |
postgres    | waiting for server to start....2025-03-04 17:52:09.409 UTC [48] LOG:  starting PostgreSQL 17.4 (Debian 17.4-1.pgdg120+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres    | 2025-03-04 17:52:09.410 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres    | 2025-03-04 17:52:09.412 UTC [51] LOG:  database system was shut down at 2025-03-04 17:52:09 UTC
postgres    | 2025-03-04 17:52:09.416 UTC [48] LOG:  database system is ready to accept connections
ensrainbow  | Failed to find Response internal state key
ensrainbow  | {"level":30,"time":1741110729436,"pid":25,"hostname":"c588e4354456","msg":"ENS Rainbow server starting on port 3223..."}
ensrainbow  | {"level":30,"time":1741110729436,"pid":25,"hostname":"c588e4354456","msg":"Opening existing database in directory: /app/apps/ensrainbow/data"}
ensrainbow  | {"level":30,"time":1741110729448,"pid":25,"hostname":"c588e4354456","msg":"Starting database validation (lite mode)..."}
ensrainbow  | {"level":30,"time":1741110729449,"pid":25,"hostname":"c588e4354456","msg":"Precalculated rainbow record count: 133856894"}
postgres    |  done
postgres    | server started
postgres    | CREATE DATABASE
postgres    |
postgres    |
postgres    | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres    |
postgres    | waiting for server to shut down...2025-03-04 17:52:09.553 UTC [48] LOG:  received fast shutdown request
postgres    | .2025-03-04 17:52:09.553 UTC [48] LOG:  aborting any active transactions
postgres    | 2025-03-04 17:52:09.554 UTC [48] LOG:  background worker "logical replication launcher" (PID 54) exited with exit code 1
postgres    | 2025-03-04 17:52:09.554 UTC [49] LOG:  shutting down
postgres    | 2025-03-04 17:52:09.555 UTC [49] LOG:  checkpoint starting: shutdown immediate
postgres    | 2025-03-04 17:52:09.569 UTC [49] LOG:  checkpoint complete: wrote 921 buffers (5.6%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.004 s, sync=0.010 s, total=0.015 s; sync files=301, longest=0.003 s, average=0.001 s; distance=4238 kB, estimate=4238 kB; lsn=0/1908950, redo lsn=0/1908950
postgres    | 2025-03-04 17:52:09.571 UTC [48] LOG:  database system is shut down
postgres    |  done
postgres    | server stopped
postgres    |
postgres    | PostgreSQL init process complete; ready for start up.
postgres    |
postgres    | 2025-03-04 17:52:09.664 UTC [1] LOG:  starting PostgreSQL 17.4 (Debian 17.4-1.pgdg120+2) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
postgres    | 2025-03-04 17:52:09.664 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres    | 2025-03-04 17:52:09.664 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres    | 2025-03-04 17:52:09.665 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres    | 2025-03-04 17:52:09.666 UTC [64] LOG:  database system was shut down at 2025-03-04 17:52:09 UTC
postgres    | 2025-03-04 17:52:09.667 UTC [1] LOG:  database system is ready to accept connections
ensindexer  | 5:52:09 PM INFO  telemetry  Ponder collects anonymous telemetry data to identify issues and prioritize features. See https://ponder.sh/docs/advanced/telemetry for more information.
ensindexer  | 5:52:10 PM INFO  build      Using Postgres database postgres:5432/ponder (from DATABASE_URL env var)
ensindexer  | 5:52:10 PM INFO  database   Using database schema 'ens'
ensindexer  | 5:52:10 PM INFO  database   Created tables [domains, accounts, resolvers, registrations, wrapped_domains, transfers, new_owners, new_resolvers, new_ttls, wrapped_transfers, name_wrapped, name_unwrapped, fuses_set, expiry_extended, name_registered, name_renewed, name_transferred, addr_changed, multicoin_addr_changed, name_changed, abi_changed, pubkey_changed, text_changed, contenthash_changed, interface_changed, authorisation_changed, version_changed]
ensindexer  | 5:52:10 PM INFO  server     Started listening on port 42069
ensindexer  | 5:52:10 PM INFO  server     Started returning 200 responses from /health endpoint
postgres    | 2025-03-04 17:52:10.710 UTC [70] ERROR:  relation "public.kysely_migration" does not exist at character 15
postgres    | 2025-03-04 17:52:10.710 UTC [70] STATEMENT:  SELECT * FROM public.kysely_migration LIMIT 1
ensindexer  | 5:52:11 PM INFO  historical Started '1' historical sync with 0% cached
ensindexer  | 5:52:11 PM INFO  historical Started '59144' historical sync with 0% cached
ensindexer  | 5:52:11 PM INFO  historical Started '8453' historical sync with 0% cached
ensindexer  | 5:52:16 PM INFO  app        Indexed 1 events with 0.0% complete and 4739h 15m 55s remaining
ensrainbow  | {"level":30,"time":1741110736358,"pid":25,"hostname":"c588e4354456","msg":"Successfully healed labelhash 0x4f5b812789fc606be1b3b16908db13fc7a9adf7ca72641f84d75b47069d3d7f0 to label \"eth\""}
ensrainbow  | {"level":30,"time":1741110736362,"pid":25,"hostname":"c588e4354456","msg":"Successfully healed labelhash 0xdec08c9dbbdd0890e300eb5062089b2d4b1c40e3673bbccb5423f7b37dcf9a9c to label \"reverse\""}
ensrainbow  | {"level":30,"time":1741110736363,"pid":25,"hostname":"c588e4354456","msg":"Successfully healed labelhash 0xe5e14487b78f85faa6e1808e89246cf57dd34831548ff2e6097380d98db2504a to label \"addr\""}
ensindexer  | 5:52:16 PM INFO  app        Indexed 4 events with 0.0% complete and 7h 33m 09s remaining
ensindexer  | 5:53:19 PM WARN  rpc        Failed eth_getLogs request after 10 attempts
ensindexer  | HttpRequestError: HTTP request failed.
ensindexer  |
ensindexer  | Status: 400
ensindexer  | URL: https://eth.drpc.org/
ensindexer  | Request body: {"method":"eth_getLogs","params":[{"fromBlock":"0x32f056","toBlock":"0x331892","topics":["0xd8c9334b1a9c2f9da342a0a2b32629c1a229b6445dad78947f674b44444a7550"]}]}
ensindexer  |
ensindexer  | Details: {"message":"ranges over 10000 blocks are not supported on freetier","code":13}
ensindexer  | Version: viem@2.22.13
ensindexer  |     at Object.request (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/rpc/http.ts:148:17)
ensindexer  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
ensindexer  |     at fn (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/clients/transports/http.ts:129:19)
ensindexer  |     at request (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/clients/transports/http.ts:134:39)
ensindexer  |     at withRetry.delay.count.count (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/buildRequest.ts:130:22)
ensindexer  |     at attemptRetry (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/promise/withRetry.ts:44:22)
ensindexer  | 5:53:19 PM ERROR sync       Fatal error: Unable to sync '1' from 3338326 to 3348626.
ensindexer  | HttpRequestError: HTTP request failed.
ensindexer  |
ensindexer  | Status: 400
ensindexer  | URL: https://eth.drpc.org/
ensindexer  | Request body: {"method":"eth_getLogs","params":[{"fromBlock":"0x32f056","toBlock":"0x331892","topics":["0xd8c9334b1a9c2f9da342a0a2b32629c1a229b6445dad78947f674b44444a7550"]}]}
ensindexer  |
ensindexer  | Details: {"message":"ranges over 10000 blocks are not supported on freetier","code":13}
ensindexer  | Version: viem@2.22.13
ensindexer  |     at Object.request (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/rpc/http.ts:148:17)
ensindexer  |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
ensindexer  |     at fn (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/clients/transports/http.ts:129:19)
ensindexer  |     at request (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/clients/transports/http.ts:134:39)
ensindexer  |     at withRetry.delay.count.count (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/buildRequest.ts:130:22)
ensindexer  |     at attemptRetry (file:///app/node_modules/.pnpm/viem@2.22.13_typescript@5.7.3_zod@3.24.2/node_modules/viem/utils/promise/withRetry.ts:44:22)
ensindexer  | 5:53:19 PM WARN  process    Received fatal error, starting shutdown sequence
ensindexer exited with code 1
Gracefully stopping... (press Ctrl+C again to force)

[tk-o]

Possibly we need to apply rate-limiting configuration on our indexer end to avoid getting rate-limited by the RPC provider (which ends up in runtime error and indexer shutting down).

One idea is to use rateLimit function to wrap network transports object:
https://ponder.sh/docs/utilities/transports#usage-in-ponder-1

[lightwalker-eth]

@tk-o Great job flagging this 👍

Perhaps we should remove any default values for RPC endpoints such that:

1. Users are required to provide their own RPC endpoint URLs.
2. We verify that we provide very helpful error messages if a required RPC endpoint URL is not provided or is not in a valid format.
3. In our example .env file and also in our docs (such as https://www.ensnode.io/ensnode/usage/configuration/) provide explicit instructions that ENSNode requires a high volume of RPC calls and a paid RPC service provider (such as https://drpc.org/ and https://www.alchemy.com/). We should also communicate that any use of free RPC providers will likely fail due to rate limiting or other constraints.

[lightwalker-eth]

@tk-o Suggest we not try to fix this issue through smarter rate limiting logic.

Anyone deploying their own ENSNode instance should be using a paid RPC plan. They will have a terrible experience if they try to make it work using rate limited free RPC calls. That might take forever!

[lightwalker-eth]

@tk-o Please also include a link to https://www.quicknode.com/ in the list of suggested rpc providers.

[lightwalker-eth]

The following summarizes the known open tasks required to close this issue:

• Clean up how we define RPC endpoints in the ENSIndexer .ens.local.example file.
  • KEEP: information in comments with specific examples of how RPC endpoints must be manually configured.
  • KEEP: information in comments with warnings about how each of the configured RPC endpoints need to be prepared to receive and quickly process millions of RPC requests.
  • REMOVE: any default RPC endpoint values that are not in comments. It must be impossible to even start ENSIndexer until the operator explicitly and manually configures their RPC endpoints.
• Increase all of the default RPC request rate limits.
  • 20 is CRAZY low. We should increase all to 500.
  • Cleanup any comments about the use of public RPC endpoints. We must explicitly stop recommending any use of public (rate limited) RPC endpoints. That creates a terrible onboarding experience. We must explicitly communicate how public RPC endpoints will not provide the acceptable performance and that only private (paid) RPC endpoints should be used.

[tk-o]

Regarding the rate limits:

Since Ponder 0.11, the provider rate limits settings were deprecated and replaced with Ponder's auto-adjusted settings: https://ponder.sh/docs/config/chains#rpc-endpoints