feat(stateless-r2): custom-domain R2 target with HTTP/2, keeping the raw S3 endpoint path

AGENTS.md

@@ -40,7 +40,7 @@ The project uses nightly `2026-02-03` toolchain (edition 2024, rust-version 1.95
 | `stateless-db`         | `crates/stateless-db`         | redb-backed persistence: table definitions, read/write helpers, `ContractCache`                                                                                                          |
 | `stateless-common`     | `crates/stateless-common`     | RPC client, metrics/logging utilities, witness size estimation                                                                                                                           |
 | `stateless-test-utils` | `crates/stateless-test-utils` | Test fixtures (blocks, witnesses, contracts) and env-var lock for integration tests                                                                                                      |
-| `stateless-r2`         | `crates/stateless-r2`         | Shared R2 (S3) witness primitives: SigV4 signer, object-key layout, endpoint parsing, signed PUT, and the retrying witness-object GET fetcher; consumed by mega-reth's uploaders (write) and both binaries' R2 witness sources (read) |
+| `stateless-r2`         | `crates/stateless-r2`         | Shared R2 witness primitives: SigV4 signer, object-key layout, endpoint parsing, signed PUT, and the retrying witness-object GET fetcher over either the signed S3 API or an unsigned Cloudflare custom domain; consumed by mega-reth's uploaders (write) and both binaries' R2 witness sources (read) |
 | `stateless-validator`  | `bin/stateless-validator`     | Main binary: chain sync, parallel validation workers (`app.rs` / `workers.rs` / `main.rs`)                                                                                               |
 | `debug-trace-server`   | `bin/debug-trace-server`      | Standalone RPC server for debug/trace methods                                                                                                                                            |
 
@@ -125,6 +125,19 @@ Below the response cache, a bounded in-memory `BlockData` cache keyed by block h
 The cache pins 4 shards (largest cacheable entry = `max_bytes / 4` on any host), counts non-retained inserts, and drops an entry when a trace fails for a data-attributable reason (`TraceError::Data` — bad witness) while request-attributable failures (invalid tracer configs) never evict.
 In local cache mode with a `--witness-generator-endpoint` plus at least one fallback `--witness-endpoint`, request-serving witness fetches route by block age: blocks at least `--witness-local-window` blocks below the local tip skip the generator (which prunes beyond its `BACKUP` window) and fetch from the fallbacks; without the generator flag, witness endpoints are plain failover and routing is disabled.
 With the `--r2-*` flag group (endpoint, bucket, access key id, secret), every request-serving witness fetch tries a direct SigV4-signed R2 GET (light decode, capped at half the remaining witness budget) before the RPC chain, falling back on any failure; `--r2-max-concurrent-requests` caps R2 GETs separately from the RPC witness semaphore.
+`--r2-custom-domain` is the alternative R2 target (mutually exclusive with `--r2-endpoint`, rejected at startup by name): unsigned GETs of `/{key}` through a Cloudflare custom domain fronting the bucket, which negotiates HTTP/2 (many in-flight GETs multiplex over a few connections instead of holding one each against the h1.1-only S3 endpoint) and can serve the immutable witness objects from edge cache; optional `--r2-access-client-id`/`--r2-access-client-secret` attach Cloudflare Access service-token headers (rejected at startup on a non-loopback `http://` domain, and validated as header values there so a stray newline fails by name instead of becoming a per-GET retryable transport error).
+Leftover S3 flags alongside the domain are rejected by name rather than silently ignored, the client sends a `User-Agent` (Cloudflare's Browser Integrity Check 403s requests without one), and the configured target is published as `debug_trace_r2_target_info{target}` / `r2_target_info{target}` so the target-less R2 series can be attributed during a rollout.
+Every `--r2-*` coherence rule — empty values, target exclusion, leftovers, an incomplete S3 quad, the Access pair, the connection count, and tuning flags with no target — lives in `stateless_common::validate_r2_flags`, so both binaries give the same verdict in the same words; each error names the offending flag, which clap cannot do without its `error-context` feature.
+Version selection on the custom domain is pure ALPN (no `http2_prior_knowledge`, so the plaintext loopback path keeps working), which means a grey-clouded record, a non-Cloudflare origin, or a zone with HTTP/2 off degrades to HTTP/1.1 while the h2 tuning goes inert: the fetcher warns once with the protocol it actually got and publishes `..._r2_negotiated_http_version_info{version}`, and `pool_max_idle_per_host` is bounded so the h1.1 fallback cannot accumulate idle sockets that `pool_idle_timeout(None)` would never reap.
+One `reqwest::Client` holds exactly one HTTP/2 connection and hyper never opens a second to relieve a saturated one (a pooled h2 connection reports liveness rather than stream capacity, and its dispatch channel is unbounded), so the edge's per-connection stream limit — Cloudflare advertises 100 in the `SETTINGS_MAX_CONCURRENT_STREAMS` it sends on every connection (`CLOUDFLARE_MAX_CONCURRENT_STREAMS`; `nghttp -nv https://<domain>/` reads what a given zone offers) — is a per-process ceiling rather than a per-request one.
+`--r2-connections` (default 1) is what lifts it: it holds that many clients and picks one *per attempt*, so a retry leaves the connection that just failed, and one dropped connection no longer takes every in-flight GET down with it — which is the availability argument, and the one that matters in the validator's fallback-less R2 mode.
+The pick is work-conserving: a connection with a free permit, searched from a rotating cursor, so a GET is never queued behind a connection whose permits are held by a slow transfer while another sits idle, and one budget of `max` is not silently partitioned into `N` budgets of `max/N` (which queues distinctly worse at the same offered load). Only when every connection is full does a fetch wait, and it waits on the cursor's own pick rather than on whichever has the most room — under saturation that one is the connection that just dropped every GET riding it.
+`--r2-max-concurrent-requests` stays the cap across all of them, split evenly and rounded up (rounding down would leave some connection at zero permits and wedge every GET routed to it), so raising the connection count alone spreads the same concurrency thinner instead of raising the ceiling; the per-connection share is what must stay at or below the stream limit, and the fetcher warns at startup when it exceeds it.
+The count is published as `debug_trace_r2_connections` / `r2_connections`, and is rejected by name at zero, on a non-numeric or blank value, on the S3 target (HTTP/1.1 already opens a socket per in-flight GET there), and above the cap it divides — more connections than permits would leave some of them permanently idle.
+It travels as text and is parsed after clap, so a blank env line — what a templated env file renders for a variable a role does not set — is named rather than aborting startup through clap's unnamed value error, and stays inert on the validator under `--witness-source rpc`, where every `--r2-*` flag is deliberately unread.
+On the validator the shared semaphore is `--witness-max-concurrent-requests`, which sizes the RPC gateway too, so the two consumers trade off against each other.
+`stateless-common`'s shared JSON-RPC client pins `http1_only`: `stateless-r2` enables reqwest's `http2` feature and Cargo unifies it workspace-wide, which would otherwise move the multi-MB witness RPC payloads onto one non-adaptive h2 connection per host.
+Client-side routing, budgets, and fallback match the S3 target, but edge behavior is zone configuration: **a cache rule making these objects cacheable must set 404s to bypass cache**, or a pre-upload frontier miss gets pinned for the negative-cache TTL (stalling the validator's tip-following in its fallback-less R2 mode) and a cached 404 can false-fire the below-band `kind="missing"` bucket-integrity alarm.
 The bucket is the same store the public gateway reads and can lead the generator at the frontier (uploader and generator RPC server publish from different files), so frontier hits are real; the frontier band is a small near-tip window (`R2_FRONTIER_WINDOW`, 32 blocks of uploader-lag grace on either side of the local tip — deliberately far narrower than the 4096-block routing window, so a stale catching-up tip cannot silence holes above it), hits there are labeled `witness_r2_frontier` (vs `witness_r2` past the band), the speculative frontier probe runs on an eighth of the remaining stage (vs half for blocks R2 must hold, so degraded R2 cannot burn half of every near-tip request's budget), and a `missing` classifies by band: in-band is the expected probe-ahead outcome (excluded from the alarm), below-band feeds `debug_trace_r2_witness_errors_total{kind="missing"}` (the bucket-integrity alarm, still covering recent-but-below-tip holes), and above-band — only reachable behind a stale catching-up tip — lands on its own `kind="missing_above_tip"` series, visible without flooding the alarm on every catch-up.
 Any witness-chain RPC attempt under a deadline is capped at the tightest of three bounds — half the full witness stage (`RpcClientConfig::witness_per_attempt_timeout`, derived from `--witness-timeout`), the global `--rpc-per-attempt-timeout-ms` (an explicitly stricter operator setting is honored, never loosened), and — only while the round still has an untried provider to rotate to — half of what the call still has as the attempt starts (recomputed after any concurrency-permit wait, so neither an old-block-clamped stage, a post-R2 remainder, nor a long permit queue defeats the reserve).
 The round's last hop, and every hop of a single-provider chain, takes the remainder whole under the ceiling instead: rotation stays protected without structurally condemning a slow-but-honest transfer, and the witness decode runs outside the attempt window (bounded by the deadline alone), so CPU-bound decode neither burns the reserve nor reads as a provider stall while a corrupt payload still rotates as the provider's error; deadline-less chain-sync fetches keep the general 20s cap so a slower-than-cap transfer still completes.