fix(debug-trace-server): stop frontier witness misses from surfacing as timeouts - #179
Conversation
…as timeouts
Two client-visible -32001s on ore, both frontier blocks whose witness was
not generated yet:
Witness fetch deadline exceeded block_number=23461272
source="witness_generator" old_block=false budget_ms=7999 elapsed_ms=8001
The 8s bought roughly four provider rotations, all structurally useless:
the fallback endpoints are fed by the same generation pipeline and cannot
be ahead of the generator, so each rotation spent a public-gateway round
trip to learn what the generator had already said. Chain sync has had an
exclusive generator grace since #164; the request path never got one.
Adds it, with one deliberate difference from chain sync's twin: that one
also serves as the trust horizon for a head observation that goes stale
whenever the fetcher has work queued, which silently disables the routing.
The request path's freshness signal is the local DB tip, read fresh from
redb on every fetch, so this grace carries one meaning only and cannot be
switched off by a busy window. The grace is clamped to half the remaining
witness budget so an unavailable generator still leaves the fallback chain
a real budget rather than a nearly-expired one.
A frontier witness that still never arrives is no longer reported as a
deadline. `DataProviderError::WitnessNotReady` renders as -32002 "witness
for block N is not generated yet; retry shortly", so a client can tell a
retryable chain-lag apart from a nonexistent block — today both leave as
-32001 — and `reason="deadline_witness"` goes back to meaning only what it
says. Parity trace_transaction degrades it to null like its siblings.
This converts the failure into a typed, actionable one; it does not make
the request wait indefinitely. Unbounded waiting needs the response-size
cap first, or it reintroduces the OOM shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude review status
🛠️ Review did not finish Attempted head This round did not publish: MODEL_ACTION_FAILED in phase review_retry. Anything listed below is from the last round that did. Re-run the workflow or push a new commit to try again. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5850203345
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if frontier { | ||
| return Err(DataProviderError::WitnessNotReady { block_number }); |
There was a problem hiding this comment.
When frontier is true, this branch converts every Err from the full witness provider chain into WitnessNotReady, but get_witness_light_with_deadline_from also returns RpcDeadlineExceeded after repeated provider stalls, transport errors, or decode failures until the deadline expires. In a local-cache deployment with generator+fallback endpoints, a gateway outage or overload for an above-tip block will now be reported to clients as -32002 ... not generated yet and counted as witness_not_ready instead of preserving the real witness deadline signal, masking the incident the separate reason labels are meant to expose.
AGENTS.md reference: AGENTS.md:L127-L127
Useful? React with 👍 / 👎.
|
Converting to draft: the premise this PR is built on does not hold. Log forensics on ore (2026-08-09, 3.01h) shows the request-path gateway hop is trimodal, not uniformly useless:
So "the fallbacks cannot be ahead of the generator" is false. The generator's miss is a file lookup ( This PR's grace would therefore delay 13 successful fetches by up to the grace to rescue 2, pushing the ones near the budget edge into new failures. What the evidence does support: What it does not yet establish: whether those 7.9s were spent inside the gateway call or waiting on the witness semaphore. Both are silent and produce identical observations. Instrumentation first; the fix follows the data. |
|
Closing rather than iterating: the premise is inverted, so there is nothing here to salvage. Reading the generator (
So for a frontier block, R2 can legitimately hold the witness while the RPC server's This PR gave the generator an exclusive grace before rotating — i.e. it prioritises the source that is structurally later. Wrong direction. Two follow-ups instead:
|
Summary
Stacked on #178. Gives the generator a bounded exclusive grace for frontier blocks on the request path, and reports a still-missing frontier witness as
-32002 not generated yetinstead of a deadline.Root cause
Two client-visible
-32001s on ore, both frontier blocks whose witness was not generated yet:The budget bought roughly four provider rotations, all structurally useless: the fallback endpoints are fed by the same generation pipeline and cannot be ahead of the generator (
chain_sync.rs:27), so each rotation spent a public-gateway round trip to learn what the generator had already said. Chain sync has had an exclusive generator grace since #164 (chain_sync.rs:119); the request path (data_provider.rs:1160) never got one — that call site is the only one in the tree that does not use the grace probe.Fix
DataProviderError::WitnessNotReadyrenders as-32002 witness for block N is not generated yet; retry shortly, so a client can tell a retryable chain-lag apart from a nonexistent block — today both leave as-32001— andreason="deadline_witness"goes back to meaning only what it says. Paritytrace_transactiondegrades it tonulllike its siblings.debug_trace_frontier_grace_total{outcome="served"|"expired"}.Testing
cargo test --workspacegreen;fmt/clippy --all-targets --all-features/cargo sortclean. The new variant is covered byerror_reason_separates_deadlines_from_not_found(distinct code, distinct reason, retry-shaped message).Notes
This converts the failure into a typed, actionable one; it does not make the request wait until it succeeds. Unbounded waiting needs the response-size cap first, or it reintroduces the OOM shape from the 2026-08-07 tko incident.
Worth adding in review: an integration test driving the grace-expiry fall-through.
test_support::scripted_witness_rpccan withhold the witness for N attempts, which is the shape needed. This is the riskiest part of the change (production witness routing) and currently only has compile-time coverage.