Skip to content

refactor(eval-author)!: split trace reads into list, overview, and spans - #1502

Merged
aleckhoury merged 1 commit into
eval-author-intake-trace-understanding/akhouryfrom
fix-eval-author-inspect-trace-defects/akhoury
Aug 24, 2026
Merged

refactor(eval-author)!: split trace reads into list, overview, and spans#1502
aleckhoury merged 1 commit into
eval-author-intake-trace-understanding/akhouryfrom
fix-eval-author-inspect-trace-defects/akhoury

Conversation

@aleckhoury

Copy link
Copy Markdown
Contributor

Stacked on eval-author-intake-trace-understanding/akhoury.

Why

I ran the eval-author-inspect-trace skill end to end against a live platform and then reported the defects. The root problem was not any single bug: reading a trace was one command that returned everything, so on a real trace it was unusable and I fell back to ad-hoc curl, skipping the skill's page draining and its evaluator join. This reshapes the scripts around the three reads an inspection actually makes.

What changed

One entry point, three verbs. inspect_trace.py now takes list, overview, and spans. Discovery folds into list, retiring a second entry point.

Structure is read compactly. Intake serves spans in a summary form that measured 82x smaller than detailed and carries every structural field the overview and timeline need. Detail is spent only on spans the caller names, selected by --status, --kind, --parent, or --span-id.

Trace Before After
67 spans 1.9 MB 24 KB
605 spans 26.3 MB 154 KB

A named span stops the read early. Intake accepts no equality operator on a span id, so --span-id is applied after the fetch, and reading stops once every named span is found. On a 605-span trace that is 0.29s for a span on page 1 against 1.84s to drain all seven pages.

Payloads are bounded honestly. input alone was 85% of a detailed payload. Each of input, output, and raw_attributes is capped at --max-chars (2000 by default), and a shortened field records its full length beside it. --full opts out. The result always reports max_chars, because null is the difference between whole and shortened evidence.

Python 3.9 instead of a guard against it. The original crash was a TypeError from 3.12-only syntax under the 3.9 that macOS ships. Rather than keep a guard file, the scripts target 3.9, and a test parses every script with feature_version=(3, 9) so it stays that way.

Dead code removed. The flat span query and its reference shaping had no caller; the four tests that reached through it now exercise IntakeClient.drain directly, which is where the behavior lives.

Fixes carried in from the defect report: error messages in the overview are bounded, a per-span timeline supplies offset_ms and duration_ms, trace_ref has one canonical form, read-phase OSError becomes a structured error, and argument errors report the documented {error, hint} object instead of an argparse exit.

Honest note on scope

This does not reduce line count. The scripts go from 699 to 1006 lines, because capability I had been hand-rolling with curl now lives in them. What went down is the number of files (8 to 6 in the working tree), entry points (2 to 1), dead code, and output size. Further deletion is possible; say the word and I will take another pass.

Test plan

  • uv run --frozen pytest plugins/nemo-eval-author/tests — 89 passed
  • uv run ruff check, ruff format --check, uv run --frozen ty check — clean
  • uv run pre-commit run --files ... — passed
  • All six scripts compile under /usr/bin/python3 (3.9.6)
  • All three verbs run against a live platform on 3.9.6
  • Eight failure modes return structured JSON with exit 1
  • New doc-rot test confirmed to fail when a document names a deleted script

Reading a trace was one command that returned everything, which made it
unusable on a real trace. A 605-span run produced 26 MB of JSON, so the
skill got bypassed in favor of ad-hoc curl calls that skipped its page
draining and its evaluator join.

Intake serves spans in a compact form that measures 82x smaller and still
carries every structural field an overview needs. Structure is now read
that way, and detail is spent only on the spans the caller names, selected
by status, kind, parent, or span ID. The same 605-span trace reports in
154 KB, and a named span stops the read as soon as it is found instead of
paying for every remaining page.

Discovery becomes a `list` verb, which retires the separate entry point.
The scripts target Python 3.9 rather than guarding against it, because an
agent runs whatever `python3` its machine offers and macOS ships 3.9.

Also drops the flat span query and its reference shaping, which nothing
called, and points the tests that reached it at the HTTP client they were
really exercising.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
@aleckhoury
aleckhoury requested review from a team as code owners August 24, 2026 21:57
@github-actions github-actions Bot added breaking breaking change (!-marked title) refactor labels Aug 24, 2026
@aleckhoury
aleckhoury merged commit 93ce8d3 into eval-author-intake-trace-understanding/akhoury Aug 24, 2026
3 checks passed
@aleckhoury
aleckhoury deleted the fix-eval-author-inspect-trace-defects/akhoury branch August 24, 2026 22:04
aleckhoury added a commit that referenced this pull request Aug 25, 2026
…ans (#1502)

Reading a trace was one command that returned everything, which made it
unusable on a real trace. A 605-span run produced 26 MB of JSON, so the
skill got bypassed in favor of ad-hoc curl calls that skipped its page
draining and its evaluator join.

Intake serves spans in a compact form that measures 82x smaller and still
carries every structural field an overview needs. Structure is now read
that way, and detail is spent only on the spans the caller names, selected
by status, kind, parent, or span ID. The same 605-span trace reports in
154 KB, and a named span stops the read as soon as it is found instead of
paying for every remaining page.

Discovery becomes a `list` verb, which retires the separate entry point.
The scripts target Python 3.9 rather than guarding against it, because an
agent runs whatever `python3` its machine offers and macOS ships 3.9.

Also drops the flat span query and its reference shaping, which nothing
called, and points the tests that reached it at the HTTP client they were
really exercising.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury added a commit that referenced this pull request Aug 25, 2026
…ans (#1502)

Reading a trace was one command that returned everything, which made it
unusable on a real trace. A 605-span run produced 26 MB of JSON, so the
skill got bypassed in favor of ad-hoc curl calls that skipped its page
draining and its evaluator join.

Intake serves spans in a compact form that measures 82x smaller and still
carries every structural field an overview needs. Structure is now read
that way, and detail is spent only on the spans the caller names, selected
by status, kind, parent, or span ID. The same 605-span trace reports in
154 KB, and a named span stops the read as soon as it is found instead of
paying for every remaining page.

Discovery becomes a `list` verb, which retires the separate entry point.
The scripts target Python 3.9 rather than guarding against it, because an
agent runs whatever `python3` its machine offers and macOS ships 3.9.

Also drops the flat span query and its reference shaping, which nothing
called, and points the tests that reached it at the HTTP client they were
really exercising.

Signed-off-by: Alec Khoury <akhoury@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

breaking breaking change (!-marked title) refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant