refactor(eval-author)!: split trace reads into list, overview, and spans - #1502
Merged
aleckhoury merged 1 commit intoAug 24, 2026
Conversation
Reading a trace was one command that returned everything, which made it unusable on a real trace. A 605-span run produced 26 MB of JSON, so the skill got bypassed in favor of ad-hoc curl calls that skipped its page draining and its evaluator join. Intake serves spans in a compact form that measures 82x smaller and still carries every structural field an overview needs. Structure is now read that way, and detail is spent only on the spans the caller names, selected by status, kind, parent, or span ID. The same 605-span trace reports in 154 KB, and a named span stops the read as soon as it is found instead of paying for every remaining page. Discovery becomes a `list` verb, which retires the separate entry point. The scripts target Python 3.9 rather than guarding against it, because an agent runs whatever `python3` its machine offers and macOS ships 3.9. Also drops the flat span query and its reference shaping, which nothing called, and points the tests that reached it at the HTTP client they were really exercising. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
merged commit Aug 24, 2026
93ce8d3
into
eval-author-intake-trace-understanding/akhoury
3 checks passed
15 tasks
aleckhoury
added a commit
that referenced
this pull request
Aug 25, 2026
…ans (#1502) Reading a trace was one command that returned everything, which made it unusable on a real trace. A 605-span run produced 26 MB of JSON, so the skill got bypassed in favor of ad-hoc curl calls that skipped its page draining and its evaluator join. Intake serves spans in a compact form that measures 82x smaller and still carries every structural field an overview needs. Structure is now read that way, and detail is spent only on the spans the caller names, selected by status, kind, parent, or span ID. The same 605-span trace reports in 154 KB, and a named span stops the read as soon as it is found instead of paying for every remaining page. Discovery becomes a `list` verb, which retires the separate entry point. The scripts target Python 3.9 rather than guarding against it, because an agent runs whatever `python3` its machine offers and macOS ships 3.9. Also drops the flat span query and its reference shaping, which nothing called, and points the tests that reached it at the HTTP client they were really exercising. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
aleckhoury
added a commit
that referenced
this pull request
Aug 25, 2026
…ans (#1502) Reading a trace was one command that returned everything, which made it unusable on a real trace. A 605-span run produced 26 MB of JSON, so the skill got bypassed in favor of ad-hoc curl calls that skipped its page draining and its evaluator join. Intake serves spans in a compact form that measures 82x smaller and still carries every structural field an overview needs. Structure is now read that way, and detail is spent only on the spans the caller names, selected by status, kind, parent, or span ID. The same 605-span trace reports in 154 KB, and a named span stops the read as soon as it is found instead of paying for every remaining page. Discovery becomes a `list` verb, which retires the separate entry point. The scripts target Python 3.9 rather than guarding against it, because an agent runs whatever `python3` its machine offers and macOS ships 3.9. Also drops the flat span query and its reference shaping, which nothing called, and points the tests that reached it at the HTTP client they were really exercising. Signed-off-by: Alec Khoury <akhoury@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on
eval-author-intake-trace-understanding/akhoury.Why
I ran the
eval-author-inspect-traceskill end to end against a live platform and then reported the defects. The root problem was not any single bug: reading a trace was one command that returned everything, so on a real trace it was unusable and I fell back to ad-hoccurl, skipping the skill's page draining and its evaluator join. This reshapes the scripts around the three reads an inspection actually makes.What changed
One entry point, three verbs.
inspect_trace.pynow takeslist,overview, andspans. Discovery folds intolist, retiring a second entry point.Structure is read compactly. Intake serves spans in a
summaryform that measured 82x smaller thandetailedand carries every structural field the overview and timeline need. Detail is spent only on spans the caller names, selected by--status,--kind,--parent, or--span-id.A named span stops the read early. Intake accepts no equality operator on a span
id, so--span-idis applied after the fetch, and reading stops once every named span is found. On a 605-span trace that is 0.29s for a span on page 1 against 1.84s to drain all seven pages.Payloads are bounded honestly.
inputalone was 85% of a detailed payload. Each ofinput,output, andraw_attributesis capped at--max-chars(2000 by default), and a shortened field records its full length beside it.--fullopts out. The result always reportsmax_chars, becausenullis the difference between whole and shortened evidence.Python 3.9 instead of a guard against it. The original crash was a
TypeErrorfrom 3.12-only syntax under the 3.9 that macOS ships. Rather than keep a guard file, the scripts target 3.9, and a test parses every script withfeature_version=(3, 9)so it stays that way.Dead code removed. The flat span query and its reference shaping had no caller; the four tests that reached through it now exercise
IntakeClient.draindirectly, which is where the behavior lives.Fixes carried in from the defect report: error messages in the overview are bounded, a per-span
timelinesuppliesoffset_msandduration_ms,trace_refhas one canonical form, read-phaseOSErrorbecomes a structured error, and argument errors report the documented{error, hint}object instead of an argparse exit.Honest note on scope
This does not reduce line count. The scripts go from 699 to 1006 lines, because capability I had been hand-rolling with
curlnow lives in them. What went down is the number of files (8 to 6 in the working tree), entry points (2 to 1), dead code, and output size. Further deletion is possible; say the word and I will take another pass.Test plan
uv run --frozen pytest plugins/nemo-eval-author/tests— 89 passeduv run ruff check,ruff format --check,uv run --frozen ty check— cleanuv run pre-commit run --files ...— passed/usr/bin/python3(3.9.6)