Skip to content

experiment pipeline: generate README tables from per-experiment fragments - #203

Merged
danReynolds merged 1 commit into
mainfrom
readme-generated
Jun 23, 2026
Merged

experiment pipeline: generate README tables from per-experiment fragments#203
danReynolds merged 1 commit into
mainfrom
readme-generated

Conversation

@danReynolds

Copy link
Copy Markdown
Owner

Why

This completes the conflict-reduction work — the deferred README half of phase 2 (#202 did history/devices/signals). README.md was the last shared file experiments hand-edited, and the reason it was deferred: generate_history.dart parses README rows as a source, so generating it requires breaking that dependency carefully.

How

  • experiments/index/NNN.json — one fragment per experiment with its README row ({file, title, impact, status, link}). Two experiments never touch the same file. A split experiment (014 has both an accepted and a rejected finding) stores an array of rows.
  • experiments/README.template.md — the prose + section/table headers with {{ROWS:status}} placeholders.
  • generate_readme.dart assembles README.md from template + fragments, newest-first. README.md becomes a generated, bot-owned aggregate.
  • generate_history.dart now sources experiments from the fragments — it builds the equivalent table markdown in memory and feeds the unchanged, golden-tested parser (with a README.md fallback for fixtures). Its sort is made stable (insertion-order tiebreak) so ids sharing a number (008/008b, the two 014 rows) are deterministic regardless of source.
  • check_generated_data builds README too; check_experiment_signals and finalize_experiment read the fragments instead of README rows; the bot regenerates README.md.

Correctness (round-trip proofs)

  • README: regenerated from fragments, the row set per section is identical and prose/headers are byte-identical to today's README. The only change is row order, which normalizes to id-descending (the prior order was an irregular historical artifact — Accepted was recent-desc then old-ascending).
  • history.json: byte-identical to main except the two duplicate-014 entries normalize to accepted-first (the old order was an unstable-sort artifact; consumers index by id, so it's cosmetic).
  • Golden-fixture + pipeline tests pass; dart analyze --fatal-infos clean on all changed files; this PR touches no docs/*.json or signals.json (passes guard-generated-docs).

Rollout

Mirrors the signals.json bootstrap: this PR commits the reordered README.md as the new generated baseline, so experiments/README.md is not added to guard-generated-docs here — a one-line follow-up adds it once the baseline is on main.

Test plan

  • check_generated_data — devices/history/signals/README all build cleanly
  • check_experiment_signals — valid from fragments
  • README round-trip: row set + prose preserved (content), only order normalized
  • history.json round-trip: identical modulo the 014 stable-sort normalization
  • dart test benchmark_pipeline_test benchmark_generated_outputs_test — pass (golden fixtures)
  • dart analyze --fatal-infos — clean

…ents

Completes the conflict-reduction work (the deferred README half of phase 2).
README.md was the last shared file experiments hand-edited (and a *source*
for generate_history, which is why it was deferred). Now:

- experiments/index/NNN.json — one fragment per experiment holding its README
  row {file, title, impact, status, link}. A "split" experiment (014: an
  accepted finding + a rejected one) stores an array of rows.
- experiments/README.template.md — prose + table headers with {{ROWS:status}}
  placeholders.
- generate_readme.dart assembles README.md from template + fragments, newest
  first (id descending). README.md becomes a generated, bot-owned aggregate.
- generate_history.dart now sources experiments from the fragments (it builds
  the equivalent table markdown in memory and feeds the UNCHANGED, golden-
  tested parser), with a README.md fallback for test fixtures. Its sort is
  made stable (insertion-order tiebreak) so split/suffixed ids (008/008b, the
  two 014 rows) are deterministic regardless of source.
- check_generated_data builds README too; check_experiment_signals and
  finalize_experiment read the index fragments instead of README rows.
- The bot regenerates README.md; instructions updated to "add index/NNN.json,
  never edit the README table".

Verified: README round-trips content-exactly (same row set per section, prose
byte-identical) — only the row ORDER normalizes to id-descending (the prior
order was an irregular historical artifact). history.json is byte-identical
except the two duplicate-014 entries normalize to accepted-first via the new
stable sort. Golden-fixture + pipeline tests pass; analyze clean.

Two follow-ups, mirroring the signals.json rollout:
- add experiments/README.md to guard-generated-docs once this baseline lands
  (left out so this bootstrap PR can commit the reordered README).
- README-table generation does not touch the chart pipeline beyond the stable
  sort.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@danReynolds
danReynolds merged commit bd4fd6a into main Jun 23, 2026
6 checks passed
danReynolds added a commit that referenced this pull request Jun 23, 2026
README.md is now a generated, bot-owned aggregate (assembled from
experiments/index/NNN.json fragments), so block branches from committing it
too — closing the bootstrap gap left when the generated README baseline
landed in #203.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant