experiment pipeline: make derived + accumulated files bot-owned (phases 1+2)

.claude/skills/resqlite-experiment/SKILL.md

@@ -113,14 +113,15 @@ Run this mental (or literal) checklist:
 - [ ] `git status --short benchmark/results/` — is there an untracked result file?
 - [ ] Does that file's filename timestamp match `grep "^**Date:**" experiments/NNN-*.md`?
 - [ ] Is the experiment listed in `experiments/README.md` (Accepted or Rejected section)?
+- [ ] Is the signal entry in its own file, `experiments/signals/entries/NNN.json`
+      (not hand-edited into the generated `signals.json`)?
 - [ ] Does the experiment doc have the headings the parser expects?
       (`Problem`, `Hypothesis`, `Approach` or `What We Built`, `Results`,
       `Decision` or `Why Accepted` / `Why Rejected`)
-- [ ] Did `finalize_experiment.dart` (or `generate_history.dart`) run **last**?
-      If you touched any experiment / doc / `signals.json` / result / fixture
-      file *after* finalizing, re-run it — otherwise the committed
-      `history.json` is stale and the freshness check fails (post-merge, if the
-      PR auto-merged before CI re-ran).
+- [ ] Did `finalize_experiment.dart` pass? It only *validates* sources — it does
+      not write the generated aggregates, and you must not commit
+      `docs/experiments/history.json`, `docs/benchmarks/devices.json`, or
+      `experiments/signals.json` (the bot regenerates them on `main`).
 
 The generator's section extraction tolerates a few heading variants; see
 `_extractSection` in `generate_history.dart` for the full list.
@@ -306,12 +307,18 @@ worktree off `origin/main` (steps below), leaving the current tree untouched:
 git fetch origin
 ```
 
-Every experiment PR rewrites the same three shared files —
-`experiments/README.md`, `experiments/signals.json`, and the generated
-`docs/experiments/history.json` — and CI's `check_generated_data.dart` fails
-any PR whose `history.json` predates a merge. So concurrent runs that grab the
-same number, or do the same work, collide and stale each other (exp 168 was
-claimed by three PRs; exp 175 by two runs shipping the *same* follow-up).
+Experiment PRs used to collide on shared generated files; that is now designed
+out. The generated aggregates — `docs/experiments/history.json`,
+`docs/benchmarks/devices.json`, and `experiments/signals.json` — are
+**bot-owned on `main` and never committed on a branch** (CI's
+`guard-generated-docs` job blocks them, and `check_generated_data.dart` only
+checks that the *sources* build). Each experiment's signal data lives in its
+own file, `experiments/signals/entries/NNN.json`, so two experiments never
+touch the same one. The only files a normal run still shares are
+`experiments/README.md` (rows append) and `experiments/signals/base.json` (the
+per-direction synthesis). You must still claim your number — concurrent runs
+that grab the same number, or ship the same follow-up, still collide (exp 168
+was claimed by three PRs; exp 175 by two runs shipping the *same* follow-up).
 
 **1. Claim the number atomically.** A plain "check open PRs, then pick the next
 free" *races*: two runs check, both see N free, both take N. That is exactly
@@ -363,24 +370,33 @@ which case expect to regenerate `history.json` on the later one.
 
 ## Resolving a stale derived-file conflict
 
-If a PR did fall behind `main`, the conflict is mechanical — only
-generated/narrative files collide:
+Generated aggregates no longer conflict: `docs/experiments/history.json`,
+`docs/benchmarks/devices.json`, and `experiments/signals.json` are bot-owned
+and never committed on a branch, so a stale branch just takes `main`'s copy
+with a one-sided auto-merge. If a PR falls behind `main`, the only files that
+can really conflict are hand-edited *sources*:
+
+- `experiments/README.md` — rows append; usually auto-merges. If two
+  experiments inserted at the same spot, fix the row order by hand.
+- `experiments/signals/entries/NNN.json` — one file per experiment, so a true
+  collision only happens when two runs claimed the same number `N`. That's a
+  numbering bug: renumber, never overwrite a prior experiment's entry.
+- `experiments/signals/base.json` — two experiments editing the same
+  direction's `currentRead` / `notesForExperimenters` narrative is a real
+  weave; keep both contributions.
 
 ```bash
 git merge origin/main
-dart run benchmark/generate_history.dart      # rebuilds docs/experiments/history.json
-dart run benchmark/generate_devices.dart      # rebuilds docs/benchmarks/devices.json
-# hand-reconcile experiments/signals.json: keep BOTH sides' entries
-dart run benchmark/check_generated_data.dart  # must print "up to date"
-dart run benchmark/check_experiment_signals.dart
+# resolve any of the source files above by hand, then:
+dart run benchmark/check_generated_data.dart       # sources build cleanly
+dart run benchmark/check_experiment_signals.dart   # signal map valid
 git add -A && git commit --no-edit
 ```
 
-`experiments/README.md` usually auto-merges (rows append). `signals.json`
-needs care: two experiments editing the same `currentRead` /
-`notesForExperimenters` narrative is a real weave, and two experiments
-claiming the same `experiments.<N>` key is a number collision — fix the
-numbering, never overwrite a prior experiment's entry.
+Do **not** regenerate or `git add` the generated aggregates. If your branch
+somehow carries changes to them, revert: `git checkout origin/main --
+docs/experiments/history.json docs/benchmarks/devices.json
+experiments/signals.json`.
 
 ## Post-merge
 

.github/workflows/ci.yml

@@ -33,6 +33,32 @@ jobs:
             exit 1
           fi
 
+  guard-generated-docs:
+    # history.json and devices.json are generated aggregates owned by the
+    # "Update Docs Data" bot on main. Experiment branches must NOT commit them,
+    # so a merge (or a post-merge bot commit) never re-conflicts open PRs on
+    # them — the routine churn this whole convention exists to remove. See
+    # experiments/RUNNER_INSTRUCTIONS.md ("Generated files are bot-owned").
+    name: Guard against committing generated docs
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request'
+    steps:
+      - uses: actions/checkout@v6.0.2
+        with:
+          fetch-depth: 0
+      - name: Check PR diff for bot-owned generated files
+        run: |
+          base="origin/${{ github.base_ref }}"
+          changed=$(git diff --name-only "$base...HEAD" \
+            -- 'docs/experiments/history.json' 'docs/benchmarks/devices.json' || true)
+          if [ -n "$changed" ]; then
+            echo "::error::Generated docs are bot-owned; do not commit them on a branch."
+            echo "Revert them so the PR carries no change to these files:"
+            printf '  %s\n' $changed
+            echo "Run: git checkout origin/${{ github.base_ref }} -- docs/experiments/history.json docs/benchmarks/devices.json"
+            exit 1
+          fi
+
   test:
     name: Tests
     runs-on: macos-latest

.github/workflows/update-experiments.yml

@@ -5,6 +5,8 @@ on:
     branches: [main]
     paths:
       - 'experiments/*.md'
+      - 'experiments/signals/**'
+      - 'benchmark/generate_signals.dart'
       - 'benchmark/results/*.md'
       - 'benchmark/results/*.json'
       - 'benchmark/HARDWARE_RESULTS.md'
@@ -39,6 +41,9 @@ jobs:
       - name: Generate history JSON
         run: dart run benchmark/generate_history.dart
 
+      - name: Generate signals JSON
+        run: dart run benchmark/generate_signals.dart
+
       - name: Generate devices JSON
         run: dart run benchmark/generate_devices.dart
 
@@ -51,13 +56,13 @@ jobs:
       - name: Check for changes
         id: diff
         run: |
-          git diff --quiet docs/ && echo "changed=false" >> "$GITHUB_OUTPUT" || echo "changed=true" >> "$GITHUB_OUTPUT"
+          git diff --quiet docs/ experiments/signals.json && echo "changed=false" >> "$GITHUB_OUTPUT" || echo "changed=true" >> "$GITHUB_OUTPUT"
 
       - name: Commit and push
         if: steps.diff.outputs.changed == 'true'
         run: |
           git config user.name "github-actions[bot]"
           git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-          git add docs/
-          git commit -m "Auto-update docs (experiments + devices + blog + releases)"
+          git add docs/ experiments/signals.json
+          git commit -m "Auto-update docs (experiments + signals + devices + blog + releases)"
           git push

benchmark/check_experiment_signals.dart

@@ -1,6 +1,7 @@
-import 'dart:convert';
 import 'dart:io';
 
+import 'generate_signals.dart' as generate_signals;
+
 const _experimentsDir = 'experiments';
 const _readmePath = '$_experimentsDir/README.md';
 const _signalsPath = '$_experimentsDir/signals.json';
@@ -115,26 +116,21 @@ int? _numericExperimentId(String id) {
   return match == null ? null : int.tryParse(match.group(0)!);
 }
 
+/// Assembles the signal map from its sources (`experiments/signals/base.json` +
+/// `experiments/signals/entries/NNN.json`) and validates that, rather than the
+/// committed `signals.json` — which is a generated, bot-owned aggregate that an
+/// experiment branch never has to keep fresh. A malformed fragment surfaces
+/// here as an assembly failure.
 Map<Object?, Object?>? _readSignals(List<_ValidationError> errors) {
-  final file = File(_signalsPath);
-  if (!file.existsSync()) {
-    _signalError(errors, 'Missing $_signalsPath.');
-    return null;
-  }
-
-  Object? decoded;
   try {
-    decoded = json.decode(file.readAsStringSync());
-  } on FormatException catch (error) {
-    _signalError(errors, 'signals.json is not valid JSON: ${error.message}');
-    return null;
-  }
-
-  if (decoded is! Map) {
-    _signalError(errors, 'signals.json must contain a top-level JSON object.');
+    final data = generate_signals.buildSignalsData(
+      signalsSourceDir: Directory('$_experimentsDir/signals'),
+    );
+    return data.cast<Object?, Object?>();
+  } catch (error) {
+    _signalError(errors, 'signals sources do not assemble: $error');
     return null;
   }
-  return decoded.cast<Object?, Object?>();
 }
 
 void _checkSignals(

benchmark/check_generated_data.dart

@@ -1,108 +1,65 @@
-import 'dart:convert';
 import 'dart:io';
 
 import 'generate_devices.dart' as generate_devices;
 import 'generate_history.dart' as generate_history;
+import 'generate_signals.dart' as generate_signals;
 
+/// Verifies that the generated-docs sources are well-formed enough that the
+/// generators run to completion.
+///
+/// This does **not** compare against the committed `docs/.../*.json` or
+/// `experiments/signals.json`. Those are generated artifacts owned by the
+/// post-merge "Update Docs Data" bot, not by experiment branches — see
+/// experiments/RUNNER_INSTRUCTIONS.md ("Generated files are bot-owned"). A
+/// branch never has to carry a fresh copy, which is what used to force every
+/// open experiment PR to re-resolve the same mechanical conflict on those
+/// files. The job here is to fail fast on a *source* that can't be generated
+/// (a malformed signals fragment, a missing run declaration, a broken
+/// experiment <-> benchmark-run mapping) — every one of those checks lives
+/// inside the build functions below and throws on violation.
 Future<void> main() async {
-  final mismatches = <String>[];
-
-  await _checkJsonFile(
-    label: 'devices.json',
-    currentPath: 'docs/benchmarks/devices.json',
-    buildExpected: (generatedAt) => generate_devices.buildDevicesData(
+  final builders = <String, void Function()>{
+    'devices.json': () => generate_devices.buildDevicesData(
       hardwareResultsMarkdown: File(
         'benchmark/HARDWARE_RESULTS.md',
       ).readAsStringSync(),
       resultsDir: Directory('benchmark/results'),
-      generatedAt: generatedAt,
+      generatedAt: null,
     ),
-    mismatches: mismatches,
-  );
-
-  await _checkJsonFile(
-    label: 'history.json',
-    currentPath: 'docs/experiments/history.json',
-    buildExpected: (generatedAt) => generate_history.buildHistoryData(
+    'history.json': () => generate_history.buildHistoryData(
       resultsDir: Directory('benchmark/results'),
       experimentsDir: Directory('experiments'),
-      generatedAt: generatedAt,
+      generatedAt: null,
+    ),
+    'signals.json': () => generate_signals.buildSignalsData(
+      signalsSourceDir: Directory('experiments/signals'),
+      generatedAt: null,
     ),
-    mismatches: mismatches,
-  );
+  };
+
+  final failures = <String>[];
+  for (final entry in builders.entries) {
+    try {
+      entry.value();
+      print('${entry.key}: generators build cleanly.');
+    } catch (error, stack) {
+      failures.add(entry.key);
+      stderr.writeln('::error::${entry.key} failed to generate: $error');
+      stderr.writeln(stack);
+    }
+  }
 
-  if (mismatches.isNotEmpty) {
+  if (failures.isNotEmpty) {
     stderr.writeln('');
     stderr.writeln(
-      'Benchmark-generated docs are stale. Re-run the generators and commit the updated files:',
-    );
-    stderr.writeln('  dart run benchmark/generate_devices.dart');
-    stderr.writeln('  dart run benchmark/generate_history.dart');
-    stderr.writeln('For experiment writeups, prefer:');
-    stderr.writeln(
-      '  dart run benchmark/finalize_experiment.dart --experiment=experiments/NNN-short-slug.md',
+      'Generated-docs sources do not build: ${failures.join(', ')}. '
+      'Fix the offending source (experiment doc, benchmark result, or '
+      'signals fragment) — you do NOT need to commit the regenerated '
+      'docs/*.json or signals.json; the bot owns those on main.',
     );
     exitCode = 1;
     return;
   }
 
-  print('Benchmark-generated docs are up to date.');
-}
-
-Future<void> _checkJsonFile({
-  required String label,
-  required String currentPath,
-  required Object Function(String? generatedAt) buildExpected,
-  required List<String> mismatches,
-}) async {
-  final currentFile = File(currentPath);
-  if (!currentFile.existsSync()) {
-    throw StateError(
-      'Missing generated docs artifact: $currentPath. '
-      'Re-run the generator and commit the updated file.',
-    );
-  }
-
-  final currentText = currentFile.readAsStringSync();
-  final currentJson = json.decode(currentText);
-  if (currentJson is! Map<String, Object?>) {
-    throw StateError('$currentPath does not contain a top-level JSON object.');
-  }
-
-  final expected = buildExpected(currentJson['generated']?.toString());
-  const encoder = JsonEncoder.withIndent('  ');
-  final currentComparable = '${encoder.convert(currentJson)}\n';
-  final expectedText = '${encoder.convert(expected)}\n';
-
-  if (currentComparable == expectedText) {
-    print('$label is current.');
-    return;
-  }
-
-  mismatches.add(label);
-  stderr.writeln('::error file=$currentPath::$label is stale.');
-
-  final tempDir = await Directory.systemTemp.createTemp(
-    'resqlite-generated-data-diff_',
-  );
-  try {
-    final normalizedCurrentFile = File(
-      '${tempDir.path}/current_${currentFile.uri.pathSegments.last}',
-    )..writeAsStringSync(currentComparable);
-    final expectedFile = File(
-      '${tempDir.path}/${currentFile.uri.pathSegments.last}',
-    )..writeAsStringSync(expectedText);
-    final diff = await Process.run('diff', [
-      '-u',
-      normalizedCurrentFile.path,
-      expectedFile.path,
-    ]);
-    if ((diff.stdout as String).trim().isNotEmpty) {
-      stderr.writeln(diff.stdout);
-    } else if ((diff.stderr as String).trim().isNotEmpty) {
-      stderr.writeln(diff.stderr);
-    }
-  } finally {
-    await tempDir.delete(recursive: true);
-  }
+  print('All generated-docs sources build cleanly.');
 }

benchmark/finalize_experiment.dart

@@ -7,8 +7,12 @@ const _usage = '''
 Usage:
   dart run benchmark/finalize_experiment.dart --experiment=experiments/NNN-short-slug.md
 
-Regenerates docs/experiments/history.json and runs the experiment postflight
-checks that should pass before committing an experiment record.
+Runs the experiment postflight checks that should pass before committing an
+experiment record. It does NOT write docs/experiments/history.json,
+docs/benchmarks/devices.json, or experiments/signals.json — those are
+generated aggregates owned by the post-merge bot, not committed on branches
+(see experiments/RUNNER_INSTRUCTIONS.md). The checks below verify the sources
+generate cleanly and the signal map is valid.
 ''';
 
 Future<void> main(List<String> args) async {
@@ -35,12 +39,7 @@ Future<void> main(List<String> args) async {
 
   final commands = [
     _Command(
-      label: 'generate experiment history',
-      executable: Platform.resolvedExecutable,
-      args: const ['run', 'benchmark/generate_history.dart'],
-    ),
-    _Command(
-      label: 'check generated docs data',
+      label: 'check generated docs sources build cleanly',
       executable: Platform.resolvedExecutable,
       args: const ['run', 'benchmark/check_generated_data.dart'],
     ),