Diff-style SFT export for repair turns (cut token waste) - #55
Merged
Conversation
Multi-turn episodes re-emitted the full ~100-line kernel on every repair even
when only a few lines changed (~1/3 of repairs change <10% of lines). Add an
opt-in export mode that renders repair/optimize assistant turns as a unified
diff against the previous attempt, preserving the <think> rationale.
- episodes.render_repair_as_diff + episode_to_messages(repair_style="full"|"diff").
- hf_dataset.trajectory_to_messages_record / publish_bundle_to_hf thread
repair_style; diff rows are marked metadata.repair_style="diff".
- sparkproof-publish-dataset --repair-style {full,diff} (default full).
Export-view only: the raw/attested trajectories.jsonl, code_sha256, and Merkle
leaves keep the complete runnable code, so the trust boundary is untouched. On
the current merged data this cuts repair-turn size ~23% (more for small-diff
repairs); no-op fallback to full when a repair has no code delta.
Tests: diff render (reasoning kept, shrinks, no-change fallback), episode diff
vs full, trajectory record marks repair_style + emits diffs.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the last item from the dataset-quality critique: multi-turn episodes re-emit the whole ~100-line kernel on every repair even when <10% of lines change (~⅓ of repairs), wasting most of the tokens.
What it does
New opt-in export mode that renders repair/optimize assistant turns as a unified diff against the previous attempt, keeping the
<think>rationale:episodes.render_repair_as_diff+episode_to_messages(repair_style="full" | "diff")hf_datasetthreadsrepair_stylethroughtrajectory_to_messages_record/publish_bundle_to_hf; diff rows are taggedmetadata.repair_style="diff"sparkproof-publish-dataset --repair-style {full,diff}(defaultfull, so nothing changes unless asked)Trust boundary untouched
This is an export-view transform only. The raw/attested
trajectories.jsonl,code_sha256, and Merkle leaves keep the complete runnable code — validation and verification are unaffected. Falls back to full content when a repair has no code delta.Measured savings (real data)
On the live
sparkproof-miningmulti-turn rows (17 rows / 22 repair turns): repair-turn size drops ~23% (~30.2K → ~23.4K tokens). Honest caveat: modest here because this batch's repairs have a 15% median churn (some are large rewrites); for the many <10%-churn repairs the reduction is far larger. A consumer (SparkDistill) opts in and applies the patch to reconstruct full code.Tests (+5, 381 total, ruff clean)
diff render (reasoning preserved, shrinks, no-change fallback), episode diff vs full-default, trajectory record marks
repair_styleand emits diffs.