Flag repairs that pass by deleting a hardware guard (dataset quality) - #53
Merged
Conversation
Defense-in-depth for the validator-hack found in the merged sparkproof-mining data: a repair that "passes" by removing a device/SM/compute-capability/arch assertion the failed attempt added (e.g. an SM90 guard that failed on the real SM120 GPU) rather than fixing the kernel. - anti_cheat.detect_removed_hardware_guard(prev, new): conservative — only fires on a net decrease in guard lines. - episodes.episode_removed_guard / trajectory_removed_guard: scan an episode's failed->repair assistant turns for guard removal. - release_gate: record a NON-BLOCKING repair_guard_removed_rows count (+ up to 50 task ids) in dataset_manifest.json. These kernels still passed hardware validation, so they are surfaced for review / reward down-weighting, not dropped (blocking them would discard correct kernels whose only problem was a mislabeled target arch). Complements the portable-prompt nudge in #51. Tests: detector (removed/kept/none), episode scan (failed->removed flagged, prior-passed or guard-kept not flagged), trajectory helper.
This was referenced Jul 26, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Defense-in-depth follow-up to #51 (portable prompts). Catches the exact validator-hack I found in the merged
sparkproof-miningdata: a repair that "passes" by removing a device / SM / compute-capability / architecture assertion the failed attempt had added (e.g. an SM90 guard that then failed on the real SM120 GPU:RuntimeError: targets Hopper SM90, found SM120) — instead of fixing the kernel.What it does
anti_cheat.detect_removed_hardware_guard(prev, new)— conservative: only fires on a net decrease in guard lines, so a repair that keeps or strengthens its checks is never flagged.episodes.episode_removed_guard/trajectory_removed_guard— scan an episode'sfailed → repairassistant turns for guard removal.repair_guard_removed_rowscount (+ up to 50 task ids) indataset_manifest.json.Why non-blocking (deliberate)
These kernels still passed hardware validation — the "hack" is only relative to a mislabeled target arch (the task said Hopper SM90 while the validator was Blackwell SM120). Hard-blocking would discard correct kernels. So the gate surfaces the pattern for review / reward down-weighting rather than dropping rows. #51 prevents the guard being added in the first place; this makes any that slip through visible.
Tests (+7, 366 total, ruff clean)
Detector (removed / kept / no-guard), episode scan (failed→removed flagged; prior-attempt-passed or guard-kept not flagged), trajectory helper.