Skip to content

Flag repairs that pass by deleting a hardware guard (dataset quality) - #53

Merged
ai-hpc merged 1 commit into
mainfrom
feat/flag-guard-removal-repairs
Jul 26, 2026
Merged

Flag repairs that pass by deleting a hardware guard (dataset quality)#53
ai-hpc merged 1 commit into
mainfrom
feat/flag-guard-removal-repairs

Conversation

@ai-hpc

@ai-hpc ai-hpc commented Jul 26, 2026

Copy link
Copy Markdown
Member

Defense-in-depth follow-up to #51 (portable prompts). Catches the exact validator-hack I found in the merged sparkproof-mining data: a repair that "passes" by removing a device / SM / compute-capability / architecture assertion the failed attempt had added (e.g. an SM90 guard that then failed on the real SM120 GPU: RuntimeError: targets Hopper SM90, found SM120) — instead of fixing the kernel.

What it does

  • anti_cheat.detect_removed_hardware_guard(prev, new) — conservative: only fires on a net decrease in guard lines, so a repair that keeps or strengthens its checks is never flagged.
  • episodes.episode_removed_guard / trajectory_removed_guard — scan an episode's failed → repair assistant turns for guard removal.
  • Release gate — records a non-blocking repair_guard_removed_rows count (+ up to 50 task ids) in dataset_manifest.json.

Why non-blocking (deliberate)

These kernels still passed hardware validation — the "hack" is only relative to a mislabeled target arch (the task said Hopper SM90 while the validator was Blackwell SM120). Hard-blocking would discard correct kernels. So the gate surfaces the pattern for review / reward down-weighting rather than dropping rows. #51 prevents the guard being added in the first place; this makes any that slip through visible.

Tests (+7, 366 total, ruff clean)

Detector (removed / kept / no-guard), episode scan (failed→removed flagged; prior-attempt-passed or guard-kept not flagged), trajectory helper.

Branches from main; independent of the held Kimi PR (#50). Natural deeper follow-up remains the task-label ↔ hardware fix so gpu_architecture reflects the actual validating GPU (removes the mismatch at the source).

Defense-in-depth for the validator-hack found in the merged sparkproof-mining
data: a repair that "passes" by removing a device/SM/compute-capability/arch
assertion the failed attempt added (e.g. an SM90 guard that failed on the real
SM120 GPU) rather than fixing the kernel.

- anti_cheat.detect_removed_hardware_guard(prev, new): conservative — only fires
  on a net decrease in guard lines.
- episodes.episode_removed_guard / trajectory_removed_guard: scan an episode's
  failed->repair assistant turns for guard removal.
- release_gate: record a NON-BLOCKING repair_guard_removed_rows count (+ up to 50
  task ids) in dataset_manifest.json. These kernels still passed hardware
  validation, so they are surfaced for review / reward down-weighting, not
  dropped (blocking them would discard correct kernels whose only problem was a
  mislabeled target arch). Complements the portable-prompt nudge in #51.

Tests: detector (removed/kept/none), episode scan (failed->removed flagged,
prior-passed or guard-kept not flagged), trajectory helper.
@ai-hpc
ai-hpc merged commit 2481f5d into main Jul 26, 2026
1 check passed
@ai-hpc
ai-hpc deleted the feat/flag-guard-removal-repairs branch July 26, 2026 09:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant