Steer teacher prompts to portable, non-hacky kernels (dataset quality) - #51
Merged
Conversation
Analysis of the merged sparkproof-mining dataset found repairs that "passed" by deleting a hardware/architecture assertion the model itself had added: a kernel prompted "for Hopper SM90" writes an SM90 guard, then fails on the real SM120 validation GPU (RuntimeError: targets Hopper SM90, found SM120), and the repair removes the guard rather than fixing the kernel — teaching a validator- hack habit and wasting a repair round. Fix at the generation source (prompt-only, no trust/hash impact): - default_system + IMPLEMENTATION_SECTION: instruct teachers to write portable code with NO device/compute-capability/SM/architecture assertions (SparkProof attests the GPU separately; the code must run on the validation GPU as-is). - validator_feedback_content + _repair_prompt: tell the repair step to fix the actual root cause and NOT pass by deleting assertions, weakening the torch.allclose test, or loosening tolerances. Prevents the arch-guard failure class and the validator-hack repairs it induces, improving quality for every miner. Test pins the guidance.
ai-hpc
force-pushed
the
feat/portable-prompts-no-arch-guards
branch
from
July 26, 2026 08:54
1e015fb to
b52b831
Compare
This was referenced Jul 26, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves miner dataset quality at the generation source — a small, prompt-only change with no trust/hash-boundary impact that helps every miner immediately.
Problem (found in the live merged data)
Reviewing
gittensor-model-hub/sparkproof-mining(1,029 rows) surfaced a validator-hack repair pattern: a kernel prompted "for Hopper SM90" adds a hard SM/compute-capability assertion to its self-test, which then fails on the actual SM120 (Blackwell) validation GPU:The "successful" repair passed by deleting the guard, not fixing the kernel — teaching a questionable debugging habit and burning a repair round. Root cause: the prompt induces an arch assertion that the real hardware can't satisfy.
Fix (4 prompt edits)
default_system+IMPLEMENTATION_SECTION— teachers must write portable code with no device / compute-capability / SM / architecture asserts (SparkProof attests the GPU separately; the code must run on whatever GPU validates it). Kills the failure class at the source.validator_feedback_content+_repair_prompt— the repair step must fix the root cause, not pass by deleting assertions, weakening thetorch.allclosetest, or loosening tolerances. Discourages the validator-hack directly.Impact
Tests
354 pass, ruff clean. New
test_prompts_discourage_arch_guards_and_validator_hackspins the guidance so it can't silently regress.