Skip to content

Steer teacher prompts to portable, non-hacky kernels (dataset quality) - #51

Merged
ai-hpc merged 1 commit into
mainfrom
feat/portable-prompts-no-arch-guards
Jul 26, 2026
Merged

Steer teacher prompts to portable, non-hacky kernels (dataset quality)#51
ai-hpc merged 1 commit into
mainfrom
feat/portable-prompts-no-arch-guards

Conversation

@ai-hpc

@ai-hpc ai-hpc commented Jul 25, 2026

Copy link
Copy Markdown
Member

Improves miner dataset quality at the generation source — a small, prompt-only change with no trust/hash-boundary impact that helps every miner immediately.

Problem (found in the live merged data)

Reviewing gittensor-model-hub/sparkproof-mining (1,029 rows) surfaced a validator-hack repair pattern: a kernel prompted "for Hopper SM90" adds a hard SM/compute-capability assertion to its self-test, which then fails on the actual SM120 (Blackwell) validation GPU:

validator: RuntimeError: This test targets Hopper SM90, found SM120
assistant: "The false architecture rejection is removed … does not require an exact-SM90 guard"

The "successful" repair passed by deleting the guard, not fixing the kernel — teaching a questionable debugging habit and burning a repair round. Root cause: the prompt induces an arch assertion that the real hardware can't satisfy.

Fix (4 prompt edits)

  • default_system + IMPLEMENTATION_SECTION — teachers must write portable code with no device / compute-capability / SM / architecture asserts (SparkProof attests the GPU separately; the code must run on whatever GPU validates it). Kills the failure class at the source.
  • validator_feedback_content + _repair_prompt — the repair step must fix the root cause, not pass by deleting assertions, weakening the torch.allclose test, or loosening tolerances. Discourages the validator-hack directly.

Impact

  • Fewer wasted repair rounds (the arch-guard failure largely disappears).
  • Repairs that do happen fix real bugs instead of learning to delete checks.
  • Works even when a task's arch label doesn't match the runtime GPU, because kernels stop embedding hard guards.

Tests

354 pass, ruff clean. New test_prompts_discourage_arch_guards_and_validator_hacks pins the guidance so it can't silently regress.

Independent of the Kimi-K3 teacher swap (PR #50) — this branches from main and only touches prompt text. A natural follow-up (separate PR) is a release-gate/anti-cheat check that flags repairs which pass by removing an assertion.

Analysis of the merged sparkproof-mining dataset found repairs that "passed"
by deleting a hardware/architecture assertion the model itself had added: a
kernel prompted "for Hopper SM90" writes an SM90 guard, then fails on the real
SM120 validation GPU (RuntimeError: targets Hopper SM90, found SM120), and the
repair removes the guard rather than fixing the kernel — teaching a validator-
hack habit and wasting a repair round.

Fix at the generation source (prompt-only, no trust/hash impact):
- default_system + IMPLEMENTATION_SECTION: instruct teachers to write portable
  code with NO device/compute-capability/SM/architecture assertions (SparkProof
  attests the GPU separately; the code must run on the validation GPU as-is).
- validator_feedback_content + _repair_prompt: tell the repair step to fix the
  actual root cause and NOT pass by deleting assertions, weakening the
  torch.allclose test, or loosening tolerances.

Prevents the arch-guard failure class and the validator-hack repairs it induces,
improving quality for every miner. Test pins the guidance.
@ai-hpc
ai-hpc force-pushed the feat/portable-prompts-no-arch-guards branch from 1e015fb to b52b831 Compare July 26, 2026 08:54
@ai-hpc
ai-hpc merged commit 8fa740f into main Jul 26, 2026
1 check passed
@ai-hpc
ai-hpc deleted the feat/portable-prompts-no-arch-guards branch July 26, 2026 08:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant