A data-first attack on the NVIDIA Nemotron Model Reasoning Challenge (863 hidden-rule puzzles across nine categories). Instead of treating the benchmark uniformly, this project measures where the residual difficulty actually lives, decomposes the hardest category into two structurally distinct modes, and routes each mode to the solver class that can handle it. Both the positive and the negative results are reported with numbers.
| Result | Number |
|---|---|
Residual unsolved mass concentrated in cryptarithm_deduce |
47 of ~66 unsolved puzzles (77% of the category) |
| Mode A permutation-DSL solver, validation accuracy | 7/7 on gold-labeled cryptarithm_deduce puzzles |
| PBE version-space solver, raw predictions on unknown puzzles | 30 of 47 |
| PBE predictions passing the trust filter (admitted to SFT corpus) | 4 (medium-trust, multi-example-constrained) |
Frontier-LLM teachers on Mode B (cryptarithm_deduce residue) |
0 usable traces across DeepSeek R1, GPT-5.5, Claude Opus 4.8 |
Depth-3 functional DSL on residual bit_manipulation |
0/8 (published negative result) |
| Released augmented SFT corpus | 419 rows (augmented_corpus.jsonl) |
The benchmark presents 863 puzzles in nine categories (bit_manipulation, cipher, cryptarithm_deduce, cryptarithm_guess, equation_numeric_deduce, equation_numeric_guess, gravity, numeral, unit_conversion). Each puzzle gives 3–10 (input, output) examples generated by a hidden rule and asks for the output on a held-out input, scored by exact string match.
The strongest public pipeline (tonghuikang's solver-then-teacher cascade, ~0.85 LB) treats the task uniformly. Re-expanding its collapsed category buckets and measuring per-category unsolved mass shows the headroom is not uniform at all:
| Category | Puzzles | Unsolved by prior pipeline | % unsolved |
|---|---|---|---|
| bit_manipulation | 157 | 8 | 5.1% |
| cipher | 152 | 0 | 0.0% |
| cryptarithm_deduce | 61 | 47 | 77.0% |
| cryptarithm_guess | 12 | 6 | 50.0% |
| equation_numeric_deduce | 43 | 1 | 2.3% |
| equation_numeric_guess | 7 | 4 | 57.1% |
| gravity | 146 | 0 | 0.0% |
| numeral | 135 | 0 | 0.0% |
| unit_conversion | 150 | 0 | 0.0% |
| Total | 863 | 66 | 7.6% |
Five categories are effectively closed by existing deterministic solvers. Nearly all remaining leaderboard headroom is cryptarithm_deduce.
Each cryptarithm_deduce instance is a five-character string whose middle character is an operator, and the hidden rule branches on the operator. Empirically the category splits into two modes:
- Mode A (positional permutation). For most
*and+cases, the output is a permutation (with repetition/dropping) of the four non-operator input characters. Solvable by exhaustive enumeration over a tiny positional DSL. - Mode B (substitution + arithmetic). For
-,/, and non-standard operators, output characters do not appear in the input at all — a positional DSL provably cannot produce them. The hidden rule is a symbol-to-number bijection followed by arithmetic in an unstated base, rendered back through the inverse bijection.
This decomposition explains mechanistically why a uniform permutation-style solver leaves 77% of the category unsolved: it only fits the Mode A subset.
Enumerates all output patterns (p_0, …, p_{k-1}), p_i ∈ {0..3}, k ≤ 6, grouped by operator; retains patterns consistent with every example; answers only when all surviving patterns agree. 7/7 on the gold-labeled validation slice. On the 47 unknown puzzles it cracks the Mode A subset and abstains with a structured rationale otherwise — the abstention cleanly identifies Mode B puzzles for routing.
A version-space-algebra engine over three atoms: PICK_POS(k), CONST(c), and SUBST(k, σ) (learned bijection applied to the character at position k). Validates 7/7 on the gold slice and produces 30 raw predictions on the 47 unknown puzzles, tiered by trust:
- Medium trust (bijection constrained by ≥2 examples): 4 puzzles — admitted to the SFT corpus.
- Low trust (1 example, mixed atoms): 9 puzzles — documented, excluded.
- Low-trust CONST-copy (mostly literal constants): 17 puzzles — high overfit risk, excluded.
On the cryptarithm_guess and equation_numeric_guess categories the question's operator is by construction absent from the examples; the operator-independent fallback produced 0 safe cracks, supporting the conclusion that these sub-categories require operator extrapolation no rule-miner can perform without an explicit transfer prior.
Mode B puzzles were routed to three frontier reasoning models via OpenRouter with a verification-first system prompt. On a representative instance (00457d26), all three failed in three different ways:
- DeepSeek R1 silently dropped leading backticks/backslashes from every example and produced a confident answer against a corrupted view of the puzzle — unsafe for SFT.
- GPT-5.5 enumerated hypotheses, found none consistent, and honestly answered
uncertain— desired behavior, but no usable SFT row. - Claude Opus 4.8 was still mid-analysis when the 16,000-token budget ran out — no misread, just unfinished.
The same models solve numeral puzzles in seconds, which localizes the failure to Mode B's combination of dense ASCII metacharacter sequences (tokenizer-hostile) and an example count below the information-theoretic floor needed to pin the rule down. The cross-model agreement filter admits 0 Mode B teacher traces to the corpus. Raw traces are in traces/ and traces_v4/.
A functional DSL enumerator (depth ≤ 3 over {NOT, ROL/ROR/SHL/SHR_k, reverse, swap_nibbles, swap_pairs, XOR, AND, OR, AND_NOT, OR_NOT}) returns 0/8 on the residual bit_manipulation puzzles: they are not expressible as depth-3 programs over this family. Depth-4-with-constants or SMT-backed synthesis is the natural follow-up; it was deliberately not run given the 8-puzzle payoff.
Fine-tuning experiments (v3_stacked.py, v4_balanced.py, hyperband_search.py, adapter_soup.py, splice_*.py)
Kaggle training runs stacking: warm-start from a public high-scoring adapter, micro-skill augmenters, per-token branch-weighted cross-entropy, stratified category batching, Hyperband LR/steps/delta search, and multi-seed LoRA soup (v3) later simplified to single-seed with inverse-proportional category sampling (v4) after soup showed no gain at rank 32. The splice_*.py scripts deterministically assemble the runnable Kaggle notebooks from these sources.
No data is distributed in this repository — it contains code and documentation only. The pipeline produces the following artifacts when run locally:
| Artifact | Rows | Contents |
|---|---|---|
augmented_corpus.jsonl |
419 | Augmented SFT corpus (andy279-compatible schema: messages, category, source, puzzle_id, predicted_answer) |
failure_anatomy.jsonl |
74 | Structured per-puzzle failure analysis |
puzzle_typing.jsonl |
74 | Per-puzzle structural type labels |
surface_variants.jsonl |
271 | Surface-form augmentation variants |
pbe_cracked_*.jsonl |
30 raw / 4 safe | PBE predictions with trust tiers |
traces/, traces_v4/ |
— | Raw frontier-model teacher traces |
The corpus schema extends (rather than replaces) the public andy279 corpus, so it concatenates directly into existing training scripts. The source field enables per-teacher ablations.
solver_v1.py Mode A positional-permutation DSL solver
pbe_skeleton.py PBE version-space solver (PICK_POS / CONST / SUBST)
pbe_z3_cryptarithm.py Z3-backed cryptarithm PBE experiments
z3_bitmanip.py Depth-3 functional DSL enumerator (negative result)
llm_teacher.py Frontier-LLM teacher harness (OpenRouter), trace persistence, SFT export
llm_cryptarithm_teacher.py Cryptarithm-specialized teacher harness
gen_failure_anatomy.py Failure-anatomy corpus generator
gen_puzzle_typing.py Structural typing corpus generator
gen_surface_variants.py Surface-variant augmentation generator
merge_corpus.py Corpus merge/dedup
v3_stacked.py, v4_balanced.py, hyperband_search.py, adapter_soup.py
Kaggle training runs and search harnesses
splice_*.py Deterministic notebook assembly from the .py sources
dry_run.py End-to-end smoke test without API calls
API keys are read from environment variables only (OPENROUTER_API_KEY, etc.); nothing is hardcoded.
- Mode B has an information-theoretic floor. Puzzles exposing 3–5 examples of a multi-operator substitution-arithmetic system do not contain enough constraint to identify the transformation uniquely. We report cracked/total honestly and admit nothing speculative to the corpus.
- Bit-manipulation analysis stops at DSL depth 3 without constants. Partial returns should be expected from deeper SMT-backed search.
- The final leaderboard ablation (baseline vs. augmented corpus) was not completed, so no leaderboard delta is claimed for the released corpus.
- tonghuikang — prior-art pipeline and public code (https://github.com/tonghuikang/nemotron); the andy279 augmented corpus (https://huggingface.co/datasets/andy279/nemotron-reasoning-challenge).
- Public Kaggle notebooks and forum findings from the Nemotron challenge community (kuangyicheng, Kien Tinker, Taha, lopure) informed the fine-tuning experiments; specific borrowings are credited in the module docstrings.
- Raw benchmark puzzle data belongs to the NVIDIA Nemotron Model Reasoning Challenge and is not redistributed in this repository.
Apache License 2.0 — see LICENSE.