Ask every miner the same question, and stop paying for the easy tier - #3
Open
bitzic wants to merge 9 commits into
Open
Ask every miner the same question, and stop paying for the easy tier#3bitzic wants to merge 9 commits into
bitzic wants to merge 9 commits into
Conversation
Three defects in the path that ranks and pays miners, all visible in the two scored rounds. PROMPT PARITY. HFRunner wrapped every prompt as a user turn; GGUFStudentRunner has no chat path at all. So a safetensors submission and a GGUF submission were asked different questions and then ranked against each other, and the parent's reference step was produced the safetensors way — GGUF miners were compared to a prompt they never saw. A trajectory prefix is a transcript to be continued, not a message to reply to, so continuation is also the semantically right reading. - new runners.continuation(): asks for a raw continuation from any runner, and is a no-op for the ~15 sim runners and doubles that never had a chat path. Inspects the signature rather than catching TypeError, which would swallow a TypeError raised from inside a runner - routed observer_round (both legs), the freeze, ladder_probe and batch_invariance through it; the last two compare regenerated steps against scored ones and would otherwise report drift that was entirely their own DEGENERACY. Round 2's ternary tier took four entries and three were broken; one emitted subword salad, another collapsed to `0 0 0 0`. degeneracy_flags was imported by the live loop and never called, and the floor that did apply was retention_lb > 0.02, so only one miner uploading a working model kept a broken quantiser off the throne. - call it, and extend it: the salad defeats the n-gram and length tests completely. Repeated-character runs catch it; script-mixing was the tempting signal and is wrong, since a genuine Chinese step in this corpus switches script more often than the salad does - validated against the real recorded steps of all 11 submissions, vendored as a fixture: 3 rejected, 8 kept, no false positives on markdown, tables, code, Hindi, Chinese or arithmetic EMISSION. Tiers paid equally and Tournament.weights normalised over tiers that HAD a king, so the two empty tiers funded the two occupied ones and sub4 — one llama-quantize invocation — collected half of everything. An equal split predicts exactly the field we drew: 9 sub4, 6 ternary, nothing in binary or sub2. - difficulty-weighted shares, binary 0.40 down to sub4 0.10 - an unclaimed tier's share is no longer redistributed; weights() may sum to less than 1 and unclaimed() reports the remainder - carried through the record and burned at set_weights, without which the renormalisation would have handed it back and the change would be cosmetic - `unclaimed` is SPARSE: both anchored records still digest to what they signed Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The crowns live in miner repos with 116 and 119 downloads, named after a miner, with nothing saying they are netuid 40's champions. `main` happens to point at the scored commits today; the miner can push over it whenever they like. - eval/publish_champions.py mirrors every occupied throne into one repo, one GGUF per tier, the same shape as any multi-quant repo so PocketPal and Enclave list both and a user picks by size - it VERIFIES rather than copies: each artifact is fetched at the pinned revision from the record, re-hashed with identity.content_hash, and published only if it matches the record's model_id. Confirmed end-to-end against the live ternary crown — 2.43 GB fetched, hash reproduced exactly - refuses an artifact_uri with no pinned revision; skips a tier already published at the same model_id; dry-run by default, --push to upload - champions.json records source repo, revision, round, retention and miner, so a reader can fetch the original and check it themselves - generated README, because a repo holding one file per tier needs a page about the choice between them rather than a card about one artifact koth.kings_from_events: `hold` counts as an occupied throne. model_card .from_record scanned only crown/dethrone and kept the last it saw, so a round that held both crowns raised "no crowned challenger", and with several tiers live it wrote one model's card for every tier. Now per-tier. publish_champions prefers the incumbent row when a crown was held: the same bytes scored 0.3030 as challenger and 0.2875 as incumbent in round 2, and the re-score is the number the crown decision used. test_a_sparse_field_cannot_move_an_already_published_digest is now generic over _SPARSE — it reconstructed the legacy payload by deleting one hard-coded key, so the second sparse field broke it while the property still held. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Option B: weights follow product value, not difficulty. The goal is a model on people's phones, and sub2 is the only tier that produces one. - no burn. A subnet writing part of its weight vector to a burn uid reads as taxing its own miners. What fixed the original problem was the weights being unequal, not withholding emission: the same field that paid sub4 50% under an equal split now pays it 37%, and taking sub2 moves a miner to 50% - unclaimed() is reporting only, so a record can say "a four-tier schedule paid two tiers" instead of leaving a concentrated payout looking accidental - sub2 cap 2.0 -> 2.3. Q2_0 group-64 is 2.25 bpw, so at 2.0 the one phone-native format could enter no tier but sub4, where models carrying twice the bits beat it. A tier built for a format that could not enter it - weights: sub2 .40, ternary .25, binary .20, sub4 .15. Binary is the hardest problem and pays third because its winner needs a private fork to run GGML type table had no Q1_0 or Q2_0, so read_gguf returned None and refused the artifact — sub2 could not have accepted a submission in the format it exists for. Ids and block layouts read from llama.cpp's ggml.h / ggml-common.h: Q1_0 = 41, 128 elems / 18 bytes = 1.125 bpw; Q2_0 = 42, 64 / 18 = 2.25 bpw. bitrate: detect the group size instead of assuming 128. A genuinely ternary group-64 artifact, truly 1.835 bpw, measured 2.447 and was thrown out of ternary into sub4 — normalising a 128-wide window holding two 64-wide groups turns 3 levels into 5. Minimising over candidate groups is the right selector: scale overhead is scale_bits/group_size, so claiming a smaller group than you used makes your number worse. Affects safetensors only; GGUF reads its type table. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corrects the previous commit, which weighted emission on the belief that only Q2_0 reached a phone and that sub-2-bit needed PrismML's private fork. Checked against mainline llama.cpp's Metal sources: ggml-metal.metal carries get_rows/dequantize kernels for q1_0, q2_0, tq2_0, iq1_s, iq1_m, iq2_xxs and q4_K. Q1_0 — PrismML's 1.125 bpw format — has been upstreamed. Every tier here produces something a phone can run. With that correction three orderings coincide, so the ladder is steepest at the bottom: fewer bits is a smaller file, a smaller file fits more phones and decodes faster (decode is bandwidth-bound, so halving the weights roughly doubles tokens/sec), and fewer bits is the harder research problem. - emission: binary .40, ternary .25, sub2 .20, sub4 .15 - UNRUNNABLE_FORMATS gate. TQ1_0 fits the binary and ternary budgets at 1.6875 bpw and has NO Metal kernels in mainline, so a crown in it would be a champion that crashes on every iPhone. Refused by every tier; its sibling TQ2_0 passes, so this cannot be waved through as "the ternary formats are broken" - BitReport.formats carries the GGUF type histogram, kept separate from per_tensor, which is keyed by tensor name on the safetensors path and by type name on the GGUF one The moving-up test compared a two-tier field against a three-tier one and read the extra dilution as a lost incentive. Renormalisation moves every share when a tier is added, so the comparison now holds the live tiers fixed. Both emission tests derive from TIER_EMISSION_WEIGHT instead of hard-coding it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page reported `rounds published: 2` and, separately, how many miners had committed. A count is not a history: it could not say what happened in round 2, how often rounds run, which of them paid anyone, or whether anybody has submitted since the last one closed. Both halves were on the page and nothing joined them. - trail.rounds: per-round summary from the signed records — scored, rejected with reasons, by tier, crowns, exam size, parent, observer, weights. Newest first, capped at 12, and the cap travels with the list because a bounded history that does not say so reads as the whole history - chain.cohort.pending: what is committed that the newest round did NOT score, splitting NEW ENTRANTS from RESUBMISSIONS. Different miners, different meaning — this field iterates on recipe far more often than it grows - a miner rejected last round is not reported as a new entrant; they were seen and told why, and calling them new would erase the rejection - records fetched through the same verifier as the headline crowns, and a round that fails to verify is dropped AND NAMED: a gap in a timeline is indistinguishable from a round that never happened - an unreadable trail yields no timeline rather than an empty one, same rule that already governs every other number on the page RoundRecord gains started_at/published_at, sparse, so rounds 1-2 still digest to what they signed and render as "not recorded" rather than as instant. Phase is declared via RALPH_FIRST_LIVE_ROUND, not inferred: a shakedown round computes a weight vector exactly like a live one, so nothing in a record distinguishes them, and the default classifies everything as shakedown. Incumbents are counted separately from challengers throughout. A held crown appears twice in a record, so len(submissions) says 13 for a round that scored 11 miners. UI: RoundTimeline + PendingPanel, 8 new status tests, next build clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The SDK was documented in requirements.txt as a version range and never pinned
anywhere installable, so the only working copy on the dev box belonged to
another project and whatever happened to be there was what ran.
- requirements-chain.txt (bittensor>=11.1,<12) and requirements-gpu.txt, both
real `pip install -r` targets. The base file stays wallet-free and GPU-free so
the crown-path suite remains installable on a box with neither
11.x is the ground-up rewrite the old docstring warned about: Subtensor has no
get_current_block, metagraph, get_commitment, set_commitment, set_weights or
.substrate. Ported and verified against finney:
- block -> st.block; block hash -> st.block_info(n).hash
- storage via typed items: st.query_map(item, params). uid<->hotkey from
SubtensorModule.Keys, coldkey from SubtensorModule.Owner
- mg() builds its own {uid: hotkey} rather than bt.metagraph.fetch, which is a
COROUTINE in 11.x while the whole validator is synchronous
- writes build a call and submit it: bt.SetWeights / Commitments.set_commitment
through one _submit(), so there is a single line to audit for what can sign
- _item() isolates the typed-storage lookup. 10.x addressed storage by strings,
so this module was stubbable with no SDK installed — a property requirements
.txt states out loud; one indirection keeps it true
Live against netuid 40: 256 uids, 153 commitment slots, 16 v2 envelopes
(sub4 9, ternary 6, sub2 1).
_decode_commitment is unchanged and still right — it WALKS the RawN fields
rather than indexing them. A throwaway decoder that read only the first field
reported ZERO v2 commitments on a subnet that has sixteen.
_coldkey_of used getattr(v, "value", "") and returned empty for every hotkey:
11.x hands back a plain str for that item, so the default won. Falls back to the
object itself now.
Test stubs model the SDK boundary, so they move with it. The empty-subnet stub
had to be emptied on BOTH read paths — an empty map makes read_commitments fall
back to the per-uid read, and a stub that empties only the map keeps serving
commitments through the fallback and stops being an empty subnet at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Confirmed on chain, not inferred: uid 0 on netuid 40 is 5HijSRHd9wUmk51UE8Kia7vmx6kD2jwqJLn9bY1frQ4aiTUs — the validator's own hotkey. Bittensor has no designated burn address; uid 0 is whoever registered first, and here that is us. `set_burn_weights` defaults to burn_uid=0 and is called by the AUDITOR, which third parties are meant to run. With the shipped default an independent auditor would have written 100% of its weight to us and called it a burn. That is a self-vote wearing the word "burn", and it would be found. Refuses and says why, pointing at RALPH_BURN_UID. The round path never calls this — emission now splits across occupied tiers and nothing is withheld — so this only affects an auditor with no verified round to pay. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
publish_champions -> publish_crowns, and the round calls it. A crown nobody can find is not a product: the artifacts live in miner-named repos with ~116 downloads, and `main` there can be pushed over at any time. - eval/publish_crowns.py, default RALPH_CROWNS_REPO=RalphLabsAI/ralph-crowns, manifest crowns.json - run_orchestrated calls it LAST, after publish, anchor and weights, so a mirroring failure leaves a round that is still scored, signed, anchored and paid. publish() cannot raise — a bug here must not convert a successful round into a failed one at the final step - RALPH_PUBLISH_CROWNS=0 turns it off; on by default - per-tier isolation: a deleted repo, a missing revision or an absent dependency skips ONE tier and says so. It also exits non-zero when nothing was staged but something failed, or "nothing new to publish" would look healthy - idempotent, so an unchanged crown costs one small read instead of re-uploading gigabytes every round A HELD CROWN LEAVES TWO ROWS AND NEITHER IS PUBLISHABLE ALONE. The incumbent carries the re-score the crown decision used (0.2875, not the challenger's luckier 0.3030 — the 0.0155 gap is the measured run-to-run floor); the challenger carries the bit measurement, because an incumbent is re-scored rather than re-ingested and its code_bits/container_bits/params are zero. Taking the incumbent wholesale rendered a 4.61 GB model as "0.0 bits, 0.00 GB" — caught in a dry run on the box. Now merged, and pinned by a test. Verified against the live trail: sub4 4.61 GB / 4.0 bits / 0.2875, ternary 2.42 GB / 1.714 / 0.2406. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The Submissions table shows the CURRENT cohort — who is committed now and what the running round is doing with them. It cannot answer "who was in round 1": a commitment slot holds only its latest value, so once a miner resubmits, nothing anywhere still says who that round scored. The signed record is the only durable answer, so the field belongs with the round. - trail.rounds[].submissions: miner, tier, role, retention, retention_lb, code_bits, container_bits, artifact_uri, crowned, gates_ok, reasons. Best retention first - PROJECTED HARD. The record's submissions carry steps, effects, slices and per_point — megabytes that exist so an auditor can recompute a score and have no business in a snapshot a browser polls. Asserted absent, along with the exam fields that must never appear mid-round. 10.5 KB for two rounds, ~63 KB at the 12-round cap - incumbents are shown and LABELLED rather than hidden: the re-score is half of the paired dethrone test, so the margin is unexplainable without it. It carries no bit measurement (re-scored, not re-ingested) and renders as a dash, never 0.00 - gated submissions show the reason. Scored-but-gated and simply-low-scoring look identical from a number alone, and only the first is actionable - rejections listed under the round they were rejected from UI: collapsible <details> per round, no client state, inside the same table so the summary columns stay aligned across rounds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three defects in the path that ranks and pays miners, all visible in the two scored rounds.
PROMPT PARITY. HFRunner wrapped every prompt as a user turn; GGUFStudentRunner has no chat path at all. So a safetensors submission and a GGUF submission were asked different questions and then ranked against each other, and the parent's reference step was produced the safetensors way — GGUF miners were compared to a prompt they never saw. A trajectory prefix is a transcript to be continued, not a message to reply to, so continuation is also the semantically right reading.
DEGENERACY. Round 2's ternary tier took four entries and three were broken; one emitted subword salad, another collapsed to
0 0 0 0. degeneracy_flags was imported by the live loop and never called, and the floor that did apply was retention_lb > 0.02, so only one miner uploading a working model kept a broken quantiser off the throne.EMISSION. Tiers paid equally and Tournament.weights normalised over tiers that HAD a king, so the two empty tiers funded the two occupied ones and sub4 — one llama-quantize invocation — collected half of everything. An equal split predicts exactly the field we drew: 9 sub4, 6 ternary, nothing in binary or sub2.
unclaimedis SPARSE: both anchored records still digest to what they signed