Skip to content

ops: record measured server, tokenizer, data and FineWeb2 probe state - #12

Merged
AuraIis merged 3 commits into
mainfrom
ops/record-machine-measurements
Jul 10, 2026
Merged

ops: record measured server, tokenizer, data and FineWeb2 probe state#12
AuraIis merged 3 commits into
mainfrom
ops/record-machine-measurements

Conversation

@AuraIis

@AuraIis AuraIis commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Measured machine-lane values

  • Python 3.11.12, torch 2.7.0+cu128, Triton 3.3.0, CUDA build 12.8
  • NVIDIA RTX PRO 5000 Blackwell, sm_120, 48 GB
  • required compatibility override TRITON_OVERRIDE_ARCH=sm89
  • exact tokenizer path, size, 200k vocabulary and full SHA-256
  • exact training-bin paths, byte sizes and token counts
  • exact 7,461,038,089-token distinct German inventory
  • source paths/sizes and the completed FineWeb2-DE source SHA-256
  • raw FineWeb2-HQ probe report: 45.11% dropped against german_fresh on shard 1/570

Evidence boundaries

The probe is recorded as one shard against one reference using the legacy report. Its single-reference gross projection is ~15.64B retained tokens; this is not presented as the all-reference full-pull result.

The full pull is approved after disk preflight, but promotion remains blocked until the main-branch dedup script produces a complete all-reference report.

Remaining launch blockers

  • exact launch commit from the clean server checkout
  • basis-checkpoint file manifest hash
  • hashes of the actual tokenized training bins
  • completed four-reference dedup manifest
  • full FineWeb2-HQ all-reference dedup report

Tests lock the delivered measurements and prevent the probe from being treated as full evidence.

@AuraIis
AuraIis merged commit 9723d4f into main Jul 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant