Skip to content

ops: record artifact hashes, RAM limit, and resume regression#15

Merged
AuraIis merged 2 commits into
mainfrom
ops/record-resume-and-capacity-findings
Jul 10, 2026
Merged

ops: record artifact hashes, RAM limit, and resume regression#15
AuraIis merged 2 commits into
mainfrom
ops/record-resume-and-capacity-findings

Conversation

@AuraIis

@AuraIis AuraIis commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Artifact identity completed

Records SHA-256 for:

  • all three German training bins
  • selected step_60000.pt, sidecar and run manifest
  • related but explicitly non-selected best.pt and sidecar

The selected launch base remains step_60000.pt; best.pt is not treated as equivalent.

Dedup capacity decision

  • measured 1,447.6 bytes/doc over a combined 1M-document sample
  • 30,441,496 reference documents total
  • projected LSH RSS 44.1 GB against 49 GB available
  • exact-hash set adds an unmeasured ~1.5–3 GB
  • combined index rejected
  • required execution: sequential single-reference passes, expected 6–12 GB peak/pass

The larger source pools are recorded as conservative overlap references without claiming that each was the exact tokenization source.

Resume evidence correction

Replaces the earlier unreproducible “bit-identical” observation with the clean-checkout result:

  • scheduler LR identical
  • eval loss nearly identical
  • training loss diverges immediately
  • root cause: loader-owned language/shuffle RNG streams restart from seed

Launch remains blocked until #14 is merged and the clean-checkout resume smoke passes with the fix.

@AuraIis
AuraIis merged commit 73de330 into main Jul 10, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant