Tokenizer Request: larger variants

[kooshi]

The README suggests that multiple variants would be available, but there's only the 1024 version.
https://huggingface.co/datasets/willdepueoai/parameter-golf/tree/main/datasets/datasets

Can we get official tokenizers and datasets up to 16k?

[clarkkev]

I made a sp4096 vocab and got good results with it! The tokenizer and pre-tokenized documents are on HuggingFace. See my README for download/re-creation commands.

[MatoTeziTanka]

All variants are now available on HuggingFace: LightSpeedUp/parameter-golf-data

Pre-tokenized shards + tokenizer files for each:

# SP4096 (81 shards, ~10.9 GB)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp4096/*" --local-dir ./data

# SP8192 (81 shards, ~8.5 GB)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp8192/*" --local-dir ./data

# SP12288 (81 shards, ~7.5 GB)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp12288/*" --local-dir ./data

# SP16384 (81 shards, ~5.7 GB)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_sp16384/*" --local-dir ./data

# Byte260 (198 shards, ~39 GB)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_byte260/*" --local-dir ./data

# Scylla v2 — corrected byte-exact tokenizer (Issue #897 / PR #1314)
huggingface-cli download LightSpeedUp/parameter-golf-data --include "fineweb_scylla_v2/*" --local-dir ./data

Each variant includes the .model/.vocab tokenizer files alongside the shards. Compatible with the existing cached_challenge_fineweb.py loader — set HF_DATASET_REPO=LightSpeedUp/parameter-golf-data.

Background: the Scylla v2 tokenizer fixes the BPB underestimation bug reported in Issue #897 (the ▁ space-token gap). PR #1314 has the corrected tokenizer path. The rest of the variants were requested here and in the Discord — they're now all in one place.

---

@MatoTeziTanka — The Agora