[DIAGNOSTIC - DO NOT MERGE] fix(load_tokenizer): default to use_fast=True

[hallerite]

Summary

load_tokenizer currently calls AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False) without specifying a backend. AutoTokenizer is a factory that picks between two backends:

• TokenizersBackend (Rust, fast) — the desired path.
• PythonBackend (pure Python) — has a latent crash in its __init__.

PythonBackend.__init__ calls _add_tokens(...), which calls self.get_vocab(). In transformers/tokenization_utils_base.py:1439:

def get_vocab(self):
    raise NotImplementedError()

PythonBackend does not override this. So whenever HF silently falls back to the Python path, pool construction raises a bare NotImplementedError that surfaces to env workers as ModelError() -> NotImplementedError().

Reproduction

Observed in prime-rl rollouts on PrimeIntellect/GLM-4.5-Air with trust_remote_code=False:

• Intermittent at batch_size=256, rollouts_per_example=8.
• Reliable at batch_size=512, rollouts_per_example=16.

The identical AutoTokenizer.from_pretrained call returned TokenizersBackend on the head node and PythonBackend on the compute nodes — likely a race in HF's backend selection under concurrent first-time loads.

Stack:

RendererPool.__init__ -> factory() -> load_tokenizer()
  -> AutoTokenizer.from_pretrained(GLM-4.5-Air, trust_remote_code=False)
  -> tokenization_python.py:_add_tokens
  -> self.get_vocab()
  -> tokenization_utils_base.py:1439  raise NotImplementedError()

Fix

Pass use_fast=True explicitly. This forces the Rust path. If fast tokenizer files are genuinely missing, AutoTokenizer raises a clear OSError instead of silently routing to the half-implemented Python class.

This is also a latent upstream bug in transformers (PythonBackend._add_tokens -> get_vocab is unreachable), but the renderers-side default is the right call regardless.

🤖 Generated with Claude Code