Low-priority tiktoken compat features not needed for initial release:
encode_with_unstable(text, ...) -> tuple[list[int], list[list[int]]]: returns stable prefix tokens + possible suffix completions. Marked unstable in tiktoken itself.
decode_with_offsets(tokens) -> tuple[str, list[int]]: decodes tokens and returns character offset per token. Requires offset tracking during decode.
Encoding(name, *, pat_str, mergeable_ranks, special_tokens, explicit_n_vocab) constructor: for building custom encodings from raw data.
Ref: #369
Low-priority tiktoken compat features not needed for initial release:
encode_with_unstable(text, ...) -> tuple[list[int], list[list[int]]]: returns stable prefix tokens + possible suffix completions. Marked unstable in tiktoken itself.decode_with_offsets(tokens) -> tuple[str, list[int]]: decodes tokens and returns character offset per token. Requires offset tracking during decode.Encoding(name, *, pat_str, mergeable_ranks, special_tokens, explicit_n_vocab)constructor: for building custom encodings from raw data.Ref: #369