Skip to content

Python: tiktoken compat nice-to-haves (encode_with_unstable, decode_with_offsets, Encoding constructor) #376

Description

@antimora

Low-priority tiktoken compat features not needed for initial release:

  • encode_with_unstable(text, ...) -> tuple[list[int], list[list[int]]]: returns stable prefix tokens + possible suffix completions. Marked unstable in tiktoken itself.
  • decode_with_offsets(tokens) -> tuple[str, list[int]]: decodes tokens and returns character offset per token. Requires offset tracking during decode.
  • Encoding(name, *, pat_str, mergeable_ranks, special_tokens, explicit_n_vocab) constructor: for building custom encodings from raw data.

Ref: #369

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions