Skip to content

Remove transformers versions with Kimi tokenizer regression#783

Merged
derek-tml merged 1 commit into
mainfrom
derek/ban-kimi-bad-transformers
Jun 24, 2026
Merged

Remove transformers versions with Kimi tokenizer regression#783
derek-tml merged 1 commit into
mainfrom
derek/ban-kimi-bad-transformers

Conversation

@derek-tml

@derek-tml derek-tml commented Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Exclude Transformers releases that force Kimi K2.5/K2.6 through TokenizersBackend, while allowing 5.5.4 where the upstream forced-backend entry for kimi_k25 is removed.

Details

A version audit over the relevant released range found:

  • 4.57.6, 5.0.0, 5.1.0, 5.2.0: no MODELS_WITH_INCORRECT_HUB_TOKENIZER_CLASS forced-backend table
  • 5.3.0: forced-backend table exists, but does not include kimi_k25
  • 5.4.0, 5.5.0, 5.5.1, 5.5.2, 5.5.3: table includes kimi_k25, causing AutoTokenizer to ignore Kimi tokenization_kimi.TikTokenTokenizer auto-map
  • 5.5.4: released on PyPI and no longer includes kimi_k25 in the forced-backend table

This changes the dependency bound to transformers>=4.57.6,!=5.4.*,!=5.5.0,!=5.5.1,!=5.5.2,!=5.5.3,<=5.5.4.

Validation

  • git diff --check
  • parsed pyproject.toml with tomllib and confirmed the dependency spec

@derek-tml derek-tml changed the title [codex] Ban transformers versions with Kimi tokenizer regression Remove transformers versions with Kimi tokenizer regression Jun 23, 2026
@derek-tml derek-tml force-pushed the derek/ban-kimi-bad-transformers branch from 4cde59d to 200b164 Compare June 23, 2026 22:03
@derek-tml derek-tml marked this pull request as ready for review June 23, 2026 22:04
@derek-tml derek-tml requested a review from erikwijmans June 23, 2026 22:06
Comment thread pyproject.toml Outdated
@derek-tml derek-tml force-pushed the derek/ban-kimi-bad-transformers branch from 200b164 to 04a8bda Compare June 23, 2026 22:18
@derek-tml derek-tml marked this pull request as draft June 23, 2026 22:18
@derek-tml derek-tml marked this pull request as ready for review June 23, 2026 22:41
@derek-tml derek-tml requested a review from erikwijmans June 23, 2026 22:42
@derek-tml derek-tml force-pushed the derek/ban-kimi-bad-transformers branch from 04a8bda to 9f0eee0 Compare June 23, 2026 23:02
@derek-tml derek-tml marked this pull request as draft June 23, 2026 23:03
@derek-tml derek-tml force-pushed the derek/ban-kimi-bad-transformers branch from 9f0eee0 to 369a595 Compare June 23, 2026 23:36
@derek-tml derek-tml marked this pull request as ready for review June 23, 2026 23:36
@derek-tml derek-tml merged commit a3a345a into main Jun 24, 2026
4 checks passed
@derek-tml derek-tml deleted the derek/ban-kimi-bad-transformers branch June 24, 2026 05:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants