Guided decoding with xgrammar for TurboMind

[windreamer]

Motivation

LMDeploy’s TurboMind backend is the fastest inference stack in the ecosystem, yet it still lacks Guided Decoding – a feature that is already available in the PyTorch backend and heavily requested by the community.
This PR closes the gap by bringing token-level, C++ native Guided Decoding to TurboMind while keeping the API 100 % compatible with the existing PyTorch backend.
The implementation is built on xGrammar (Apache-2.0), a high-performance C++ library that compiles JSON / Choice / Regex grammars into token FSMs and applies them with negligible overhead.

Modification

1. Build-system

   • Add xgrammar as a header-only dependency via CMake FetchContent (CUDA & Python bindings disabled).
   • Export xgrammar::tokenizer_info and xgrammar::grammar_compiler symbols under lmdeploy::xgrammar.

2. Core C++ changes

   • DynamicDecodeLayer pipeline extended with two new layers:
     • GuidedDecodeMaskLayer: in setup() compiles / reuses grammar → builds per-request token bitmask; in forward() launches a light CUDA kernel to mask disallowed logits to -INF.
     • GuidedDecodeUpdateLayer: in forward() calls matcher->AcceptToken(output_id) to advance the FSM.
   • Grammar compiler cache (LRU, keyed by schema hash) shared across all sessions to avoid re-compilation.

3. Python frontend

   • Re-use existing guided_decoding utilities from PyTorch backend; no new API surface.
   • turbo.TurboMindEngine now accepts the same response_format= / guided_json= / guided_choice= arguments.

Checklist

• Pre-commit hooks (clang-format, flake8, mypy) passed.
• Document updated

[shell-nlp]

good job!

[windreamer]

Could we split this PR into two separate ones? One for the TurboMind engine and another for the PyTorch engine.

Done

[windreamer]

I don't know much about guided decoding. But I think there are bugs in the pytorch implementation (from the main branch)
The matcher is maintained in instances of RegexLogitsProcessor or JSONLogitsProcessor. And _get_guided_logits_processor will only cache 32 instances. Different request with same guide would get the same processor and old processor would be removed if more than 32 guide request comes in.

You are right! It is a bit tough...

Should tenatively solved in #4028

[lvhan028]

May update the "structed_output.md"

[windreamer]

May update the "structed_output.md"

Done