Grammar API v2

[mhayes853]

Introduces a Grammar API to enable structured generation in coming PRs.

For now, this isn't directly integrated with completion, but rather exposes all the necessary APIs to make that happen in the near future. The underlying state machine used is XGrammar, and for the most part the implementation directly wraps it.

There are 4 essential types:

• Grammar an actual representation of a grammar, with conveniences for plain JSON, JSON schema, regex, empty, universal, structural tags, etc.
  • Also supports mathematical operations such as unioning and concatenating, as well as getting the underlying EBNF string.
• GrammarMatcher the state machine type responsible for computing bitmasks.
  • accept and next_bitmask do most of the driving. However, there are also methods to rollback, and to check the termination state.
  • fork is an efficient way to create a copy of matcher (at it's current state) without having to go through GrammarEngine.
• GrammarEngine responsible for compiling GrammarMatcher instances from a Grammar instance.
  • Under the hood, this type wraps both GrammarCompiler and CompilerGrammar from XGrammar. I felt it was a conceptually simpler API to keep these concepts bound together.
    • I also felt exposing the concept of a "compiled" grammar was a bit unecessary since it's only purpose is to be used in matcher construction, and the matcher already has fork which can be used for efficient copies.
  • The class exposes a singular compile_matcher method which takes a Grammar instance and returns a GrammarMatcher.
• GrammarVocabulary tokenizer info required by the other grammar types.
  • This type merely wraps TokenizerInfo from XGrammar, there is also now a new tokenizer method to produce it.
  • At the FFI level, this is constructed from a model instance/path directly.

Among other things, I also extracted a helper for constructing the tokenizer from the model path, and also updated the parsing logic for the HF tokenizer file within the tokenizer. This is mainly because the grammar matcher needs the tokenizer info, but the only way to load the tokenizer previously was to go through the entire model initialization (when only the tokenizer is needed). XGrammar also provides a transitive dependency on picojson for proper JSON parsing which should be useful going forward.