Related to #2 - potentially use a template engine to deal with prompt formats?
Some examples.
Important things (not necessarily in this order):
- Flexibility. Support things like built-in templates for special tokens in various models that we can store/support in repo by name such as "mistral": see mistral example format. We can also extend/include support for transformers chat templating but that will require fetching the tokenizer config for the user configured model. See key "chat_template" for Mistral-Instruct as an example. In the event a user specifies a HF model we can attempt to "autoload" this but it brings up issues of fetching, caching, grabbing specific revisions, etc. Also allow users to extend this to support models they may have finetuned with specific/proprietary/goofy prompt/instruction formats.
- Performance.
- Rust support.
- Use/familiarity/popularity.
- Validation. Not all models support all ChatML/OpenAI roles. Ideally we would validate requests on input against the roles the model actually supports and return error if the user steps outside of these supported roles.
I'm guessing that for performance we can/should use something that supports compilation on startup? In the end what OpenAI clients will provide is often referred to as "ChatML". See the HF chat templating example for input above.
I believe this may be the most powerful, flexible, and performant way to support what is described in #2 for prompt handling and definition.
Related to #2 - potentially use a template engine to deal with prompt formats?
Some examples.
Important things (not necessarily in this order):
I'm guessing that for performance we can/should use something that supports compilation on startup? In the end what OpenAI clients will provide is often referred to as "ChatML". See the HF chat templating example for input above.
I believe this may be the most powerful, flexible, and performant way to support what is described in #2 for prompt handling and definition.