Background:
We currently use FastChat's conversation template (src/utils/chat_template_manager.py).
Problem:
- FastChat's conversation template is not maintained anymore (last commit 2 years ago).
- Users often have to modify the src/utils/chat_template_manager.py to add new chat templates for models that are not yet implemented.
- FastChat's conversation template is a lot of lines
Solution:
- HuggingFace has their own chat template, and its nicely abstracted away where one can customize chat templates quite easily with jinja. I think this is much easier than our current FastChat modification.
Additional Notes:
- This refactor is quite large as many things depend on the FastChat conversation template we use. This is mainly due to our reliance on hardcoding special tokens for each model. We should instead use all of the provided functionalities from the HuggingFace API for finding these special tokens in our prompt.
Background:
We currently use FastChat's conversation template (
src/utils/chat_template_manager.py).Problem:
Solution:
Additional Notes: