-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Closed
Labels
tech-debtTechnical DebtTechnical Debt
Description
🤔 What is the technical debt you think should be addressed?
When the inference providers were originally created, shared mixins like OpenAIMixin and LiteLLMOpenAIMixin did not exist. As a result, many providers implemented their own logic manually and inconsistently.
Now that these mixins are available and some providers have adopted them, we have a fragmented implementation across the codebase. This results in:
- Duplicated logic (e.g. for streaming, parameter handling, response formatting)
- Inconsistent behavior across providers
- Increased maintenance burden
- Higher likelihood of subtle bugs and divergent implementations
💡 What is the benefit of addressing this technical debt?
- Consistency: All inference providers follow the same behavior.
- Maintainability: Changes (e.g. API updates, bug fixes) can be made in one place.
- Reduced Duplication: Shared logic eliminates repeated code across providers.
- Scalability: Easier to onboard or implement new providers.
- Better Testing: Shared mixins can be tested centrally, increasing reliability.
Inference providers
| provider | chat | completions | embeddings | status | notes |
|---|---|---|---|---|---|
| anthropic | yes | yes | yes | ||
| azure openai | yes | yes | yes | ||
| bedrock | yes | yes | no | #3748 | |
| cerebras | yes | yes | no | ||
| databricks | yes | no | no | ||
| fireworks | yes | yes | yes | ||
| gemini | yes | yes | yes | ||
| groq | yes | yes | yes | ||
| llama | yes | yes | yes | ||
| nvidia | yes | yes | yes | ||
| ollama | yes | yes | yes | ||
| openai | yes | yes | yes | ||
| runpod | yes | yes | yes | ||
| sambanova | yes | yes | yes | ||
| tgi | yes | yes | no | ||
| hf::serverless | yes | yes | no | TODO | BROKEN: #3415 |
| hf::endpoints | yes | yes | no | TODO | |
| together | yes | yes | yes | ||
| vertexai | yes | yes | no | ||
| vllm | yes | yes | no | ||
| watsonx | yes | yes | no | standardized on LiteLLMOpenAIMixin |
cdoern and leseb
Metadata
Metadata
Assignees
Labels
tech-debtTechnical DebtTechnical Debt