Deprecate non-OpenAI inference endpoint

[reluctantfuturist]

🚀 Describe the new functionality needed

Deprecate non-OpenAI inference endpoint

💡 Why is this needed? What if we don't build it?

Converging on a single API standard

Other thoughts

No response

[mattf]

Plan

• Migrate llama-stack-apps to use OpenAI compat endpoints before removing endpoints
• Unpublish the endpoints in the 0.3.0 release (remove their webmethod decorators)
  • Need to update llama-stack-client-python at the same time
• Migrate internal uses of deprecated apis to alternatives
• Remove implementations of deprecated apis

Inference APIs

API	Legacy	Replacement	Unpublish	Internal uses	Removal
chat completions	/v1/inference/chat-completion	/v1/chat/completions	#3609	agents - #3323, eval - #3588, safety - #3591, scoring - #3635, rag - #3367	#3654
completions	/v1/inference/completion	/v1/completions	#3297	eval - #3588	#3622
embeddings	/v1/inference/embeddings	/v1/embeddings	#3301	vector_io - #3294	#3301
batch chat completions	/v1/inference/batch-chat-completion	/v1/batches - #3162	#3297	none	#3261
batch completions	/v1/inference/batch-completion	/v1/batches - #3309	#3297	none	#3261
batch chat completions	/v1/batch-inference/chat-completion	/v1/batches - #3162	#3261	none	#3261
batch completions	/v1/batch-inference/completion	/v1/batches - #3309	#3261	none	#3261

llama-stack-apps uses

app	completions	chat completions	embeddings
DocQA	none	none	none
AgentStore	none	none	none
Agent Examples	none	none	none
Android Demo App	yes	yes	yes
Client Tools	none	none	none
Interior Design Assistant	none	none	none
iOSCalendarAssistant	none	none	none
iOSQuickDemo	client -> lib	none	none
Cybersecurity Demo	none	none	none
LlamaGuard	none	none	none
LlamaGuardMM	none	none	none

Questions / Notes

• Proposal: Simply remove non-OpenAI compatible endpoints for 0.2.11 REJECTED during 5 June 2025 community meeting.
• What about Batch endpoints? included in plan.
• Keep endpoint names? Currently /v1/openai/v1/{chat/completions,completions,embeddings} -- CHANGE THEM to /v1/...

Internal uses

chat_completion

All internal uses.

Notable

• inline meta-reference agent
• inline llama-guard safety
• inline llm-as-judge scoring
• inline rag tool
• inline meta-reference inference

embeddings

All internal uses.

Notable

• VectorDBWithIndex - migrate to openai_embeddings