Road to v1

[reluctantfuturist]

🚀 Describe the new functionality needed

Overview

The goal for Llama Stack v1 is to enable ISVs and enterprise developers to build AI applications in on-prem and VPC environments. It is not meant to be a comprehensive list of all tasks, but rather a guide to help us stay on track.

Phase 1: Foundation & Infrastructure

Milestone 0.2.10

• MCP server deployment and Oauth integration
• Developer-facing UI for chat completions and tracing
• Update store to use postgres

Phase 2: Production Ready APIs and Containers

Standardize all APIs to OpenAI format where possible
Milestones 0.2.11 through 0.2.22

• Embeddings API #2358
• OpenAI Compatible Vector Stores and Files API #2338
• AWS k8s deployment for Llama Stack #2340
• Implement Embedding, keyword, and hybrid search #2297

Phase 3: API Hardening

Finalize API work in preparation for the first app deployment
Milestone 0.3.0

API stabilization and conformance

• Full streaming support in Responses API #2364 (@ashwinb )
• Downrank Agents API to /v1alpha #3611 ( @cdoern )
• Add support for OpenAI Conversations #3235 (@franciscojavierarceo )
• Deprecate non-OpenAI APIs #2455 (@franciscojavierarceo )
• Add safety extensions to the Responses API #3325 ( @raghotham to post options on Discord )
• test: introduce api conformance test #3257 (@cdoern )
• feat: introduce api leveling proposal #3317 (@cdoern )

Frameworks integration

• feat: Added llama stack-langchain integration example scripts #3211 (@wukaixingxp )
• feat: Add langchain llamastack Integration example notebook #3314 (@slekkala1 )
• feat: add llamastack + CrewAI integration example notebook #3275 (@omaryashraf5 )

Auth and auditing

• Support auth for Vector Store APIs #2761
• Support auth for Files API #2760

Documentation updates

• Update documentation #2453 (@reluctantfuturist )

Testing and CI

• Performance testing #2384 (@ehhuang )
• CI/CD improvements #2352 (@slekkala1 )

Phase 4: Enterprise readiness features

Milestones 0.3.1+

• API separation for independent containers #2359
• Add /health endpoints for each container within the Stack #2372
• Support authentication #2373
• API key management for partners #2376
• Kubernetes Operator #2378
• Standardize provider errors #2379
• Observability: Add Additional Metrics to Llama Stack Telemetry #2596
• Rework CLI commands #2878
• Llama Stack API Conformance Tests and API Stability #3237
• Auditing functionality #2377

Phase 5: Polish and First On-Prem PoC

• Extensibility #2385
• Configuration Management #2386
• Phone-home for usage metrics #2382
• Phone-home for canary datasets #2383
• Deprecate Agents API #3313 (@ashwinb )

💡 Why is this needed? What if we don't build it?

Having a clear plan to get to v1 will help the community prioritize the most important features and improvements.

Other thoughts

P1s and Nice-to-haves

• Llama Stack Playground #1373
• Prometheus and 23ai provider integrations #2371
• Support for per-distro UI components #2380
• Allow updating resource attributes in the Auth API / ABAC structure #2374

No response

[rhuss]

Thanks for the great roadmap @reluctantfuturist!

I have collected some additional non-functional requirements ideas, but I'm unsure whether they fit into V1 or a future version. Many of these components may lie outside the core LLS component, allowing for an independent release cycle.

See my list of non-functional requirements for LLS. Please feel free to comment on those or add additional items that are missing. It's more of a brainstorming collection than a rigorous plan.

For V1, I see those non-functional requirements as of highest priorities, in addition to the multi-user and authN/authZ points already mentioned:

• Performance: Performance testing and meeting some self-defined SLAs around Latency/Throughput/Concurrency for a defined baseline setup
• Extensibility: Being able to reconfigure/reassemble LLS at deploy time (extending the distribution concept). deploy time lies between build time (aka lls build and runtime (reconfiguration via API endpoints)
• Configuration Management:
  • A 'sealing concept' to not allow reconfiguring of a setup via API endpoint (like the available tools)
  • Allowing to disable any dynamic reconfiguration during runtime, keeping the 'source of truth' in the configuration files (this is important for GitOps-like automation for easy reproducability/recovery).
  • Self-contained provider configuration for allowing flexibility for the storage location on the local filesystem.

These might be more something for M4 as listed above.

[cdoern]

Enable integration tests for all APIs (post-training is missing)

post_training was added back in #2132 , I am going to be doing some work on expanding these tests though!