Model cache is not cleared when model list in run.yaml is changed

[bparees]

System Info

Using llama-stack 0.2.17, I don't think any other deps/levels come into play in what i've observed.

Information

• The official example scripts
• My own modified scripts

🐛 Describe the bug

I added this to my run.yaml:

models:
  - model_id: my_llm
    provider_id: openai
    model_type: llm
    provider_model_id: gpt-4o-mini

I started llamastack and queries the models endpoint, I see the model in the list:

{
  "data": [
    {
      "identifier": "my_llm",
      "provider_resource_id": "gpt-4o-mini",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    {
      "identifier": "openai/gpt-4-turbo",
      "provider_resource_id": "gpt-4-turbo",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    .............

I then delete the model from the run.yaml and restart llamastack, the model still appears in the list:

{
  "data": [
    {
      "identifier": "my_llm",
      "provider_resource_id": "gpt-4o-mini",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    {
      "identifier": "openai/gpt-4-turbo",
      "provider_resource_id": "gpt-4-turbo",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
    ............

I then removed the .llama directory and restarted llamastack, the model no longer appears in the list.

{
  "data": [
    {
      "identifier": "openai/gpt-3.5-turbo-0125",
      "provider_resource_id": "gpt-3.5-turbo-0125",
      "provider_id": "openai",
      "type": "model",
      "metadata": {},
      "model_type": "llm"
    },
   ............

Error logs

no error in the logs, it's not a failure per se.

Expected behavior

If i update the run.yaml to remove a model, i would not expect to see that model listed in the models endpoint.

Some discussion on the community call lead to a proposal to compute a hash of the run.yaml and clear the DBs on startup if the run.yaml hash doesn't match the hash stored in the DB. Possibly that could be even further refined to compute a hash only based on the parts of run.yaml that contribute to the available models list (e.g. the providers section and the models section)

[Ygnas]

/assign @Ygnas

[mattf]

Issue Summary

Models defined in run.yaml persist in the database even after being removed from the configuration file. Auto-discovered provider models also persist when no longer available from the provider.

Understanding the Problem

Llama Stack has registered models for inference. Each registered model maps:

identifier → (provider_id, provider_model_id)

Example: "my-llm" → ("openai", "gpt-4-turbo")

Three Sources of Registered Models

1. User Registration (API)

POST /v1/models  # Creates custom aliases

→ Should persist (user data)

2. Config Registration (run.yaml)

models:
  - model_id: my_llm  # Admin-defined baseline models

→ Should sync with config (infrastructure data)

3. Auto-Discovery (Provider)

openai/gpt-3.5-turbo-0125  # Mix of static (models.py) + dynamic (endpoints)

→ Dynamic lifecycle (provider-controlled)

Auto-discovery combines:

• Static models from provider's models.py
• Dynamic results from provider's model listing endpoints
• Both registered as {provider_id}/{provider_model_id}

The Problem

All three sources create database entries and no cleanup is performed. When models are removed from run.yaml, they persist because the system can't tell they came from configuration rather than user registration. When provider models are no longer available, they also persist in the database.

Suggested Implementation

For Auto-Discovery Models (Provider-Controlled):

• Cleanup all models with source listed_from_provider during stack shutdown
• Cleanup all models with source listed_from_provider during stack startup
• Fresh registration of provider models on each stack session

For Config vs User Registration:

• Add source tracking to distinguish between user-registered and config-registered models
• Enable selective cleanup of config-registered models when removed from run.yaml

This ensures auto-discovered models are always current and config models sync properly with configuration changes.

[cdoern]

@mattf @Ygnas @bparees does this issue include:

1. models listed which the user has not registered
2. out of date model listings (from a previous run)
3. database management, cleanup, etc?

It was discussed in the community meeting last week that the DB issue might warrant a separate work stream here given that we probably need some syncing mechanism for all resource types from the persisted db on disk.

If this is included in this issue lmk, but I figured I'd ask given that I was thinking of creating a new one just for that.

Also #3143 takes care of part of this, I opened it last week.

[bparees]

@cdoern the excellent writeup @mattf just added seems cover 1+2(in addition to considering how to handle another source of models i hadn't considered, the dynamically registered ones), i'm not sure what cleanup/db management you're referring to (beyond removing obsolete model entries).

[cdoern]

yeah after reading again this seems to be pretty comprehensive, thanks for the writeup @mattf !

[iamemilio]

I think this fix is fine in cases where llama stack is your only model registry, but the caching mechanism might need some more thought. It seems prone to data consistency issues in distributed use cases.

Error Case

A user has an external service that serves models and maintains its own registry that gets updated when users add/remove/update models and register them to it. They have a high user volume, and are using three llama stack servers to handle inference requests to their models. As they spin up the llama stack servers, they come online at different times. The last llama stack server takes 0.5 seconds longer to start up. While it was starting up, a new model got added to the registry, which it discovered and cached. Users that get served by server number 3 have a different list of models than the other servers, resulting in a failure of data consistency. It's not necessarily a mission critical failure, but it would certainly be very confusing for the user on the other side and the administrator who has to clean it up.

Follow Up Questions

1. Do we need to use the cache at all in this case? Its complex to manage in a distributed system, when another service is already acting as a source of truth and meeting the engineering requirements to solve that problem. Maybe we add a config option to disable the model cache for a given provider?
2. If llama stack needs a cache, then this is now a distributed data problem. What do we prioritize? Availability, Consistency, or Partition Tolerance? How does the cache get synchronized? That is a lot more complexity, and its not clear to me if its worth it.

[mattf]

A user has an external service that serves models and maintains its own registry that gets updated when users add/remove/update models and register them to it. They have a high user volume, and are using three llama stack servers to handle inference requests to their models. As they spin up the llama stack servers, they come online at different times. The last llama stack server takes 0.5 seconds longer to start up. While it was starting up, a new model got added to the registry, which it discovered and cached. Users that get served by server number 3 have a different list of models than the other servers, resulting in a failure of data consistency. It's not necessarily a mission critical failure, but it would certainly be very confusing for the user on the other side and the administrator who has to clean it up.

there are two mechanisms to reduce the likelihood of this -

• for a user to use a model the model needs to be registered. if the user is doing this manually, the provider's check_model_availability method can directly check w/ the provider's remote service endpoints. note: not all providers have an efficient way of doing this.
• if the user is relying on the stack server to register a list of available models, providers can be configured with should_refresh_models to periodically get a new listing from the provider's remote model listing endpoint.

we could add a feature to refresh the model listing when model lookup fails. it should be off by default.

please break this out to a separate RFE.