fix: clear model cache when run.yaml model list changes

[Ygnas]

What does this PR do?

closes: #3150

Fixes the model cache not clearing when run.yaml model list changes by implementing proper cleanup mechanisms and adds unit tests.

Not sure exactly what is with all those models as listed_from_provider. At the moment they are still there.

Test Plan

Adding 2 models with:

curl -X POST http://127.0.0.1:8321/v1/models \
  -H "Content-Type: application/json" \
  -d '{
    "model_id": "my_llm",
    "provider_model_id": "gpt-3.5-turbo-0125",
    "provider_id": "openai",
    "model_type": "llm",
    "metadata": {}
  }'

And run.yaml

models:
  - metadata: {}
    model_id: testing-model
    provider_id: openai
    model_type: llm
    provider_model_id: gpt-4o-mini

Returned models looks like:

    "data": [
        {
            "identifier": "my_llm",
            "provider_resource_id": "gpt-3.5-turbo-0125",
            "provider_id": "openai",
            "type": "model",
            "metadata": {},
            "model_type": "llm"
        },
        {
            "identifier": "testing-model",
            "provider_resource_id": "gpt-4o-mini",
            "provider_id": "openai",
            "type": "model",
            "metadata": {},
            "model_type": "llm"
        },
...

After server restart and model removed from run.yaml only the manually added one remains:

    "data": [
        {
            "identifier": "my_llm",
            "provider_resource_id": "gpt-3.5-turbo-0125",
            "provider_id": "openai",
            "type": "model",
            "metadata": {},
            "model_type": "llm"
        },
...

[ashwinb]

I don't think we should do "store + cleanup" instead let's just not store things we don't want to be stored persistently. It seems like what we want to say generically is that there are three ways of registering models:

• user provided (via the register API)
• admin provided (via run.yaml)
• provider provided (listed via the provider)

We need to only store the first parts and the rest are dynamically refreshed / loaded on every single startup of the server.

[cdoern]

@Ygnas are you still working on this one?