What's wrong
pkg/config/auto.go defines DefaultModels, a map[string]string of provider → default model, used for auto-selecting a model when a user has credentials for a provider but hasn't picked a model explicitly (AutoModelConfig, pkg/creator/agent.go, etc.).
Nothing in the test suite ever validates that these default model strings actually exist in the models.dev catalog:
pkg/config/auto_test.go (TestAutoModelConfig_*) only asserts literal equality against the map's own values (i.e. it re-asserts auto.go's current contents back at itself) and that AutoModelConfig returns whatever is in the map — it never calls into modelsdev.Store.GetModel.- The only place that does validate model references against the real catalog is
pkg/config/examples_test.go (TestParseExamples), which only covers examples/*.yaml files — DefaultModels isn't exercised there at all.
Why this matters
DefaultModels can silently go stale (a provider's models.dev catalog moves on, the referenced model is deprecated/removed) with zero CI signal. In practice, when auditing model references after the models.dev snapshot refresh in #4121, DefaultModels["google"] (gemini-3.5-flash) was already one generation behind the latest available flash-tier model in the very snapshot shipped in the same PR — an easy thing to miss without one authoritative check.
Unlike examples/*.yaml, whose failure mode is "an example in the docs breaks," a stale DefaultModels entry means real end users hitting AutoModelConfig in production get a broken auto-selected model with no warning — arguably a worse failure mode than an example test failing.
Suggested fix direction
Add a test (in pkg/config or pkg/modelsdev) that iterates DefaultModels and calls modelsdev.Store.GetModel for each provider/model pair, reusing the same modelsDevAbsentProviders-style skip-list examples_test.go already has for providers not present (or present under a different id) in the models.dev catalog.
References
What's wrong
pkg/config/auto.godefinesDefaultModels, amap[string]stringof provider → default model, used for auto-selecting a model when a user has credentials for a provider but hasn't picked a model explicitly (AutoModelConfig,pkg/creator/agent.go, etc.).Nothing in the test suite ever validates that these default model strings actually exist in the models.dev catalog:
pkg/config/auto_test.go(TestAutoModelConfig_*) only asserts literal equality against the map's own values (i.e. it re-assertsauto.go's current contents back at itself) and thatAutoModelConfigreturns whatever is in the map — it never calls intomodelsdev.Store.GetModel.pkg/config/examples_test.go(TestParseExamples), which only coversexamples/*.yamlfiles —DefaultModelsisn't exercised there at all.Why this matters
DefaultModelscan silently go stale (a provider's models.dev catalog moves on, the referenced model is deprecated/removed) with zero CI signal. In practice, when auditing model references after the models.dev snapshot refresh in #4121,DefaultModels["google"](gemini-3.5-flash) was already one generation behind the latest available flash-tier model in the very snapshot shipped in the same PR — an easy thing to miss without one authoritative check.Unlike
examples/*.yaml, whose failure mode is "an example in the docs breaks," a staleDefaultModelsentry means real end users hittingAutoModelConfigin production get a broken auto-selected model with no warning — arguably a worse failure mode than an example test failing.Suggested fix direction
Add a test (in
pkg/configorpkg/modelsdev) that iteratesDefaultModelsand callsmodelsdev.Store.GetModelfor each provider/model pair, reusing the samemodelsDevAbsentProviders-style skip-listexamples_test.goalready has for providers not present (or present under a different id) in the models.dev catalog.References