fix(release): guard against dated Anthropic model IDs rotting out of a published wheel (#1112)

.github/workflows/release.yml

@@ -41,6 +41,17 @@ jobs:
             exit 1
           fi
 
+      # 0.9.1 shipped five retired model IDs and 404'd for every new user (#1112).
+      # This fails the release before it can happen again: every DEFAULT_*_MODEL
+      # must be a date-less alias AND must resolve against the live API.
+      # REQUIRE_LIVE means a missing/broken ANTHROPIC_API_KEY secret fails the
+      # release rather than silently skipping the check that matters most.
+      - name: Model-default release guard
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          MODEL_GUARD_REQUIRE_LIVE: "1"
+        run: uv run python -m scripts.check_model_defaults
+
       - name: Build
         run: uv build
 

CHANGELOG.md

@@ -7,7 +7,14 @@ and this project aims to follow [Semantic Versioning](https://semver.org/spec/v2
 
 ## [Unreleased]
 
-Over 200 commits since v0.9.1. `SECURITY.md` supports only the latest release, so the
+## [0.9.2]
+
+**0.9.1 could not work for anyone** and should not be used: it pinned five
+Anthropic model IDs that have since been retired, so every LLM-backed command
+returned a 404 on a fresh install even with a valid API key (#1112). The IDs
+were already corrected in the repository and had simply never been released.
+
+Over 200 further commits since v0.9.1. `SECURITY.md` supports only the latest release, so the
 security section below is the one to read before deploying anything older.
 
 ### Security
@@ -69,6 +76,13 @@ Several of these change defaults and will require configuration on an existing d
 
 ### Added
 
+- **A release guard that fails the build on an unresolvable model default
+  (#1112).** `scripts/check_model_defaults.py` runs before `uv build` and
+  rejects any dated model ID in the defaults or at a live call site, and — with
+  `MODEL_GUARD_REQUIRE_LIVE=1`, as the release job sets — verifies each default
+  actually resolves against the API. A missing `ANTHROPIC_API_KEY` secret fails
+  the release rather than silently skipping the check.
+
 - **Phase 5.5 — GitHub Issues import.** Connect a repo with a PAT from Settings →
   Integrations (#563), browse its open issues with search, label filter and pagination
   (#564), and import selected issues as tasks with `github_issue_number`/`external_url`
@@ -99,6 +113,14 @@ Several of these change defaults and will require configuration on an existing d
 
 ### Fixed
 
+- **Retired Anthropic model IDs in the published package (#1112).** The
+  `DEFAULT_*_MODEL` constants use undated aliases (`claude-sonnet-4-5`,
+  `claude-haiku-4-5`), which Anthropic repoints, rather than dated IDs, which it
+  retires.
+- **`cf auth` reported valid API keys as invalid (#1112).** Key validation
+  probed `claude-3-haiku-20240307`, itself already retired. It now uses the
+  shared alias, as does the settings API's Anthropic key verification.
+
 - Token/cost data was silently dropped: `react_agent` int-cast UUID task ids and stored
   NULL in `token_usage` (#712, #558).
 - A bad GitHub PAT returned 401, which the web UI treated as session expiry and logged the
@@ -162,6 +184,7 @@ name claim is being pursued in parallel. The CLI entry point remains `cf`.
 - Version bumped from a placeholder `0.1.0` to an honest beta `0.9.0`; development status classifier moved to `4 - Beta`.
 - README installation section now leads with `uv tool install` instead of git-clone; status badge updated to **beta** with a stability statement.
 
-[Unreleased]: https://github.com/frankbria/codeframe/compare/v0.9.1...HEAD
+[Unreleased]: https://github.com/frankbria/codeframe/compare/v0.9.2...HEAD
+[0.9.2]: https://github.com/frankbria/codeframe/compare/v0.9.1...v0.9.2
 [0.9.1]: https://github.com/frankbria/codeframe/compare/v0.9.0...v0.9.1
 [0.9.0]: https://github.com/frankbria/codeframe/releases/tag/v0.9.0

README.md

@@ -74,7 +74,7 @@ THE CLOSED LOOP
 ---
 
 > [!NOTE]
-> **CodeFRAME is in public beta (`0.9.1`).** The vision and the Golden Path CLI
+> **CodeFRAME is in public beta (`0.9.2`).** The vision and the Golden Path CLI
 > (`cf init/prd/tasks/work/proof/pr`) and v2 API are stable enough to build on;
 > the web UI and anything marked "in progress" in
 > [`docs/PRODUCT_ROADMAP.md`](docs/PRODUCT_ROADMAP.md) are still moving, and

codeframe/cli/auth_commands.py

@@ -157,10 +157,13 @@ def validate_anthropic_credential(api_key: str) -> Tuple[bool, str]:
         from anthropic import Anthropic, AuthenticationError as AnthropicAuthError
         from anthropic import APIConnectionError, RateLimitError, APIStatusError
 
+        from codeframe.adapters.llm.base import DEFAULT_GENERATION_MODEL
+
         client = Anthropic(api_key=api_key)
-        # Make a minimal request to validate
+        # Make a minimal request to validate. The model must be an undated alias:
+        # a dated ID gets retired and then a *valid* key reports as invalid (#1112).
         client.messages.create(
-            model="claude-3-haiku-20240307",
+            model=DEFAULT_GENERATION_MODEL,
             max_tokens=1,
             messages=[{"role": "user", "content": "hi"}],
         )

codeframe/ui/routers/settings_v2.py

@@ -52,6 +52,7 @@
     save_notifications_config,
     vet_webhook_host,
 )
+from codeframe.adapters.llm.base import DEFAULT_GENERATION_MODEL
 from codeframe.core.workspace import Workspace
 from codeframe.notifications.webhook import (
     WebhookNotificationService,
@@ -368,12 +369,13 @@ def _verify_anthropic_sync(key: str) -> tuple[bool, str]:
     stable, always-present API surface across all supported SDK versions
     (>=0.18). max_tokens=1 keeps the cost trivial. Non-auth exceptions
     are logged but not echoed to the client to avoid leaking provider
-    internals.
+    internals. The model is the undated alias: a dated ID gets retired and
+    then a *valid* key starts reporting as rejected (#1112).
     """
     try:
         client = _AnthropicClient(api_key=key)
         client.messages.create(
-            model="claude-haiku-4-5-20251001",
+            model=DEFAULT_GENERATION_MODEL,
             max_tokens=1,
             messages=[{"role": "user", "content": "ping"}],
         )

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "codeframe-ai"
-version = "0.9.1"
+version = "0.9.2"
 description = "A project delivery system that orchestrates frontier coding agents: Think, Build, Prove, Ship."
 readme = "README.md"
 requires-python = ">=3.11"

scripts/check_model_defaults.py

@@ -0,0 +1,143 @@
+#!/usr/bin/env python3
+"""Release guard: no dated Anthropic model ID may ship in a published artifact (#1112).
+
+`codeframe-ai==0.9.1` pinned `claude-3-5-haiku-20241022` and four siblings. They
+were valid the day they were written and 404 today, so every LLM-backed command
+in the published wheel failed for every new user with a perfectly good API key.
+
+Dated IDs get retired. The undated aliases (`claude-haiku-4-5`) do not — Anthropic
+repoints them. So the rule is: **defaults and live call sites use aliases.**
+
+Two checks:
+
+  offline (always) — every `DEFAULT_*_MODEL` is a date-less alias, and no live
+                     call site hardcodes a dated ID
+  live (only when ANTHROPIC_API_KEY is set) — `models.retrieve()` each default,
+                     so a retired *alias* would also be caught
+
+The live half is opt-in on purpose: the release build must not depend on a
+secret or on the network being up. The offline rule is the one that always runs,
+and it is the one that would have caught 0.9.1.
+
+Run: `python -m scripts.check_model_defaults`  (exit 1 on any violation)
+"""
+
+from __future__ import annotations
+
+import os
+import re
+import sys
+from pathlib import Path
+
+# `-YYYYMMDD` at the end of a claude-* model id.
+DATED_MODEL_RE = re.compile(r"claude-[a-z0-9.-]*?-(20\d{6})\b")
+
+# Modules that issue real API calls with a hardcoded model. A dated ID here rots
+# exactly the way the defaults did. Pricing tables and tests are excluded: they
+# *look up* dated names the API returns, which is the opposite problem.
+LIVE_CALL_SITES = (
+    "codeframe/adapters/llm",
+    "codeframe/cli",
+    "codeframe/core",
+    "codeframe/lib",
+    "codeframe/ui",
+)
+EXCLUDED_FILES = (
+    # Pricing lookups keyed by the dated names the API reports back.
+    "codeframe/lib/metrics_tracker.py",
+    "codeframe/core/adapters/streaming_chat.py",
+)
+
+
+def _defaults() -> dict[str, str]:
+    from codeframe.adapters.llm import base
+
+    return {
+        name: getattr(base, name)
+        for name in dir(base)
+        if name.startswith("DEFAULT_") and name.endswith("_MODEL")
+    }
+
+
+def check_defaults_are_aliases() -> list[str]:
+    """Return a violation message per `DEFAULT_*_MODEL` carrying a date suffix."""
+    return [
+        f"{name} = {value!r} is a dated model ID; use the undated alias so it cannot be retired"
+        for name, value in _defaults().items()
+        if DATED_MODEL_RE.search(value)
+    ]
+
+
+def check_call_sites(repo_root: Path) -> list[str]:
+    """Return a violation message per source line hardcoding a dated model ID."""
+    violations = []
+    for area in LIVE_CALL_SITES:
+        for path in sorted((repo_root / area).rglob("*.py")):
+            rel = path.relative_to(repo_root).as_posix()
+            if rel in EXCLUDED_FILES:
+                continue
+            for lineno, line in enumerate(path.read_text().splitlines(), 1):
+                match = DATED_MODEL_RE.search(line)
+                if match:
+                    violations.append(
+                        f"{rel}:{lineno} hardcodes dated model ID {match.group(0)!r}; "
+                        f"use a DEFAULT_*_MODEL alias from codeframe.adapters.llm.base"
+                    )
+    return violations
+
+
+def check_defaults_resolve() -> list[str]:
+    """Ask the API whether each default actually exists.
+
+    No-op without a key, so local and CI-gate runs stay offline. The release job
+    sets ``MODEL_GUARD_REQUIRE_LIVE=1``: there, a missing key is itself a failure,
+    otherwise an unconfigured secret would silently disable the strongest check
+    and we would be back to shipping unverified model IDs.
+    """
+    if not os.getenv("ANTHROPIC_API_KEY"):
+        if os.getenv("MODEL_GUARD_REQUIRE_LIVE"):
+            return [
+                "MODEL_GUARD_REQUIRE_LIVE is set but ANTHROPIC_API_KEY is not — "
+                "the live model-resolution check cannot run. Configure the "
+                "ANTHROPIC_API_KEY secret for this job."
+            ]
+        return []
+    import anthropic
+
+    client = anthropic.Anthropic()
+    violations = []
+    for name, value in sorted(_defaults().items()):
+        try:
+            client.models.retrieve(value)
+        except anthropic.NotFoundError:
+            violations.append(f"{name} = {value!r} does not resolve — the API does not know this model")
+        except Exception as exc:  # network, auth, rate limit — not the model's fault
+            # Keep whatever violations this loop already found. Replacing them
+            # with the transient-failure message would throw away the name of
+            # the model that is actually broken — and without REQUIRE_LIVE it
+            # would return "passed" despite a confirmed 404, which is exactly
+            # the class of silent pass this guard exists to prevent.
+            detail = f"{name} = {value!r}: {type(exc).__name__}: {exc}"
+            if os.getenv("MODEL_GUARD_REQUIRE_LIVE"):
+                return violations + [f"live model check could not complete — {detail}"]
+            print(f"  ! skipped live check for {detail}", file=sys.stderr)
+            return violations
+    return violations
+
+
+def main() -> int:
+    repo_root = Path(__file__).resolve().parent.parent
+    violations = (
+        check_defaults_are_aliases() + check_call_sites(repo_root) + check_defaults_resolve()
+    )
+    if violations:
+        print("Model-default release guard FAILED (#1112):", file=sys.stderr)
+        for v in violations:
+            print(f"  - {v}", file=sys.stderr)
+        return 1
+    print("Model-default release guard passed.")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

tests/adapters/test_model_defaults_guard_1112.py

@@ -0,0 +1,123 @@
+"""#1112 — the release guard that stops a dated model ID rotting out of a published wheel.
+
+0.9.1 shipped `claude-3-5-haiku-20241022` and friends. Those IDs were valid the
+day they were written and 404 today, so every LLM command in the published
+artifact was dead on arrival. Dated IDs get retired; the undated aliases do not.
+This pins that rule so it cannot recur silently.
+"""
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import httpx
+import pytest
+
+from scripts.check_model_defaults import (
+    DATED_MODEL_RE,
+    check_call_sites,
+    check_defaults_are_aliases,
+    check_defaults_resolve,
+)
+
+pytestmark = pytest.mark.v2
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+
+
+def test_dated_ids_are_recognised_and_aliases_are_not():
+    assert DATED_MODEL_RE.search("claude-3-5-haiku-20241022")
+    assert DATED_MODEL_RE.search("claude-haiku-4-5-20251001")
+    assert not DATED_MODEL_RE.search("claude-haiku-4-5")
+    assert not DATED_MODEL_RE.search("claude-sonnet-4-5")
+
+
+def test_every_default_model_is_a_dateless_alias():
+    assert check_defaults_are_aliases() == []
+
+
+def test_no_live_call_site_hardcodes_a_dated_model_id():
+    assert check_call_sites(REPO_ROOT) == []
+
+
+def test_require_live_without_a_key_is_a_failure(monkeypatch):
+    """An unconfigured release secret must fail loudly, not skip the live check."""
+    monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
+    monkeypatch.setenv("MODEL_GUARD_REQUIRE_LIVE", "1")
+    violations = check_defaults_resolve()
+    assert len(violations) == 1
+    assert "ANTHROPIC_API_KEY" in violations[0]
+
+
+def test_no_key_and_no_require_live_skips_quietly(monkeypatch):
+    monkeypatch.delenv("ANTHROPIC_API_KEY", raising=False)
+    monkeypatch.delenv("MODEL_GUARD_REQUIRE_LIVE", raising=False)
+    assert check_defaults_resolve() == []
+
+
+class TestAPartialLiveFailureKeepsWhatItFound:
+    """A transient error on one default must not discard a confirmed 404 on another.
+
+    Raised in review: the loop returned early on any non-NotFoundError, throwing
+    away violations already collected. Without REQUIRE_LIVE that returned `[]` —
+    "passed" — despite a model the API had just said does not exist.
+    """
+
+    @staticmethod
+    def _client_where_second_call_blows_up():
+        import anthropic
+
+        calls = {"n": 0}
+
+        def retrieve(model_id):
+            calls["n"] += 1
+            if calls["n"] == 1:
+                raise anthropic.NotFoundError(
+                    "not found", response=httpx.Response(404, request=httpx.Request("GET", "http://x")), body=None
+                )
+            raise ConnectionError("transient network blip")
+
+        client = MagicMock()
+        client.models.retrieve.side_effect = retrieve
+        return client
+
+    def _run(self, monkeypatch, require_live: bool) -> list[str]:
+        import anthropic
+
+        monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-test")
+        if require_live:
+            monkeypatch.setenv("MODEL_GUARD_REQUIRE_LIVE", "1")
+        else:
+            monkeypatch.delenv("MODEL_GUARD_REQUIRE_LIVE", raising=False)
+        monkeypatch.setattr(
+            anthropic, "Anthropic", lambda *a, **k: self._client_where_second_call_blows_up()
+        )
+        return check_defaults_resolve()
+
+    def test_the_confirmed_404_survives_under_require_live(self, monkeypatch):
+        violations = self._run(monkeypatch, require_live=True)
+        assert any("does not resolve" in v for v in violations), violations
+        assert any("could not complete" in v for v in violations), violations
+
+    def test_the_confirmed_404_is_not_silently_dropped_without_require_live(self, monkeypatch):
+        violations = self._run(monkeypatch, require_live=False)
+        assert any("does not resolve" in v for v in violations), (
+            "a confirmed 404 must not be swallowed by a later transient error"
+        )
+
+
+def test_script_exits_zero_offline():
+    """The release workflow runs this as a build step, so its exit code matters."""
+    # Drop the key so this stays offline even on a machine (or CI runner) that
+    # has one — the live half is the release job's business, not this test's.
+    env = {k: v for k, v in os.environ.items() if k != "ANTHROPIC_API_KEY"}
+    result = subprocess.run(
+        [sys.executable, "-m", "scripts.check_model_defaults"],
+        cwd=REPO_ROOT,
+        capture_output=True,
+        text=True,
+        env=env,
+    )
+    assert result.returncode == 0, result.stdout + result.stderr