fix(release): guard against dated Anthropic model IDs rotting out of a published wheel (#1112) - #1125
Conversation
…a published wheel (#1112) 0.9.1 on PyPI pins five now-retired model IDs, so every LLM-backed command 404s on a fresh install with a perfectly good API key. The IDs were fixed on main but never released — the broken thing is the artifact, not the code. What changes here is the part that stops it recurring, plus the version bump that makes the release cuttable: - scripts/check_model_defaults.py: every DEFAULT_*_MODEL must be a date-less alias (aliases get repointed; dated IDs get retired), no live call site may hardcode a dated ID, and — when a key is available — each default must actually resolve via models.retrieve(). - release.yml runs it before the build with MODEL_GUARD_REQUIRE_LIVE=1, so an unconfigured ANTHROPIC_API_KEY secret fails the release instead of silently disabling the strongest check. - Two live call sites were still pinning dated IDs and the guard caught them: cf auth's key validation used claude-3-haiku-20240307, which is *already* retired — a valid key was being reported as invalid. The settings API's Anthropic key verification used claude-haiku-4-5-20251001, same trajectory. - Version 0.9.1 -> 0.9.2. The tag push (and any yank of 0.9.1) is deliberately left to the operator: publishing to PyPI is irreversible.
|
Claude finished @frankbria's task in 4m 35s —— View job ✅ GLM precision review: no defects foundScoped to concrete defects only (logic errors, crashes/data loss, security, contract breaks). Style/coverage handled by the general review. Per the no-defects outcome, no inline comments were posted. Verified against the checked-out tree (not the diff alone):
Not flagged (already documented by the author as Known limitations, not shipped defects): the call-site scan is a regex with a manual two-file exclusion list, so a new pricing table elsewhere would trip it — but that fails loud (release blocked), the safe direction, and the fix is a one-line One non-blocking observation outside this PR's defect scope: Clean. |
ReviewSolid, well-scoped fix. The root-cause framing (broken artifact, not broken code) is right, and the guard is a reasonable structural fix rather than a one-off patch. A few notes: Correctness
In Concretely: if This is exactly the class of bug the PR is trying to eliminate (a check that can silently pass despite real breakage), so it seems worth closing: accumulate the "could not complete" message alongside Not blocking — Minor / non-blocking
What's good
|
…1112) tests/test_root_docs_950.py pins README and CHANGELOG to the pyproject version, which the bump to 0.9.2 broke. The existing Unreleased body is what 0.9.2 ships, so it becomes the 0.9.2 section rather than a new one above it.
|
Claude finished @frankbria's task in 5m 10s —— View job ✅ GLM precision review: no new defects foundScoped to concrete defects only (logic errors, crashes/data loss, security, contract breaks, resource leaks). Style/coverage handled by the general review. Verified against the checked-out tree, not the diff alone.
What I verified independently:
On the two issues already on record (general review, not repeated here per the no-duplicate rule):
Both are non-blocking; no inline comments were posted. |
Review (follow-up)I reviewed this branch earlier (see my comment above) after the first commit. The second commit only touches The CHANGELOG/README changes in the second commit are accurate and consistent with the PR description — the No new issues found. This remains a solid, well-scoped release-hygiene fix with honest scope boundaries (code fix vs. the still-open operator actions: tag, cleanroom verification, yanking 0.9.1). |
#1112) Both from PR review. The live loop returned early on any non-NotFoundError, discarding violations it had already collected. Without MODEL_GUARD_REQUIRE_LIVE that returned [] — i.e. "passed" — despite a model the API had just said does not exist, which is the exact class of silent pass this guard exists to prevent. With REQUIRE_LIVE the build still failed, but the message named a network blip instead of the broken model. Now the accumulated violations survive either way. Also: EXCLUDED_FILES listed codeframe/lib/metrics_tracker.py while LIVE_CALL_SITES never scanned codeframe/lib, so that entry was dead. Scanning lib now, which widens coverage and makes the exclusion mean what it says.
|
Both review findings were real and are fixed in 5654be5. The mid-loop early return — confirmed exactly as described. Without Two tests cover it, and I verified they are not tautological — reverting the fix fails both: The dead Everything else in the review matches my own reading, including that the operator actions (tag, cleanroom verification against the published wheel, yanking 0.9.1) stay open by design. |
|
Claude finished @frankbria's task in 4m 28s —— View job ✅ GLM precision review: no defects foundScoped to concrete defects only — logic errors, crashes/data loss, security, contract breaks, resource leaks. Style/architecture/coverage handled by the general review. Verified against the checked-out tree at
Both prior findings are correctly fixed:
New-defect hunt on the fix commit — nothing found:
One residual non-blocking observation (not a defect — already documented and not re-flagged): the transient-error branch still Clean. The operator actions (tag |
Review (follow-up — verifying the fix commit)I reviewed the first two commits earlier (see my comments above). This adds the third commit ( Mid-loop early return — confirmed fixed. Dead Also re-checked, unchanged since my first pass:
No new issues found. Both previously-flagged points are resolved with real test coverage, not just asserted fixed. This is ready from a code-review standpoint — the remaining checklist items (tag push, cleanroom verification, yanking 0.9.1) are explicitly scoped as operator actions outside this PR, which I agree with. |
|
Claude finished @frankbria's task in 2m 10s —— View job ✅ GLM precision review: no defects found (synchronize — merge from
|
| Area | Check | Result |
|---|---|---|
| Merge scope | git diff 5654be5 50e387d --stat |
✅ only uv.lock + web-ui locks; zero source conflicts |
| Dated-ID scan (the merge-risk) | Repo-wide grep claude-…-20YYYYMMDD under scanned areas (adapters/llm, cli, core, lib, ui) |
✅ only hits are metrics_tracker.py (comments) + streaming_chat.py:585 (pricing) — both in EXCLUDED_FILES; no merged-in code added a dated ID |
| Version bump | pyproject.toml:7 + uv.lock:579 (codeframe-ai) |
✅ both 0.9.2 post-merge |
| SDK pin | uv.lock:181 |
✅ anthropic==0.70.0 still exposes models.retrieve + NotFoundError |
check_defaults_resolve (the 5654be5 fix) |
scripts/check_model_defaults.py:122,124 |
✅ return violations + […] under REQUIRE_LIVE / return violations otherwise — confirmed-404 survives a later transient error; both prior findings still resolved |
| Defaults | base.py:47-51 |
✅ all 5 are dateless aliases |
| Call-site fix | auth_commands.py:160,166 (import inside try, returns (bool, str)) |
✅ intact; covered by except ImportError/broad except |
| Call-site fix | settings_v2.py:55,378 (module-level import) |
✅ intact; no circular import (base.py is stdlib-only) |
| Release wiring | release.yml:52-56 |
✅ guard runs with MODEL_GUARD_REQUIRE_LIVE=1 before uv build |
On prior findings: both were fixed in 5654be5 and remain fixed through this merge — the merge did not regress them (it didn't touch the guard or the call sites). Not re-posted.
No new defects introduced by the merge. Clean. The operator actions (tag v0.9.2, cleanroom verify against the published wheel, yank 0.9.1) remain correctly out of scope for this code PR.
Review (final pass — post-merge-commit)I reviewed this PR across the first three commits already (see comments above); this pass covers the fourth commit (
Nothing new to flag. This remains a well-scoped, correctly-tested fix:
No blocking issues. Approving from a code-review standpoint. |
1 participant
Closes #1112.
What was actually broken
codeframe-ai==0.9.1on PyPI pins five retired Anthropic model IDs, so everyLLM-backed command 404s on a fresh install — with a valid API key. The IDs were
already corrected on
mainand never released. The broken thing is theartifact, not the code, so the fix is a release plus a guard that stops the
same rot recurring.
Changes
scripts/check_model_defaults.py— the release guard. Three rules:DEFAULT_*_MODELis a date-less alias. Anthropic repoints aliases(
claude-haiku-4-5); it retires dated IDs (claude-3-5-haiku-20241022).Pinning a date is the bug.
models.retrieve()— this catches an alias being retired too..github/workflows/release.yml— runs the guard beforeuv build, withMODEL_GUARD_REQUIRE_LIVE=1. That flag matters: without it, an unconfiguredANTHROPIC_API_KEYsecret would silently skip the strongest check, which is thesame failure mode one level up. With it, a missing or broken secret fails the
release.
Two live call sites the guard caught, neither of which I went looking for:
codeframe/cli/auth_commands.py:163validated API keys againstclaude-3-haiku-20240307. That model is already retired — socf authwas reporting valid keys as invalid. This was a live bug, not just a latent one.
codeframe/ui/routers/settings_v2.py:376verified Anthropic keys againstclaude-haiku-4-5-20251001— still resolvable today, same trajectory.Both now use
DEFAULT_GENERATION_MODEL.pyproject.toml— 0.9.1 → 0.9.2.Pricing lookup tables (
metrics_tracker.py,streaming_chat.py) are explicitlyexcluded: they key on the dated names the API reports back, which is the
opposite problem.
Evidence
exit=0, "release guard passed"claude-3-5-haiku-20241022auth_commands.pyreverted tomainexit=1, names file:lineMODEL_GUARD_REQUIRE_LIVE=1with no keyexit=1, names the missing secretmodels.retrieveagainst the real APIclaude-3-5-haiku-20241022andclaude-3-haiku-20240307bothNotFoundError— the guard has teeth on exactly the IDs that broke 0.9.16 new tests. Third-party review (
codex review --base main): no blocking issues.Acceptance criteria — what this PR does and does not close
DEFAULT_*_MODELis notresolvable. Done, both offline and live.
a
v*tag, and I deliberately did not push one — publishing to PyPI isirreversible. Run
git tag v0.9.2 && git push origin v0.9.2when ready.run.sh(no--source) reachescf prd generate. Onlyverifiable against the published package, so it is gated on the tag above.
it — it cannot work for anyone.
The three unchecked items are all operator actions on PyPI, not code.
Known limitations
ANTHROPIC_API_KEYconfigured as a repo secret for thepypirelease job. If it is not, the release now fails loudly rather thanpublishing unverified — that is the intended behaviour, but it does mean the
secret must exist before the next tag.
core/cli/ui/adapters, with atwo-file exclusion list for pricing tables. A new pricing table elsewhere
would trip it; the fix is to add the file to
EXCLUDED_FILES.