test(ai): pin the aiTools registry against TOOL_TIERS, both directions (#3300) - #3306
Conversation
LanternOps#3300) `TOOL_TIERS` gates AI-chat visibility twice: `BREEZE_MCP_TOOL_NAMES = Object.keys(TOOL_TIERS)` is the `allowedTools` list handed to the SDK, and `createSessionPreToolUse` rejects `!TOOL_TIERS[toolName]` as "Unknown tool". Nothing asserted that the two maps agree, and they have drifted in both directions: - 86 registered tools have no `TOOL_TIERS` entry, so a working, fully-tested tool is one the chat agent says does not exist. Whole feature families are affected — backup, MSSQL, Hyper-V, DR, incident response, PAM elevation. - 4 `TOOL_TIERS` keys have no registered tool, so those names are advertised to the SDK as callable and then fail at execution rather than simply not being offered: get_backup_health, get_recovery_readiness, propose_action_plan, run_backup_verification. This is the LanternOps#2605 drift class, and it reached 86 precisely because nothing failed CI when it happened. This adds only the guard, not the fix. Assigning a tier is an approval-gate decision that wants per-tool review — a wrong tier on `execute_containment` or `revoke_elevation` is worse than the tool being invisible — so the 86 and the 4 go into frozen allowlists instead. Both lists may only shrink, and a test in each block fails on a stale entry, so removing a gap is what un-blocks CI and re-adding one cannot pass. Measured on f400fc3: 215 registered, 133 tiered, 86 missing, 4 dead. The suite also asserts that every shared name carries the SAME tier in both maps. That held for all 133 with no exceptions, so it starts with no allowlist. Presence alone would be the weaker contract: an entry that is present but carries a lower tier than the registry assigned it has had its approval gate silently downgraded, which is strictly worse than being invisible. Controls (each assertion proven load-bearing, not vacuous): - drop a real gap from KNOWN_MISSING -> registry-subset test fails - add a tiered tool to KNOWN_MISSING -> shrink-only test fails - drop a dead entry from KNOWN_UNREGISTERED-> tiers-subset test fails - add a live tool to KNOWN_UNREGISTERED -> shrink-only test fails - set execute_command 3 -> 1 in TOOL_TIERS -> agreement test fails with "execute_command: TOOL_TIERS=1 registry=3" Test-only change; no source, dependency, or behaviour changes. Local gate: apps/api vitest 1298 files / 20766 tests pass, tsc 0 errors, eslint clean.
|
Rebased onto This PR freezes exact global counts (215 registered / 133 tiered / 86 missing / 4 dead), so a merge that adds, removes, or re-tiers a tool would break it after it lands rather than on the PR. #3308 touched Re-ran the suite against the rebased tree: 5/5 pass, both allowlists still exact. |
539d0c5 to
079f5c8
Compare
|
The It's the known InvoiceWorkspace flake (#3219 / #3277): |
Part of #3300 — the guard only, not the tier assignments.
Per the issue's own sequencing ("Step 2 is arguably worth doing first"), this adds the contract test and leaves all 86 tier decisions alone. A wrong tier on
execute_containmentorrevoke_elevationis worse than the tool being invisible, so those want the deliberate per-tool review the issue describes.What drifted
TOOL_TIERSgates chat visibility twice —BREEZE_MCP_TOOL_NAMES = Object.keys(TOOL_TIERS)is theallowedToolslist handed to the SDK, andcreateSessionPreToolUserejects!TOOL_TIERS[toolName]as "Unknown tool". Nothing asserted the two maps agree, and they drifted both ways:The second direction is not in the issue. The four are
get_backup_health,get_recovery_readiness,propose_action_plan,run_backup_verification. Same #2605 drift class, so the guard is bidirectional.Measured on
f400fc315: 215 registered, 133 tiered, 86 missing, 4 dead. (215/133 rather than the issue's 217/135 because #3291 merged in between; the missing count is 86 either way.)What this adds
One test file. Three contracts:
KNOWN_MISSING_TOOL_TIERS, the 86 frozen.KNOWN_UNREGISTERED_TOOL_TIERS, the 4 frozen.Contract 3 is the one worth keeping past the cleanup. Presence alone is the weaker claim: an entry that is present but lower than the registry's tier has had its approval gate quietly downgraded, which is worse than invisibility. It also means the eventual step-1 work has a defensible default — the registry's own tier — rather than a blank sheet.
Both allowlists may only shrink. Each block has a second test that fails on a stale entry, so deleting a name is what un-blocks CI once a tier lands, and re-adding one cannot pass. The lists walk to empty on their own.
Controls
Every assertion was proven load-bearing by breaking it:
KNOWN_MISSINGKNOWN_MISSINGKNOWN_UNREGISTEREDKNOWN_UNREGISTEREDexecute_command: 3→1inTOOL_TIERSexecute_command: TOOL_TIERS=1 registry=3The last row is the one that matters — a silent approval-gate downgrade is caught.
Verification
Test-only. No source, dependency, or behaviour changes, so the security and dependency jobs are unaffected.
vitest run(apps/api) — 1298 files, 20766 tests passed, 5 files / 40 tests skippedtsc --noEmit --project apps/api/tsconfig.json— 0 errorseslinton the new file — clean