feat(brain): FP-discipline triage model + no-silent-substitution safety#16
Open
venkatas wants to merge 2 commits into
Open
feat(brain): FP-discipline triage model + no-silent-substitution safety#16venkatas wants to merge 2 commits into
venkatas wants to merge 2 commits into
Conversation
Keep the working tree clean and prevent accidental commits of: - .friends_*/ — codex/grok/agy review output that may reference client names (leak-guard discipline: never commit) - cloned external tool repos under tools/ (LinkFinder, SecretFinder, XSStrike, graphql-cop, nomore403) and tools/__pycache__/ Local copies are untouched; these are simply no longer tracked. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-16 benchmark (73-judge FP-discipline panel over 7 local models) found openmythos-27b best for triage; the RavenX line invents confirmations. Wire the finding into the repo, plus the model-selection safety the codex+grok review surfaced. - brain.py: rank TRIAGE_MODEL_PRIORITY by false-positive discipline (openmythos first, clean alternates, baron demoted). PROVISIONAL, pending a true-positive re-bench — the fixture had 0 true-positives so false-negatives were not measured. Narrator/scanner lists unchanged (triage-only evidence). - No silent model substitution: a set-but-uninstalled pin (BRAIN_MODEL/TRIAGE_MODEL/ BRAIN_SCANNER_MODEL) warns loudly, records provenance (MODEL_SELECTION_LOG), and raises under BRAIN_REQUIRE_PIN=1. Honors :latest aliases; empty/unreachable inventory no longer bypasses the check; triage fallback can no longer pick excluded/assert-biased tags (xploiter/bugtraceai/aya) and disables triage instead; scanner distinguishes model-not-found from a network outage; standalone brain_scanner honors brain.env (loaded once at import); hunt.py propagates strict-pin failures instead of swallowing them. - docs: README + backlog model recommendations corrected; new docs/benchmarks/2026-07-16-triage-fp-discipline.md (leaderboard, caveats, re-bench validation gate, follow-ups: provenance persistence + agent pin-awareness). - tests: tests/test_brain_model_selection_safety.py (16, incl. the review repros). Full tests/ suite green, 0 regressions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
2026-07-16 benchmark (73-judge FP-discipline panel over 7 local models, on a client header-sqlmap false-positive fixture) found openmythos-27b best for triage; the RavenX line invents confirmations. This wires the finding into the repo, plus the model-selection safety hardening a codex+grok review surfaced.
Triage ranking (provisional)
TRIAGE_MODEL_PRIORITYreordered by false-positive discipline: openmythos first, clean alternates (nemesis/devstral/glm),baron-llmdemoted, uninstalled historical tags dropped.MODEL_PRIORITY) and scanner lists left unchanged (triage-only evidence).No silent model substitution (safety)
A set-but-uninstalled pin (
BRAIN_MODEL/TRIAGE_MODEL/BRAIN_SCANNER_MODEL) no longer silently falls back:MODEL_SELECTION_LOG); raises underBRAIN_REQUIRE_PIN=1:latestaliases; empty/unreachable inventory no longer bypasses the checkxploiter/bugtraceai/aya) — disables triage insteadbrain_scanner.pyhonorsbrain.env;hunt.pypropagates strict-pin failuresDocs
docs/benchmarks/2026-07-16-triage-fp-discipline.md— leaderboard, caveats, re-bench validation gate, follow-upsTest plan
tests/test_brain_model_selection_safety.py(16 tests, incl. the exact review repros)tests/suite green (0 regressions)Follow-ups (not in this PR)
MODEL_SELECTION_LOGinto the session/reportagent.py's ReAct selector pin-aware (strict mode currently covers brain.py + brain_scanner, not the--agentpath)🤖 Generated with Claude Code