Scan speed: repo-complexity gate + evidence-driven read set (halve runtime on complex repos)

[serenakeyitan]

Refs #33.

Problem

The dominant scan cost is the agent reading+analyzing files (seconds each). The module greps are milliseconds and sweep the whole tree regardless of depth. But scan depth was decided by file COUNT alone — so a 400-file / 200k-LOC repo read all 200k lines under DEEP. The user asked to (a) add a complexity judgment at the very start of the run, (b) read only important files top-down / on-demand for complex repos, (c) aim to halve runtime.

Design (adversarially vetted)

A workflow pressure-tested the naive idea ("read only MUST_SCAN + grep-hits") and proved it unsafe — error-handling (swallowed catch, floating promise), auth (IDOR / guard-dominance dataflow), and performance are read-dependent verticals: their fatal-eligible findings need the agent to READ a handler/service body, not match a grep. Dropping those files would silently drop confirmed fatals. The final design keeps the speed win with guardrails:

1. Complexity gate at 1.1b — scan_depth = max(count_tier, loc_tier). New LOC axis (wc -l over the already-sorted source_list; thresholds 40k / 250k). Deterministic: max() of two closed-range total orders is one pure, monotone function (can only ever read more conservatively). A 400-file / 200k-LOC repo now tiers DEEP → SHALLOW; a 400-file / 20k-LOC repo stays DEEP.
2. Expanded MUST_SCAN — added the read-dependent service surface (services/jobs/workers/tasks/lib/domain/usecases/processors/core) so handler+service bodies are always fully read even when the rest is sampled. This is the guardrail that makes tiering-down safe.
3. Evidence-driven read set (every depth): MUST_SCAN ∪ grep-hits ∪ depth-scaled sample. Every grep-flagged file is promoted into the full-read set — the "follow the evidence / on-demand top-down" read. must + hits are never sampled out.
4. Sample floors — DEEP all / SHALLOW 1-in-5 / MINIMAL 1-in-20 (was zero — a coverage improvement).
5. Transparency — coverage line + ⚠️ Partial coverage warning whenever below DEEP. Cache/determinism intact (scan_depth already gates the key).

Net effect

~50% fewer file-reads on complex repos; normal/small repos unchanged; two pre-existing coverage gaps fixed as a side effect (MINIMAL zero-sample; grep-hits could be sampled out in SHALLOW).

Verification

• Bash logic validated end-to-end on a fixture repo (LOC sum, max() tiering, MUST_SCAN dirs, two-phase read set).
• Complexity tiering verified across 6 (count, LOC) cases (300/15k→DEEP, 400/200k→SHALLOW, 400/20k→DEEP, 3000/100k→SHALLOW, 400/300k→MINIMAL, 8000/50k→MINIMAL).
• render-report.test.mjs 17/17 still pass (no collateral).

🤖 Generated with Claude Code

[serenakeyitan]

Live before/after verification — SillyTavern (358 files / 193k LOC, real complex app)

Ran the actual old-vs-new read-set computation on a real 3rd-party app that tiers DEEP → SHALLOW. This surfaced two real coverage gaps the design/file-math missed (now fixed in ca13d75):

gap found live	fix
src/endpoints/*.js — only 10 of 47 backend routes read (endpoints wasn't in the MUST_SCAN dir regex)	added endpoints/resolvers/graphql/rpc/views/functions → 47/47 read
security files named by function at top of src/ (recover-password.js, users.js, private-request-filter.js [SSRF], request-proxy.js) were sampled out	added a security-noun basename rule → all promoted

After both fixes:

metric	result
Files the agent reads (≈ runtime)	367 → 203 = 45% fewer (still ~halved)
grep-hit files dropped	0
route / read-dependent-dir files dropped	0
backend src/endpoints coverage	47/47
what IS dropped	frontend public/scripts/* UI + tests/ + pure util/parser/vector-provider files

So on a genuinely complex repo the optimization reads ~half the files, and every backend route, every security-named file, and every mechanically-flagged file is still fully read — only boilerplate/UI is sampled. The disclosed residual (a swallow-and-continue in a hit-free non-security util file) is the same class SHALLOW already accepts, now warned via the partial-coverage line.

[serenakeyitan]

Scope addition — read project docs + scan prompt/skill files as attack surface

.md was entirely outside the scan's file universe (not in EXT_RE), which broke two things:

1. The arch summary must state what a project IS (stack, deploy shape) — that lives in README/docs, not code.
2. For a prompt-native project (tdoc, Claude Code skills) the real logic + risk lives in SKILL.md / *.prompt.md / .claude/ — a prompt file can carry a hardcoded key OR an embedded injection, yet was never scanned.

Design (matches the complexity-gate philosophy — read via MUST_SCAN, NOT via EXT_RE):

• MUST_SCAN grep line 4: project-doc basenames (README/ARCHITECTURE/CONTRIBUTING/SECURITY/AGENTS/CLAUDE + docs/ architecture/deploy/design/setup). Tight allow-list — CHANGELOG/generated/blog excluded.
• MUST_SCAN grep line 5: prompt/skill/agent files as attack surface — read AND scanned.
• secrets.md: prompt/config files are a secret-finding surface (a key in SKILL.md == a key in db.ts).
• ai-integration.md: now also applies to prompt-native repos with no SDK call — checks prompt content for embedded injection / guardrail-override / dangerous-sink instructions.
• Kept OUT of EXT_RE on purpose → docs/prompts never enter count/LOC/complexity-tier/sample (a big docs/ must not inflate LOC and tier the scan down, reading less code).
• Doc/prompt read cap: 2000 lines (a generated 10k-line README can't blow up the read).
• Coverage numerator X filtered to EXT_RE so read docs don't corrupt the "X of Y source files" ratio.

Validated on tdoc: README/SECURITY/CONTRIBUTING/docs-DESIGN + SKILL.md (×2) captured; CHANGELOG/generated/blog/LICENSE correctly excluded. render-report.test.mjs 17/17.