Improve observed SQL matching and docs

[mk3008]

Issue

• Fixes Observed SQL Matching の精度改善と Advanced User Guides の見直し #683.

Customer Value

• Observed SQL ranking becomes more stable for logically equivalent boolean shapes.
• Function calls are distinguished by their arguments, so lookups like lower(email) and lower(status) are less likely to collide.
• Large corpus scans continue past broken SQL assets and now tell you how many files were read and skipped, which makes reverse lookup safer in messy repos.
• The advanced guides now describe the actual matcher behavior, so users can interpret the ranking and warnings correctly.

Outcome

• Improved observed SQL token normalization and ranking behavior in @rawsql-ts/sql-grep-core.
• Added failure-tolerant scanning with warning and summary reporting.
• Updated the observed SQL guides and the public ztd-cli README to match the current CLI surface.
• Fixed the docs consistency test that was still asserting the older users/persistence paths.

Acceptance Criteria

• Boolean branch order is normalized for observed SQL comparisons.
• Function-call matching uses argument shape, not just the function name.
• File-read and parse failures are reported as warnings, while scanning continues.
• Summary output exposes read/skipped counts.
• Observed SQL Matching docs reflect the current behavior and CLI paths.

Verification

• pnpm --filter @rawsql-ts/sql-grep-core test -- tests/observedSqlMatch.test.ts
• pnpm --filter @rawsql-ts/sql-grep-core build
• pnpm --filter @rawsql-ts/ztd-cli test -- tests/furtherReading.docs.test.ts
• git diff --check

Repository Evidence

• packages/sql-grep-core/src/observed/match.ts
• packages/sql-grep-core/src/observed/types.ts
• packages/sql-grep-core/tests/observedSqlMatch.test.ts
• docs/guide/observed-sql-matching.md
• docs/guide/observed-sql-investigation.md
• packages/ztd-cli/README.md
• packages/ztd-cli/tests/furtherReading.docs.test.ts

Supplementary Evidence

• Local command output showed the targeted sql-grep-core test suite and the ztd-cli docs consistency test passing after the fixes.

Open Questions

• None.

Merge Blockers

• None known.

Summary by CodeRabbit

• Documentation

  • Clarified ztd query match-observed behavior, including best-effort matching and file read/skip count reporting with warnings

• New Features

  • Match reports now include counts of files read and skipped
  • Enhanced error reporting with warnings for unreadable files during matching

• Bug Fixes

  • Improved matching accuracy and robustness with better boolean predicate normalization and function signature handling

[coderabbitai]

📝 Walkthrough

Walkthrough

Enhanced the observed SQL matcher with improved ranking logic (boolean operation normalization, function argument shape comparison), partial failure tolerance for file read/scan failures, and file processing metrics tracking. Updated documentation and tests to reflect new best-effort behavior and skip/warning reporting.

Changes

Cohort / File(s)	Summary
Documentation & Guides docs/guide/observed-sql-investigation.md, docs/guide/observed-sql-matching.md, packages/ztd-cli/README.md	Updated guide descriptions to clarify matcher as best-effort, refines comparisons (predicate structure/families, boolean normalization, function argument shape), and adds guidance on interpreting files-read/skipped counts and treating warnings as filtered search.
Type Definitions packages/sql-grep-core/src/observed/types.ts	Added optional readFileSync callback parameter to ObservedSqlMatchReportParams and added filesRead and filesSkipped numeric fields to ObservedSqlMatchReport.summary.
Core Matching Logic packages/sql-grep-core/src/observed/match.ts	Refactored signature normalization: extracted normalizeFunctionSignature and centralized normalizeBooleanSignature with operand flattening/sorting. Wrapped file reads and query scanning in try/catch to track failures as warnings and continue processing. Added injectable readFileSync parameter, filesRead/filesSkipped counters, and updated formatters to display file metrics.
Test Coverage & Configuration packages/sql-grep-core/tests/observedSqlMatch.test.ts, packages/ztd-cli/tests/furtherReading.docs.test.ts	Added two new test cases verifying boolean branch order stability, function-call argument differentiation, and robustness when candidate files cannot be read. Updated test configuration paths and README alignment expectations.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• feat(ztd-cli): add telemetry seam and observed SQL matcher #682: Extends the observed-SQL matcher implementation with injectable readFileSync, file-read/skipped tracking, and try/catch wrapping — directly overlaps with this PR's fault tolerance enhancements.
• feat(cli): split visible AGENTS init from starter #680: Updates observable-SQL matching behavior, types, and tests — refines the observed-SQL matching features modified in this PR.
• feat: move optional filters to SSSQL and fail fast at runtime #679: Introduces the observed-SQL matching module with predicate/literal normalization — this PR builds on and improves its ranking behavior.

Poem

🐰 Normals booleans dance in ordered rows,
Functions now remember what each argument knows,
When files go missing, brave matchers press on,
Tracking their wins with each file they've scanned—
Resilience in searching, one query at a time! 📊✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 20.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Improve observed SQL matching and docs' clearly summarizes the main changes: improving the observed SQL matcher's behavior, robustness, and documentation updates.
Linked Issues check	✅ Passed	The PR comprehensively addresses all coding objectives from issue #683: normalized boolean operand order, improved function-call matching by argument shape, failure-tolerant scanning with file read/skip tracking, and updated documentation reflecting current matcher behavior.
Out of Scope Changes check	✅ Passed	All changes directly support the PR objectives and linked issue requirements; no extraneous modifications were introduced.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch codex/683-observed-sql-matching

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (1)

packages/sql-grep-core/src/observed/match.ts (1)

136-162: Fault-tolerant scanning with appropriate warnings.

The try-catch blocks around file read and splitQueries ensure scanning continues despite individual file failures. Warning codes (file-read-failed, file-scan-failed) distinguish failure modes clearly.

One observation: filesSkipped increments for both read failures (line 142) and scan failures (line 155). A file that reads successfully but fails to parse will have both filesRead incremented (line 140) and filesSkipped incremented (line 155). This is semantically reasonable since "skipped" means "excluded from candidate pool," but users might initially expect filesRead + filesSkipped == filesScanned.

Consider adding a brief clarification in the docs or a code comment if confusion arises in practice.

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@packages/sql-grep-core/src/observed/match.ts` around lines 136 - 162, The
current loop increments filesRead when readFile succeeds and filesSkipped when
splitQueries fails, meaning a single file can be counted in both filesRead and
filesSkipped; add a brief inline comment near the loop or the counter
declarations (referencing candidateFiles, filesRead, filesSkipped, splitQueries,
and warnings) clarifying that filesRead counts files successfully read while
filesSkipped counts files excluded from the candidate pool and that these sets
may overlap (so filesRead + filesSkipped may exceed total scanned files), or
alternatively adjust counters if you prefer invariant filesRead + filesSkipped
== totalScanned by either decrementing filesRead on parse failure or introducing
a separate filesScanned/parsed counter—pick one approach and document it inline
near the existing try/catch blocks for readFile and splitQueries.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@packages/sql-grep-core/src/observed/match.ts`:
- Around line 136-162: The current loop increments filesRead when readFile
succeeds and filesSkipped when splitQueries fails, meaning a single file can be
counted in both filesRead and filesSkipped; add a brief inline comment near the
loop or the counter declarations (referencing candidateFiles, filesRead,
filesSkipped, splitQueries, and warnings) clarifying that filesRead counts files
successfully read while filesSkipped counts files excluded from the candidate
pool and that these sets may overlap (so filesRead + filesSkipped may exceed
total scanned files), or alternatively adjust counters if you prefer invariant
filesRead + filesSkipped == totalScanned by either decrementing filesRead on
parse failure or introducing a separate filesScanned/parsed counter—pick one
approach and document it inline near the existing try/catch blocks for readFile
and splitQueries.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4acbac20-ad4f-4171-96b7-52e572bb5758

📥 Commits

Reviewing files that changed from the base of the PR and between b94aeba and 10c02ea.

📒 Files selected for processing (7)

• docs/guide/observed-sql-investigation.md
• docs/guide/observed-sql-matching.md
• packages/sql-grep-core/src/observed/match.ts
• packages/sql-grep-core/src/observed/types.ts
• packages/sql-grep-core/tests/observedSqlMatch.test.ts
• packages/ztd-cli/README.md
• packages/ztd-cli/tests/furtherReading.docs.test.ts