fix(review): inspect complete file lists for large pull requests

[vincentkoc]

Related: #983

What Problem This Solves

Fixes an issue where large OpenClaw pull requests could hide config or persistent-data files in the omitted middle of the bounded 80-file review context, leaving deterministic merge-risk classification incomplete.

Why This Change Was Made

For truncated openclaw/openclaw pull requests only, ClawSweeper now fetches the complete file list into a compact non-enumerable sidecar. It reuses every bounded-context page, requires the exact count and unique filenames, verifies that base and head SHAs stayed stable across pagination, and fails closed above GitHub's 3,000-file response limit. Deterministic config and data-model classifiers consume that sidecar, while prompts, ledgers, and cache digests keep the existing bounded context.

User Impact

Large OpenClaw pull requests receive complete deterministic surface classification without increasing model prompt size or changing review cache identity. Fetch or count failures retain the existing unknown-truncated fail-closed marker.

Evidence

• Direct TypeScript build passed.
• Focused context, surface-policy, and prompt-context tests: 61/61 passed.
• Fresh uncommitted and committed-branch autoreview: no accepted or actionable findings; final exact-branch confidence 0.91.
• git diff --check: clean.
• GitHub's current List pull requests files contract caps responses at 3,000 files: https://docs.github.com/en/rest/pulls/pulls#list-pull-requests-files. Totals above that limit fail closed before complete pagination.
• The ClawSweeper changelog entry is intentional: this PR changes ClawSweeper itself. The current origin/main release-owned exception applies to OpenClaw's changelog during foreign-target PR work, not this repository's own operational release notes.

Real Behavior Proof

• Claim: a truncated 118-file pull request supplies all files to deterministic classifiers without exposing the hidden payload to model context or cache identity.
• Exercised surface: live GitHub pull-file pagination for fix(agents): improve Code Mode reliability on smaller models openclaw#115869, complete-count validation, deterministic config/data classifier input, prompt ledger, and content digest.
• Observed result: live pagination returned exactly 118 files; the sidecar accepted all 118, the config classifier reported no config change, and no unknown-truncated marker remained. The page cache fetched pages [1,2] exactly once rather than refetching both pages. The semantic false-positive cleanup is independently reviewable in fix(review): avoid false persistent data-model findings #983.
• Failure proof: count mismatch, duplicate filenames, changed base/head snapshots, paginated-fetch exceptions, and totals above GitHub's 3,000-file response limit return no sidecar, preserving unknown-truncated-pull-files.
• Limits: production review output reflects both this complete-file input fix and the separately reviewed classifier semantics fix after those drafts are landed.

Live Pagination And Merge-Order Proof - July 31, 2026

Redacted live pagination transcript for OpenClaw PR 115869:

captured_at=2026-07-31T21:02:46Z
review_head=4a434e7e1fc530785a6700e088bdbb93b04091a6
page_1_count=100
page_2_count=18
page_3_count=0
total=118
first=docs/providers/huggingface.md
last=src/commands/agent-exec.ts

The transcript was produced with authenticated read-only ghx api requests to:

repos/openclaw/openclaw/pulls/115869/files?per_page=100&page=1
repos/openclaw/openclaw/pulls/115869/files?per_page=100&page=2
repos/openclaw/openclaw/pulls/115869/files?per_page=100&page=3

Both exact merge orders were tested from base 16f01b29508632ab3a904087191e39b20968ab68:

• 983 -> 984: clean cherry-picks; 66/66 focused tests passed.
• 984 -> 983: clean cherry-picks; 66/66 focused tests passed.
• Both orders produced identical Git tree 609478921e5b04e2b1844001bfe81ca713f4f099, with 6 files changed, 560 insertions, and 25 deletions.
• git diff --check 16f01b29508632ab3a904087191e39b20968ab68..HEAD passed in both worktrees.
• Both orders produced { "config": { "change": false, "keys": [] }, "data": { "change": false, "surfaces": [] } } over the live 118-file payload.
• The same suite passed count-mismatch, duplicate-filename, snapshot-drift, fetch-exception, and over-3,000-file cases, which preserve the conservative unknown-truncated-pull-files fallback.
• Prompt, ledger, and content-digest tests confirmed the complete sidecar remains outside model context and cache identity.

Commands used direct tsc plus node --test test/context.test.ts test/pr-surface-policy.test.ts test/review-prompt-context.test.ts; no dependency reconciliation occurred in the proof worktrees.

Maintainer Decision - July 31, 2026

Approved: accept the bounded pagination cost. For a truncated OpenClaw pull request at GitHub's 3,000-file ceiling, the complete classifier path performs at most 30 file-list page reads plus one refreshed pull snapshot. Already-read bounded-context pages are reused, non-OpenClaw targets do not enter this path, and totals above 3,000 or any count, uniqueness, fetch, or source-snapshot mismatch fail closed to the existing unknown-truncated marker. This trade is acceptable for deterministic config and stored-data classification.

[clawsweeper]

Codex review: needs changes before merge. Reviewed August 2, 2026, 5:03 PM ET / 21:03 UTC.

ClawSweeper review

What this changes

Adds complete changed-file hydration for truncated OpenClaw pull requests so deterministic configuration and stored-data classifiers can inspect files omitted from the bounded model context.

Merge readiness

⛔ Blocked by patch quality or review findings - 6 items remain

Keep this PR open: the complete-file classification improvement remains useful, but the branch still implements it only in the pre-refactor monolith and cannot reach the current review pipeline. The author is a repository member, so this workflow must not auto-close it; the narrow port to current modules is the actionable next step.

Priority: P2
Reviewed head: 4a434e7e1fc530785a6700e088bdbb93b04091a6

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The real pagination proof is strong, but the unchanged branch cannot integrate with current main's split review pipeline.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The PR body provides a redacted live GitHub pagination transcript, observed 118-file result, failure-mode coverage, and merge-order proof; after porting, that proof must be rerun on the rebased head.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The PR body provides a redacted live GitHub pagination transcript, observed 118-file result, failure-mode coverage, and merge-order proof; after porting, that proof must be rerun on the rebased head.
Evidence reviewed	5 items	Current pull-file hydration path: Current main constructs bounded pull-file context in the dedicated item-context module; the PR's hydration call remains in the old monolithic entrypoint instead of this active path. Current deterministic classifier path: Current main's configuration and stored-data classifiers consume context.pullFiles in src/clawsweeper-change-detection.ts, which the PR does not modify or adapt to its sidecar. PR targets removed integration location: The branch invokes pullFileReviewHydration and attaches completePullFilesSymbol from src/clawsweeper.ts, but current main has decomposed that entrypoint and routes context construction through src/clawsweeper-item-context.ts.
Findings	1 actionable finding	[P1] Port the sidecar to the split review modules
Security	None	None.

How this fits together

ClawSweeper hydrates GitHub pull-request metadata into a bounded review context, then runs deterministic classifiers before Codex reviews the item. Complete file-list data must stay outside prompts and cache identity while still reaching the active configuration and stored-data classifiers.

flowchart LR
  A[GitHub pull request] --> B[Bounded file context]
  B --> C[Complete file sidecar]
  C --> D[Configuration classifier]
  C --> E[Stored-data classifier]
  D --> F[Merge-risk decision]
  E --> F
  B --> G[Codex review prompt]

Loading

Before merge

• Port the sidecar to the split review modules (P1) - Current main constructs pull-file context in src/clawsweeper-item-context.ts and performs deterministic classification in src/clawsweeper-change-detection.ts; this unchanged monolithic call site is no longer on the active path. Rebase and move the hydration, symbol/type support, and classifier consumption to those modules, or the branch cannot deliver its stated behavior after conflict resolution. This repeats the earlier blocker because git diff confirms the reviewed file is unchanged.
• Resolve merge risk (P1) - Merging the branch as written is not safe: GitHub reports a dirty merge and its sidecar is wired into a monolithic file that current main has replaced, so the intended classifier coverage could be dropped while resolving conflicts.
• Complete next step (P2) - A narrow, mechanical rebase-and-port can restore the intended behavior on current main; the current draft branch needs that repair before normal merge review can continue.
• Improve patch quality - Port the complete-file sidecar into src/clawsweeper-item-context.ts and src/clawsweeper-change-detection.ts.
• Improve patch quality - Re-run the live pagination proof and focused regression tests on the rebased head.
• Improve patch quality - Request a fresh ClawSweeper review after updating the PR body with the new-head proof.

Findings

• [P1] Port the sidecar to the split review modules — src/clawsweeper.ts:6724-6769

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Branch integration surface	6 files, +400/-10; 204 production-code lines target the old entrypoint	The complete-file logic must be moved across the three active review modules before the patch can affect current runtime behavior.

Merge-risk options

Maintainer options:

1. Port the sidecar onto current modules (recommended)
   Move hydration into the active item-context flow and let the current classifier module consume the complete sidecar, with regression coverage for fail-closed cases.
2. Pause the stale branch
   Keep the PR draft until its implementation is rebased, because resolving the current conflict without the module port would not deliver the stated review behavior.

Technical review

Best possible solution:

Rebase and transplant the complete, fail-closed pagination sidecar into the current item-context and change-detection modules, then prove both complete classification and bounded prompt/cache behavior on the rebased head.

Do we have a high-confidence way to reproduce the issue?

Yes—current source shows truncated pull files are still bounded in the active item-context path and the active classifiers only read that bounded list. The PR body also supplies a live 118-file pagination transcript, but the proposed implementation must first be moved onto current main's path.

Is this the best way to solve the issue?

No—the complete-sidecar approach is appropriate, but this branch applies it in a removed monolithic integration path. Porting the same fail-closed behavior into the current item-context and change-detection modules is the narrower maintainable solution.

Full review comments:

• [P1] Port the sidecar to the split review modules — src/clawsweeper.ts:6724-6769
  Current main constructs pull-file context in src/clawsweeper-item-context.ts and performs deterministic classification in src/clawsweeper-change-detection.ts; this unchanged monolithic call site is no longer on the active path. Rebase and move the hydration, symbol/type support, and classifier consumption to those modules, or the branch cannot deliver its stated behavior after conflict resolution. This repeats the earlier blocker because git diff confirms the reviewed file is unchanged.
  Confidence: 0.99
  Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is incorrect
Overall confidence: 0.99

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against de31c9959070.

Labels

Label justifications:

• P2: This improves deterministic review coverage for large OpenClaw pull requests but does not report an active user-facing outage.
• merge-risk: 🚨 automation: The branch changes review-context hydration and merge-risk classification, and its obsolete integration point could silently bypass the intended automation behavior after conflict resolution.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦞 diamond lobster and patch quality is 🦪 silver shellfish.
• status: ⏳ waiting on author: ClawSweeper has contributor-facing work open and is waiting for author action. Sufficient (terminal): The PR body provides a redacted live GitHub pagination transcript, observed 118-file result, failure-mode coverage, and merge-order proof; after porting, that proof must be rerun on the rebased head.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides a redacted live GitHub pagination transcript, observed 118-file result, failure-mode coverage, and merge-order proof; after porting, that proof must be rerun on the rebased head.

Evidence

Acceptance criteria:

• [P1] node --test test/context.test.ts test/pr-surface-policy.test.ts test/review-prompt-context.test.ts.
• [P1] pnpm run check.
• [P1] Run the redacted live pull-file pagination proof against an OpenClaw pull request with more than 80 changed files and verify the complete sidecar remains outside prompt and cache output.

What I checked:

• Current pull-file hydration path: Current main constructs bounded pull-file context in the dedicated item-context module; the PR's hydration call remains in the old monolithic entrypoint instead of this active path. (src/clawsweeper-item-context.ts:271, de31c9959070)
• Current deterministic classifier path: Current main's configuration and stored-data classifiers consume context.pullFiles in src/clawsweeper-change-detection.ts, which the PR does not modify or adapt to its sidecar. (src/clawsweeper-change-detection.ts:7, de31c9959070)
• PR targets removed integration location: The branch invokes pullFileReviewHydration and attaches completePullFilesSymbol from src/clawsweeper.ts, but current main has decomposed that entrypoint and routes context construction through src/clawsweeper-item-context.ts. (src/clawsweeper.ts:6724, 4a434e7e1fc5)
• Refactor provenance: The active split was introduced by the focused entrypoint-decomposition commit after this PR's base, explaining the dirty merge state and why the monolithic implementation no longer reaches the runtime wiring. (src/clawsweeper-item-context.ts:1, dccfa528f038)
• Re-review continuity: The branch head is unchanged from the earlier reviewed SHA, so the prior porting blocker remains unresolved rather than being a new concern. (src/clawsweeper.ts:6724, 4a434e7e1fc5)

Likely related people:

• Peter Steinberger: Introduced the current focused-module layout and has the latest main-history ownership of the item-context and classifier surfaces this PR must target. (role: refactor author and recent area contributor; confidence: high; commits: dccfa528f038, 8fd1140e7aa9, 4a39cfcf6a96; files: src/clawsweeper-item-context.ts, src/clawsweeper-change-detection.ts, src/clawsweeper-runtime.ts)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (16 earlier review cycles; latest 8 shown)

• reviewed 2026-08-01T16:18:21.086Z sha 4a434e7 :: found issues before merge. :: [P1] Rebase the sidecar onto the split review modules
• reviewed 2026-08-01T17:32:42.402Z sha 4a434e7 :: found issues before merge. :: [P1] Port the sidecar onto the split review modules
• reviewed 2026-08-01T21:22:55.741Z sha 4a434e7 :: found issues before merge. :: [P1] Port the sidecar onto the split review modules
• reviewed 2026-08-01T22:38:06.738Z sha 4a434e7 :: found issues before merge. :: [P1] Port the sidecar onto the split review modules
• reviewed 2026-08-02T08:55:19.646Z sha 4a434e7 :: found issues before merge. :: [P1] Port hydration onto the split review modules
• reviewed 2026-08-02T12:47:08.651Z sha 4a434e7 :: found issues before merge. :: [P1] Port hydration onto the split review modules
• reviewed 2026-08-02T15:25:57.239Z sha 4a434e7 :: found issues before merge. :: [P1] Port hydration onto the split review modules
• reviewed 2026-08-02T17:35:36.060Z sha 4a434e7 :: needs changes before merge. :: [P1] Port the sidecar to the split review modules

[vincentkoc]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/30666266122
• Updated: 2026-07-31T21:21:54.741Z

[vincentkoc]

@clawsweeper re-review

[vincentkoc]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Review in progress
• Detail: The exact-review queue leased this run; Codex is reviewing the item.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/30667868505
• Updated: 2026-07-31T21:49:06.532Z

[vincentkoc]

Maintainer decision for head 4a434e7: accept the bounded API budget of at most 30 paginated file-list reads plus one refreshed pull snapshot at the 3,000-file ceiling. Keep stable-snapshot validation, page reuse, OpenClaw-only scope, and fail-closed behavior above the ceiling.

[vincentkoc]

@clawsweeper re-review