fix: retract markdown docs that become oversized

[compoodment]

Summary

• Retracts previously ingested markdown documents when a later scan finds the source file has grown beyond markdownIngestionMaxTokensPerFile.
• Handles both the pre-read estimate skip path and the streamed too_large path.
• Adds an integration regression that ingests a small markdown file, grows it oversized, and verifies delete_authored_document removes stale authored content.

Fixes #315.

Verification

• pnpm exec tsc --noEmit — PASS
• pnpm run clean:test && pnpm exec tsc -p tsconfig.tests.json && node --test .ts-build/test/integration/markdown-ingest.test.js — PASS

Note: this branch is stacked on #318 so the test suite can compile. Full pnpm check and npm run test:integration are still blocked by unrelated upstream unit/integration failures after #318's compile fix.

– Vale

Summary by CodeRabbit

• Bug Fixes

  • Enhanced handling of markdown files exceeding configured token limits. Files that grow beyond the size threshold are now properly removed and retracted from the cache, instead of simply being skipped during future processing.

• Tests

  • Added integration test validating that oversized markdown files are properly retracted from the cache when they exceed the configured token size limits.

[coderabbitai]

Linter diff in the way? Review this PR in Change Stack to focus on meaningful changes and expand context only when needed.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 4266ccce-7347-44d3-86d8-6ca426d5ded6

📥 Commits

Reviewing files that changed from the base of the PR and between 6d47279 and 324e4ce.

📒 Files selected for processing (2)

• src/markdown-ingest.ts
• test/integration/markdown-ingest.test.ts

---

📝 Walkthrough

Walkthrough

When a previously ingested markdown file exceeds markdownIngestionMaxTokensPerFile, the scanner now deletes its cached document instead of deferring silently. A new helper function deleteCachedSourceDocument is integrated into both token-check and streaming-detection paths. An integration test validates the retraction flow.

Changes

Oversized file document retraction

Layer / File(s)	Summary
Deletion helper and oversized-file detection integration src/markdown-ingest.ts	New deleteCachedSourceDocument(sourceDoc) helper that enqueues a delete operation and marks the snapshot dirty when a cached document exists. Integrated into syncMarkdownFile to delete cached documents when streamed content is too_large, and into syncCandidates to delete cached documents when estimated token count exceeds the limit.
Integration test for oversized retraction test/integration/markdown-ingest.test.ts	FsDirentLike type is imported from markdown-ingest.js for reuse. New test case ingests a file under the token cap, rewrites it to exceed the cap, triggers the watcher, and asserts the previously cached authored document is retracted via delete_authored_document RPC call.

Sequence Diagram

sequenceDiagram
  participant Scanner as Markdown Scanner
  participant syncCandidates as syncCandidates
  participant syncMarkdownFile as syncMarkdownFile
  participant deleteCached as deleteCachedSourceDocument
  participant RPCClient as RPC Client
  
  Note over Scanner,RPCClient: File was previously ingested, now exceeds token limit
  Scanner->>syncCandidates: Scan oversized candidate
  syncCandidates->>syncCandidates: Check estimated tokens
  alt Tokens exceed limit
    syncCandidates->>deleteCached: Delete cached document
    deleteCached->>RPCClient: enqueue delete_authored_document
    deleteCached->>syncCandidates: filesDeleted++
  else Proceed to streaming check
    syncCandidates->>syncMarkdownFile: Stream file content
    syncMarkdownFile->>syncMarkdownFile: too_large detected
    syncMarkdownFile->>deleteCached: Delete cached document
    deleteCached->>RPCClient: enqueue delete_authored_document
    syncMarkdownFile->>syncMarkdownFile: Return "deleted"
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• xDarkicex/openclaw-memory-libravdb#188: Modifies oversized file handling in syncCandidates and syncMarkdownFile around cached-document deletion semantics.
• xDarkicex/openclaw-memory-libravdb#172: Overlaps on cached document deletion and snapshot dirty-state handling in retraction paths.
• xDarkicex/openclaw-memory-libravdb#263: Tightens deletion semantics to ensure invalid conditions trigger retraction rather than silent skip.

Suggested labels

bug, testing

Suggested reviewers

• xDarkicex

Poem

🐰 A file grows fat, too large to chew,
The scanner said "skip it," but that won't do,
Now delete we must, no ghosts remain,
Fresh memory, clean—no stale refrain! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: retracting markdown documents that become oversized, which directly addresses the primary objective.
Linked Issues check	✅ Passed	All objectives from issue #315 are met: the PR handles oversized files by deleting cached documents, covers both skip paths (pre-read estimate and streamed too_large), includes a regression test, and avoids deleting never-ingested files.
Out of Scope Changes check	✅ Passed	All changes are directly scoped to fixing the oversized markdown document retraction issue. The helper function, conditional logic in syncMarkdownFile, and integration test are all necessary to address the linked issue.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}