[codex] cap indexing to first 10k lines per file - #480
Conversation
| const MAX_INDEXED_LINES: usize = 10_000; | ||
|
|
||
| fn indexed_text(text: &str) -> &str { | ||
| match text.match_indices('\n').nth(MAX_INDEXED_LINES - 1) { | ||
| Some((idx, _)) => &text[..idx + 1], | ||
| None => text, | ||
| } | ||
| } |
There was a problem hiding this comment.
🚩 Completion features silently degrade for files exceeding 10K lines
After this change, go-to-definition, references, heading completions, tag completions, and indexed block features will silently stop working for content beyond line 10,000. The rope still stores the full text (verified by the update_vault_keeps_full_text_in_rope test at line 3385), so preview/hover features using select_line will still work for all lines, creating an inconsistency where a user can see content but not navigate to or from references within it. There is no user-facing notification or log message when truncation occurs. Whether this is acceptable depends on the use case — it may be worth logging when a file is truncated.
Was this helpful? React with 👍 or 👎 to provide feedback.
Summary
Caps markdown file indexing to the first 10,000 lines per file at the
MDFile::newchokepoint. Vault construction and open-file updates still keep the full file text inRope, so editor operations can continue to work with the full buffer while expensive regex-based reference/index parsing avoids pathological long-file stalls.Why
MDFile::newis used by both full vault scans and file updates, and it runs reference, heading, tag, footnote, link-reference, indexed-block, metadata, and code-block parsing across the input text. Very large markdown files can stall the LSP during those passes. Capping the parser input in one place bounds that work without changing the stored file contents.Validation
cargo fmt --checkcargo test(77 passed)Added regression coverage that verifies content on line 10,000 is indexed, content after line 10,000 is ignored by
MDFile, andVault::update_vaultstill stores the full text in the rope.