Honor .ignore files, process watcher events off-thread, and parallelize the matcher walk

[Shengyu Fu (shengyfu)]

Summary

Three related changes to how tgrep discovers and applies ignore rules, and how the file watcher consumes events. Supersedes #88 and #89 — both are incorporated here, adapted to the current architecture.

1. Parallelize the gitignore matcher walk

gitignore::build_matcher enumerated .gitignore files with a single-threaded WalkBuilder. The walk is almost pure I/O wait, so one thread serialized every directory read. Everything in walker.rs had used build_parallel() for a long time; this was the last serial full-tree walk.

The cost was extreme on large or network-backed trees. From a user's trace on a 285k-file repo on a network drive:

[trace] gitignore matcher build complete in 205485.7ms
[trace] stale check: ... (walk: 1611ms)

Both traversed the same tree, seconds apart — a ~128× gap that is entirely thread count.

2. Honor .ignore files (supersedes #88)

The walker now collects .ignore alongside .gitignore, and IgnoreMatcher applies both. Where the two collide, .ignore wins — within a directory via the nested sort, and at the root by being added last.

.gitignore is now git-gated, matching the indexing walk's require_git default: it applies only inside a repository. Without this the watcher could filter on rules the indexer never applied. .ignore applies either way.

3. Process watcher events off-thread (supersedes #89)

Events are handed to a worker thread over a bounded queue rather than indexed inside the notify callback. That callback runs on the platform's notification thread — on Windows it owns a fixed-size ReadDirectoryChangesW buffer — so doing file I/O and trigram extraction there stalls it, and everything arriving meanwhile is dropped by the OS with no error we can observe.

On overflow, events are dropped deliberately and reconciled with a stale check once the burst drains (a quiet interval, so one overflow doesn't trigger a stale check per queued event). The bound is configurable via --watcher-queue-cap, default 16384.

Why these couldn't be replayed as written

Both PRs predate the current nested IgnoreMatcher, and #88's flat root-anchored Gitignore would have regressed nested anchoring — its own description admits the mis-anchoring. Its .ignore idea is grafted onto IgnoreMatcher instead.

More seriously, #88 deletes build_gitignore_matcher_after_ready, which is now also called from the ignore-rules-dirty loop added after that PR was opened. Applying it unchanged would have silently broken runtime .gitignore reloads. Both call sites now route through background_refresh_stale.

Bonus: one fewer full-tree walk on warm start

Warm start built the matcher with its own full-tree walk and then walked again for the stale check. The matcher is now built from the stale walk's results, so the second traversal is gone. This also required hoisting the walk above background_refresh_stale's early returns, so gitignore_pending is released even when the index turns out to be up to date.

Bug found while testing: self-ignoring ignore files were dropped

An ignore file matched by its own rules — the common build/.gitignore containing * — is filtered out of the walk's own output, so collecting the ignore files a walk yields silently drops it. The indexer still skipped the subtree (the walker reads the file even when it won't emit it), but the matcher built from those paths did not know about it, so the watcher would index a subtree the indexer deliberately excluded — exactly the divergence should_skip_watcher_path exists to prevent.

Walks now probe each directory they descend into instead. That is also complete (a .gitignore only matters when the walk descends into its directory, and every such directory is yielded) and duplicate-free.

This was masked in tests because the fixtures had no .git directory, which made the walker's own gitignore filtering inert.

Measurements

Warm start against a 94,634-file tree (403 .gitignore files), pre-built index:

	before	after
serial matcher walk	1942.8ms	—
parallel matcher walk (separate)	616.0ms	—
matcher build from stale walk	—	80.4ms
stale-check walk	329ms	342ms
full-tree traversals	2	1

Stale check reports the same 94,637 files before and after, confirming the metadata walk's result set is unchanged.

Verification

• Full suite green (267 tests), clippy clean with -D warnings.
• Parallel vs serial collection A/B-diffed on a real 94k-file tree via a temporary dump: 403 paths each, byte-identical.
• New end-to-end test watcher_dot_ignore.rs asserts a file under an .ignored directory is never indexed and that an ordinary file created at the same moment is — the second assertion keeps a watcher that simply stopped working from passing the first. Confirmed non-vacuous by disabling .ignore support and watching it fail.
• New unit test for the self-ignoring build/.gitignore case.
• Smoke-tested on a real tree; the single-walk warm start and the watcher gate behave as intended.

Note on ordering

An earlier draft of this description claimed collection order affects ignore precedence. That is wrong: IgnoreMatcher::with_nested already re-sorts deepest-first, which is what establishes git's between-level precedence. The explicit sort exists only because that re-sort is stable, so unsorted parallel-completion order would let the built matcher vary run to run.

---

4. Make index flush ~10× faster and halve its peak memory

Diagnosed from the same user trace, where a single changed file cost 38s to flush on a 2.7 GB index:

[trace] stale check: 1 changed, 0 new, 0 deleted (walk: 1611ms)
[trace] flush: snapshotted 285006 files in 38325.1ms

None of that is disk I/O — it is all HybridIndex::full_snapshot rebuilding the in-memory index before a byte is written. Three causes:

• The old→new file id table was a HashMap<u32, u32>, consulted once per posting entry. 170M random-u32 hashes cost 3.4s on their own. Reader ids are the contiguous range 0..num_files, so a dense Vec indexes directly — and the lookups then run in file-id order, sequential rather than scattered.
• The whole index was decoded into a Vec<(u32, Vec<PostingEntry>)> and only then remapped, so every posting entry existed twice at peak (~1.4 GB × 2 on the tree below, ~7 GB at the user's scale). Decode and remap are now one fused pass, removing the intermediate copy.
• Both that pass and the final sort are per-trigram independent, so they now fan out across cores.

Measured on the linux kernel tree (94,634 files, 446,892 trigrams, 170,005,346 posting entries, 0.97 GB index), one changed file:

phase	before	after
decode all postings	1352.8ms	┐
remap reader postings	3424.7ms	┘ 479.6ms (fused)
overlay merge	344.7ms	7.9ms
sort postings	375.6ms	26.7ms
full_snapshot	5581.1ms	564.6ms (9.9×)
total flush	9.5s	2.5s

Correctness check: rebuilt the same tree from scratch and diffed search results against the incrementally-flushed index — identical output across six queries covering 64k+ hits (scheduler_tick, EXPORT_SYMBOL_GPL, struct task_struct, kmalloc, spin_lock_irqsave, netdev_priv).

What this does not fix

It lowers the constant, not the complexity. A one-file change still costs O(entire index), because the bounded-heap stream merge in append_overlay_to_index requires append-only input, and a changed file supersedes a reader entry. Teaching that merge to skip superseded file ids would remove the whole-index work entirely — that is the real fix, and it is left as follow-up.

5. Fold watcher-recorded stamps into the stale-check baseline

Part of #89 that I initially and wrongly dismissed. filestamps.json only advances on flush, but the watcher updates state.file_stamps on every change, so mid-session the on-disk copy lags. An overflow-triggered stale check reading only the on-disk stamps re-indexes everything handled since the last flush — and a file created and deleted inside that window appears in neither the stamps nor the filesystem, so it is never classified as deleted and lingers in the index indefinitely. The in-memory stamps are the fresher record and now take precedence.

[Copilot]

Pull request overview

This PR parallelizes the .gitignore discovery walk performed by tgrep_core::gitignore::build_matcher, bringing it in line with the rest of the codebase’s parallel filesystem walking approach and significantly reducing warm-start latency on large or network-backed repositories.

Changes:

• Switch .gitignore enumeration from a single-threaded WalkBuilder::build() iterator to build_parallel() with a shared Mutex<Vec<PathBuf>> collector.
• Add a dedicated thread-count helper (matcher_walk_thread_count) and sort discovered paths to keep matcher construction reproducible across runs.

Show a summary per file

File	Description
tgrep-core/src/gitignore.rs	Parallelize .gitignore enumeration during matcher construction; add thread-count helper and deterministic sorting.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 1/1 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

tgrep-cli/src/main.rs:428

• watcher_queue_cap.map(|n| n as usize) can silently truncate on 32-bit targets (and for very large values), potentially creating an unexpectedly small queue cap. Prefer parsing watcher_queue_cap as usize in clap (or use usize::try_from(n) and surface an error) so invalid values are rejected instead of truncated.

            serve::run(

tgrep-core/src/walker.rs:218

• The doc comment says hidden entries are visited so .gitignore/.ignore are seen, but this walk uses WalkBuilder::hidden(true) (hidden entries skipped). Ignore files are still discovered via the per-directory ignore_files_in probe, so this comment is misleading. Update it to describe the probing behavior (and that hidden dirs/files are still skipped for the metadata set).

/// Walk a directory tree collecting filesystem metadata (mtime, size) plus the
/// `.gitignore` / `.ignore` files encountered. No file content is read — this
/// is used for stale file detection on startup.

tgrep-cli/src/serve.rs:435

• This comment claims “Every exit path clears the gate”, but publish_watcher_matcher returns early when !state.watch_enabled without clearing gitignore_pending. Either adjust the wording (e.g., only when watching is enabled) or explicitly clear the gate on that early-return path to keep the comment and behavior aligned.

/// Every exit path clears the gate. A repo with no ignore rules at all yields
/// `None`, which is a legitimate final answer rather than a missing matcher, so
/// it clears the gate too — otherwise the watcher would stay muted forever.

• Files reviewed: 8/8 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (3)

Previously missed (3) — in code that hasn't changed since the last review.

tgrep-core/src/walker.rs:281

• walk_file_metadata probes every descended directory with gitignore::ignore_files_in(...) unconditionally. When no_ignore is true, the caller will never build/use an ignore matcher, so these extra filesystem stats are pure overhead. Consider guarding the probe with if !no_ignore { ... } (or otherwise making ignore-file collection conditional) so --no-ignore metadata walks stay as cheap as possible.

                if should_skip_dir(&entry, &exclude) {
                    return ignore::WalkState::Skip;
                }
                // Probing each descended directory finds ignore files the walk
                // itself filters out; see `gitignore::ignore_files_in`. It also
                // keeps `hidden(true)` intact, so the metadata set is unchanged.
                let (gitignore, dot_ignore) = crate::gitignore::ignore_files_in(entry.path());
                if let Some(path) = gitignore {
                    gitignore_files.lock().unwrap().push(path);
                }
                if let Some(path) = dot_ignore {
                    ignore_files.lock().unwrap().push(path);
                }

tgrep-cli/src/serve.rs:427

• The docstring for publish_watcher_matcher says “Every exit path clears the gate”, but the early return when !state.watch_enabled does not clear gitignore_pending. Either update the comment to scope that guarantee to the watch-enabled case, or clear the flag on that early return for consistency.

/// Publish the watcher's ignore matcher from ignore files discovered by a walk
/// that already happened, and release the `gitignore_pending` gate.
///
/// Reusing the caller's walk is the whole point: `gitignore::build_matcher`
/// would traverse the entire tree a second time purely to find the same ignore
/// files. On a 289k-file repo on a network drive that second walk cost 205s,
/// against 1.6s for the stale-check walk over the same tree.
///
/// Every exit path clears the gate. A repo with no ignore rules at all yields
/// `None`, which is a legitimate final answer rather than a missing matcher, so
/// it clears the gate too — otherwise the watcher would stay muted forever.
fn publish_watcher_matcher(
    state: &ServerState,
    root: &Path,
    walk: &tgrep_core::walker::MetaWalkResult,
) {
    if !state.watch_enabled {
        return;
    }

tgrep-core/src/gitignore.rs:420

• build_matcher’s filter_entry currently runs p4ignore matching on every file entry to decide whether to yield it, but the parallel walk no longer consumes file entries at all (it only probes directories via ignore_files_in inside walker.run). Consider short-circuiting file entries in filter_entry (e.g., if !is_dir { return false; }) so the enumeration walk yields only directories and avoids per-file p4ignore work that can dominate on large trees.

            if entry.file_type().is_some_and(|ft| ft.is_dir())
                && entry
                    .file_name()
                    .to_str()
                    .is_some_and(|n| n.starts_with('.'))
            {
                return false;
            }
            if entry.file_name() == ".gitignore" || entry.file_name() == ".ignore" {
                return true;
            }

• Files reviewed: 9/10 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

tgrep-cli/src/serve.rs:427

• publish_watcher_matcher’s doc comment says “Every exit path clears the gate”, but the if !state.watch_enabled { return; } early return leaves gitignore_pending unchanged. Either clear gitignore_pending in that branch too, or adjust the comment to reflect that the gate is only relevant when watching is enabled.

/// Every exit path clears the gate. A repo with no ignore rules at all yields
/// `None`, which is a legitimate final answer rather than a missing matcher, so
/// it clears the gate too — otherwise the watcher would stay muted forever.
fn publish_watcher_matcher(
    state: &ServerState,
    root: &Path,
    walk: &tgrep_core::walker::MetaWalkResult,
) {
    if !state.watch_enabled {
        return;
    }

tgrep-core/src/gitignore.rs:380

• build_matcher’s rustdoc still describes this as enumerating “every .gitignore file inside the tree” via WalkBuilder. The implementation now (a) collects .ignore as well, and (b) probes each descended directory via ignore_files_in rather than relying on yielded ignore-file entries. Updating the docs to match the probing approach would prevent future refactors from reintroducing the “hidden(true) would hide .gitignore” assumption.

/// Build a `Gitignore` matcher rooted at `root`, mirroring the same
/// ignore semantics that `walker::walk_dir` / `walker::walk_file_metadata`
/// apply during iteration. Loads:
///   * `.git/info/exclude` (if present)
///   * every `.gitignore` file inside the tree
///   * the user's global gitignore (via `GitignoreBuilder`'s defaults)
///   * root-level `p4ignore.ini`
///
/// Uses `WalkBuilder` to enumerate `.gitignore` files so we automatically
/// skip the `.git` dir and gitignored subtrees while collecting rules.
/// Returns `None` when no rules could be loaded.

• Files reviewed: 9/10 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

tgrep-cli/src/serve.rs:425

• The doc comment says "Every exit path clears the gate", but when !state.watch_enabled the function returns without clearing gitignore_pending. Even if it’s currently initialized to false in that mode, the comment is inaccurate/misleading; update the wording or clear the gate consistently.

/// Every exit path clears the gate. A repo with no ignore rules at all yields
/// `None`, which is a legitimate final answer rather than a missing matcher, so
/// it clears the gate too — otherwise the watcher would stay muted forever.
fn publish_watcher_matcher(
    state: &ServerState,
    root: &Path,
    walk: &tgrep_core::walker::MetaWalkResult,
) {
    if !state.watch_enabled {
        return;
    }

Cargo.toml:7

• The workspace package version is being downgraded from 1.0.2 to 1.0.1. Unless this PR is intentionally reverting a release, this is likely a mistake and could break release/versioning expectations; consider keeping 1.0.2 or bumping forward instead of decreasing.

[workspace.package]
version = "1.0.1"
edition = "2024"

• Files reviewed: 10/11 changed files
• Comments generated: 0 new
• Review effort level: Lite