[Bug]: tons of full rescans while tracked repo files are barely changed

[edezhic]

Continuation of #616

The issue is still present, so I've ran another deeper investigation. Here is the full report (kudos to gpt-5.6-sol-xhigh):

macOS watcher rescan storms: investigation and fix direction

Scope

This report investigates two warnings observed when fff-search watches an
active Rust workspace on macOS:

Received rescan event for paths [".../.git"], triggering full rescan
Too many affected paths in a single batch, triggering full rescan

The findings were checked against:

• fff current main at 073698c (fff-search 0.10.0).
• fff-search 0.9.6, which shows the same core behavior.
• Resolved watcher dependencies notify 9.0.0-rc.4 and
  fff-notify-debouncer-full 0.9.4.
• A real workload with timestamp-correlated watcher logs and filesystem
  metadata.
• Focused unit reproductions against current main.

Executive summary

There are three separate problems.

1. Ignored paths count toward an index-overflow limit. On macOS, fff
   recursively receives events from ignored trees such as target/. The
   handler filters those paths out of the index, but then counts the original
   raw paths against MAX_OVERFLOW_FILES = 1024. A batch of 1,025 ignored
   events therefore triggers a full rescan even though it requires zero index
   additions.

2. The unconditional recursive strategy exposes fff to avoidable event
   storms on macOS (and currently Windows). An earlier adaptive macOS
   strategy used non-recursive watches for indexed directories below a
   directory-count threshold and recursive watching above it. PR feat: Improve stability of file search discovery in background  #431 changed
   macOS and Windows to always recursively watch the workspace root. This
   means .gitignore affects indexing but does not reduce the event volume
   delivered by the OS watcher.

3. The Flag::Rescan bypass is a correctness bug. For a small set of
   non-directory, non-ignored paths, the handler responds to an OS
   “events were lost” marker with a plain break. It neither rescans nor
   processes later events in the same debounced batch. On Linux, an inotify
   queue overflow has no paths, so the empty path list also takes this bypass
   due to vacuous all().

The .git path in the first warning is not evidence that .git caused the
event storm. fff registers .git as an overlapping watch root even when the
recursive base watch already covers it, and the debouncer retains only one
rescan marker. In the measured incident, thousands of target/doc writes and
the warnings occurred at the same timestamps, while sampled .git mtimes did
not show activity in that window.

End-to-end event flow

build / generator writes ignored files under the workspace
  -> macOS recursive FSEvents stream receives those events
  -> notify translates raw flags to Events
  -> fff-notify-debouncer-full batches for 50 ms
  -> handle_debounced_events()
       -> ignored paths are excluded from index mutations
       -> raw path count still exceeds 1024
       -> full rescan

At higher event rates:
  -> FSEvents may report MustScanSubDirs (commonly with dropped-event flags)
  -> notify emits EventKind::Other + Flag::Rescan
  -> fff logs "Received rescan event..." and requests a full rescan

The two warnings are sibling consequences of the same event flood. The
user-space > 1024 branch does not manufacture a later OS Flag::Rescan
event.

Confirmed issue 1: ignored paths trigger the raw batch limit

The macOS watcher is recursive:

let use_recursive = cfg!(any(target_os = "macos", target_os = "windows"));

if use_recursive {
    debouncer.watch(base_path.as_path(), RecursiveMode::Recursive)?;
}

Source:
crates/fff-core/src/watcher/background_watcher.rs, BackgroundWatcher::new
and create_debouncer.

The handler correctly filters ignored creates and modifications:

let (is_dir, is_ignored) = if is_removed {
    (false, true)
} else {
    (path.is_dir(), filter.is_ignored(path))
};

// ...
} else if !is_ignored {
    paths_to_add_or_modify.push(path.as_path());
}

It then increments the limit with the unfiltered event paths:

affected_paths_count += debounced_event.event.paths.len();
if affected_paths_count > MAX_OVERFLOW_FILES {
    warn!(
        ?affected_paths_count,
        max = MAX_OVERFLOW_FILES,
        "Too many affected paths in a single batch, triggering full rescan",
    );
    need_full_rescan = true;
    break;
}

MAX_OVERFLOW_FILES is documented as storage reserved for files discovered
after the initial scan:

/// Capacity reserved for files the watcher discovers after the initial scan;
/// exceeding it forces a full rescan.
pub const MAX_OVERFLOW_FILES: usize = 1024;

Source: crates/fff-core/src/constants.rs.

Raw event count and index overflow capacity are different concepts:

• An ignored create consumes no index capacity.
• A modification of an existing indexed file consumes no new capacity.
• A removal consumes no overflow capacity.
• A .git event is skipped as an index entry.
• Duplicate paths are deduplicated only after the raw limit check.

The current guard can therefore force a rescan without protecting the
resource named by the constant.

Focused reproduction

A temporary unit test was run against current main:

1. Initialize a git repository with /target in .gitignore.
2. Build a picker whose initial index excludes target/.
3. Construct one debounced batch containing 1,025 Create(File) events for
   existing files under target/.
4. Call handle_debounced_events.
5. Observe a dispatched WatchEventKind::Rescan.

The test passed, confirming that 1,025 ignored paths trigger a full rescan.
The temporary test was removed after validation.

Confirmed issue 2: macOS and Windows always watch ignored subtrees

Commit 1bcbce2 from PR #431 changed the strategy from adaptive to
unconditional recursion.

Before that commit:

let use_recursive =
    cfg!(target_os = "macos") && watch_dirs.len() > MAX_MACOS_NONRECURSIVE_WATCHES;

Current behavior:

let use_recursive = cfg!(any(target_os = "macos", target_os = "windows"));

For a repository with at most 4,096 discovered indexed directories, the old
macOS strategy watched the base plus each discovered indexed directory
non-recursively. Ignored subtrees were not recursively observed. Windows also
used the non-recursive branch. The current strategy observes every descendant
on both platforms and relies on filtering after delivery.

This matters for build systems and package managers. A repository can have a
few hundred indexed source files while its ignored build directory contains
hundreds of thousands of files.

The current FSEvents rationale is stale

The source comment says each non-recursive watch() call creates a separate
FSEvent stream. That is not how the resolved notify 9.0.0-rc.4 backend
currently behaves:

• FsEventWatcher::watch_inner stops the current stream, appends the new
  path, and calls run.
• run creates one FSEventStream from the complete self.paths array.
• update_paths can apply many path operations with one stop/start cycle.

There may still be a practical limit or performance cost for a large
pathsToWatch array, so a recursive fallback remains reasonable. However,
“one stream per directory” is not a valid reason to force recursive watching
for a repository with only a small number of indexed directories.

Confirmed issue 3: the rescan bypass can leave the index stale

Current code:

if debounced_event.event.need_rescan() {
    if debounced_event.event.paths.len() < 16
        && debounced_event
            .paths
            .iter()
            .all(|p| !p.is_dir() && !filter.is_ignored(p))
    {
        break;
    }

    warn!(/* ... */);
    need_full_rescan = true;
    break;
}

need_rescan() means the backend reports that individual events may have
been lost. The bypass:

• Does not set need_full_rescan.
• Does not reconcile the reported path.
• Exits the entire event loop.
• Drops every event later in the same debounced batch.

fff-notify-debouncer-full stores the rescan marker separately, then
chronologically sorts it together with expired normal events. A rescan marker
can therefore appear before or after ordinary create/modify events:

• If it appears first, the bypass skips later events.
• If it appears later, earlier changes may be applied, but the handler still
  ignores the “events were lost” condition and can leave a partially updated
  index.

On Linux, notify emits an inotify Q_OVERFLOW marker with no paths:

Event::new(EventKind::Other).set_flag(Flag::Rescan)

For an empty list, len() < 16 is true and Iterator::all is vacuously true,
so the overflow is silently ignored and the rest of the batch is abandoned.

Focused reproduction

A second temporary unit test was run against current main. It intentionally
exercises a valid rescan-first handler input rather than claiming this is the
only debouncer ordering:

1. Build a picker containing existing.rs.
2. Create new.rs after the scan.
3. Pass a batch containing:
   • Flag::Rescan for existing.rs.
   • Create(File) for new.rs.
4. Observe that new.rs is absent from the picker after
   handle_debounced_events returns.

The test passed, confirming that the bypass discards following events without
recovering the index.

Strongly supported issue 4: overlapping .git roots obscure attribution

fff always calls watch_git_status_paths, including after installing a
recursive base watch:

if use_recursive {
    debouncer.watch(base_path.as_path(), RecursiveMode::Recursive)?;
}

watch_git_status_paths(&mut debouncer, git_workdir.as_ref());

That function adds .git and .git/logs as non-recursive roots. On macOS,
the recursive base already covers both paths.

With resolved notify 9.0.0-rc.4:

• All roots are passed to one FSEvent stream.
• The callback chooses the longest matching registered root.
• The debouncer has one rescan_event slot; a later rescan marker replaces
  the earlier one.

This makes a surviving .git label possible even when the pressure that
caused event loss came from another root in the same stream. The existing
warning logs only paths; it omits event.info(), which could distinguish
"rescan: user dropped" from "rescan: kernel dropped".

The exact FSEvents root-selection sequence cannot be reconstructed from the
existing logs. Nevertheless, the sampled mtimes are inconsistent with .git
being the high-volume source during the measured burst and strongly support
the root-attribution explanation.

Runtime case study

The affected application used fff-search 0.9.6 with watch: true on a Rust
workspace:

• Normal project tree: about 163 files and 15 directories outside .git and
  target/.
• Ignored target/: 17 GiB, about 140,858 files and 16,325 directories.
• Recorded between July 14 and July 17:
  • 3,744 Received rescan event .../.git warnings.
  • 43 Too many affected paths warnings.
• Every structured batch-overflow warning reported exactly
  affected_paths_count = 1025, matching the strict > 1024 branch.

One especially clear burst occurred at 2026-07-17T22:00:12Z:

• 5,429 existing entries under target/ had that second as their mtime.
• 3,146 more had 22:00:13.
• 388 more had 22:00:14.
• 8,719 of the 8,963 entries were rustdoc-generated .js files under
  target/doc.
• No sampled .git entry had an mtime in this window; its next sampled writes
  were at 22:01:28.
• .git rescan warnings appeared at 22:00:12.254,
  22:00:12.511, 22:00:13.120, and 22:00:23.417.

This is a timestamp-level correlation between ignored build output and
rescan warnings whose displayed path was .git.

Static size alone is not the trigger. The trigger is high-rate creation,
replacement, and deletion inside a recursively watched ignored subtree.
Deleted temporary files and reused directory mtimes also mean a
post-incident mtime count underestimates the original event volume.

Related API contract mismatch

The public watch documentation states:

Gitignored and other ignored files are never triggering watcher.

This appears in crates/fff-core/src/shared.rs and is repeated by language
bindings.

Ignored files are excluded from normal subscriber events, but an ignored
event storm can trigger watch_registry.dispatch_rescan(base_path), which is
broadcast to every subscription. The documented guarantee is therefore not
true for rescan events.

What is not causing the warnings

• A normal low-volume .git/index, HEAD, or logs/HEAD change requests the
  separate GitStatusWorker; it does not directly request a full filesystem
  rescan. Raw .git paths still count toward the flawed 1,024-path guard, so
  an unusually large git-internal batch can contribute to that branch.
• A user-space “Too many affected paths” rescan does not create a synthetic
  notify::Flag::Rescan. Both warnings can follow the same underlying flood,
  but there is no direct code path from one warning to the other.
• Full rescans do not reinstall the watcher
  (install_watcher: false). Repeated warning lines do not imply an equal
  number of concurrent scans: trigger_full_rescan_async coalesces requests
  through rescan_pending.
• Application databases stored outside the watched workspace are unrelated
  to this event stream.

Recommended fix direction

Phase 1: correctness and accounting

1. Always recover from Flag::Rescan.

   Remove the < 16 bypass. Until fff has a correct path-scoped subtree
   reconciliation operation, any backend “events were lost” marker should
   schedule a full rescan.

   Minimal direction:

   if debounced_event.event.need_rescan() {
       warn!(
           paths = ?debounced_event.event.paths,
           info = ?debounced_event.event.info(),
           "Filesystem events were lost; scheduling full rescan",
       );
       need_full_rescan = true;
       break;
   }

2. Decouple raw batch size from overflow index capacity.

   Do not compare sum(event.paths.len()) with MAX_OVERFLOW_FILES.
   First classify and deduplicate paths, then reason about actual index
   mutations.

   In particular:

   • Ignored creates/modifications should not count.
   • Existing-file modifications should not count as new overflow files.
   • Removes can be applied without consuming overflow capacity.
   • .git events should not count.

   A proactive capacity check should use the number of genuinely new,
   indexable files and the remaining overflow capacity. The existing
   handle_create_or_modify(path).is_none() and post-apply overflow check
   already provide a final safety net.

   If a separate CPU/memory guard for exceptionally large callback batches is
   still needed, give it a separate name and threshold. It should be applied
   after cheap ignore filtering and deduplication rather than reuse
   MAX_OVERFLOW_FILES.

3. Improve rescan observability and warning coalescing.

   Log:

   • event.info() (user dropped versus kernel dropped).
   • Raw path count.
   • Ignored path count.
   • Deduplicated actionable count.
   • Whether a scan was started or only marked pending.

   Avoid emitting one warning for every drop marker while a rescan is already
   active or pending. This changes noise, not correctness.

Phase 2: reduce ignored traffic at the OS watcher boundary

Restore an adaptive macOS strategy:

• When the discovered indexed-directory count is below a benchmarked
  threshold, watch the base and each indexed directory non-recursively. The
  old threshold was 4,096.
• Register the path set with one update_paths operation so notify does not
  repeatedly stop and recreate the stream during initialization.
• Fall back to one recursive base root when the indexed directory count or
  measured initialization cost exceeds a benchmarked threshold.
• Keep Windows recursive unless separately validated; its backend and failure
  modes differ from FSEvents.

The previous threshold of 4,096 is a starting point, not necessarily the
correct final value. Benchmark stream initialization, event latency, and path
limits with representative repositories.

When using the recursive base strategy, do not additionally register .git
and .git/logs on macOS/Windows. The recursive root already delivers these
events, and the handler checks status-affecting git paths before skipping
.git entries. Keep the explicit git roots for non-recursive strategies.

An optional public watcher strategy (auto, recursive, indexed-dirs)
would also let SDK consumers choose the right trade-off for build-heavy
workspaces without disabling live updates entirely.

Regression tests to add

1. Ignored batch accounting

   • Git repo with /target ignored.
   • Feed 1,025 create events under target/.
   • Assert no full rescan and no index additions.

2. Large legitimate modification batch

   • Feed more than 1,024 modifications of already indexed files.
   • Assert incremental updates do not fail due to overflow-file capacity.

3. Actual new-file overflow

   • Feed enough non-ignored new files to exhaust remaining overflow capacity.
   • Assert exactly one rescan is scheduled.

4. Rescan marker followed by ordinary event

   • Feed [Flag::Rescan(existing_file), Create(new_file)].
   • Assert a rescan is scheduled; never silently return with new_file
     absent.

5. Linux empty-path overflow

   • Feed EventKind::Other + Flag::Rescan with no paths.
   • Assert a rescan is scheduled.

6. macOS ignored-tree integration

   • Start a real watcher in a git repo with an ignored build directory.
   • Generate a high-rate ignored burst.
   • Assert ignored events do not cause the user-space 1,024 overflow branch.
   • Record raw FSEvents drop information separately; OS event loss can still
     require a rescan when recursive fallback is used.

7. Watcher-root strategy

   • For a small macOS repo, assert registered roots are non-recursive indexed
     directories plus required git roots.
   • For a large repo above the selected threshold, assert one recursive root.

Suggested implementation order

1. Remove the Flag::Rescan bypass and add its regression tests.
2. Replace raw path accounting with actionable/capacity accounting.
3. Add structured diagnostic fields and warning coalescing.
4. Gate explicit .git roots on the selected watcher strategy.
5. Restore and benchmark adaptive macOS root registration using
   update_paths.
6. Update the public ignored-event guarantee to explicitly describe rescan
   semantics.

The first two changes address correctness and false rescans without requiring
an immediate watcher architecture redesign. The adaptive macOS strategy then
prevents ignored build traffic from reaching the handler in the common
small-repository case and reduces the chance of FSEvents buffer loss.

[edezhic]

@greenierrobert882-dotcom do you mean replacing with a custom fork with these fixes? Haven't tried, but doable if necessary. Don't wanna maintain a fork thou

[gustav-fff]

[triage-bot] REPRODUCED: Reproduced on origin/main. Branch: triage-bot/issue-690.

Confirmed two of the three defects in the report:

1. Ignored batch trips the 1024 overflow guard. handle_debounced_events at crates/fff-core/src/watcher/background_watcher.rs:457 sums the raw event.paths.len() and forces a full rescan at >1024, even when every path is filtered out as ignored (target/ under .gitignore). Reproduced with a unit test that feeds 2000 Create(File) events under target/ and observes a Rescan dispatch to subscribers.
2. Flag::Rescan bypass drops events silently. background_watcher.rs:365 returns early on need_rescan() when paths.len() < 16 && all !is_dir && !is_ignored. On Linux, Q_OVERFLOW arrives with an empty path list, so all() is vacuously true and the marker is silently discarded. Same bug in any macOS batch of ≤15 small non-ignored paths — no rescan is scheduled.

Report also flags the unconditional recursive RecursiveMode::Recursive watch on macOS/Windows (background_watcher.rs:78) as the reason ignored trees produce events at all. Restoring an adaptive strategy is architectural — out of scope for this triage. Working on the two correctness fixes on triage-bot/issue-690.

Honk-Honk 🪿