fix: filter gitignored directories from inotify recursive watches#12114
Draft
warp-dev-github-integration[bot] wants to merge 1 commit into
Draft
fix: filter gitignored directories from inotify recursive watches#12114warp-dev-github-integration[bot] wants to merge 1 commit into
warp-dev-github-integration[bot] wants to merge 1 commit into
Conversation
The repo_watch_filter() used by repository and metadata watchers only pruned .git/ internal directories but did not skip gitignored directories (node_modules/, target/, etc.). When watching a repository recursively, inotify registers watches on every subdirectory — for projects with large ignored trees this consumed 11+ GB of memory from HashMap growth and rehashing in the notify crate's EventLoop. Changes: - Add repo_watch_filter_with_gitignores() that combines git-internal path filtering with gitignore-based directory pruning in the descend predicate, following the pattern already used by the codebase indexer. - Update DirectoryWatcher::start_watching_directory() and LocalRepoMetadataModel::add_repository_internal() to load root-level gitignore rules and pass them to the new filter. - Fix FileModel::register_file_path() and fallback_to_individual_watchers() to use NonRecursive mode on the parent directory (matching open()'s existing correct behavior) instead of Recursive on the file path. Co-Authored-By: Oz <oz-agent@warp.dev>
56894c1 to
cf3ee43
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fix excessive memory usage (11+ GB) caused by inotify registering watches on gitignored directory trees (e.g.
node_modules/,target/).Root cause:
repo_watch_filter()only pruned.git/internal directories from the recursive walk, but did not skip gitignored directories. When watching a repository recursively on Linux, inotify creates a watch on every subdirectory. For projects with large ignored trees, thenotifycrate's internalHashMap<PathBuf, (WatchDescriptor, WatchMask, ...)>grew to ~3 GB from rehashing, andVec<u8>::cloneduring rehashing consumed ~8 GB.Heap profile evidence (Sentry event
82189ab726af469a990bbe20819b78ca):notify::inotify::EventLoop::add_watch→add_single_watchVec<u8>::clone(HashMap buffer rehashing)hashbrown::raw::RawTable::reserve_rehashFix:
repo_watch_filter_with_gitignores()that combines the existing.git/filtering with gitignore-based directory pruning in the descend predicate. This follows the pattern already used by the codebase indexer (CodebaseIndexManager::watch_path).DirectoryWatcher::start_watching_directory()andLocalRepoMetadataModel::add_repository_internal()to load root-level gitignore rules and use the new filter.FileModel::register_file_path()andfallback_to_individual_watchers()to useNonRecursivemode on the parent directory (matchingopen()'s existing correct behavior) instead ofRecursiveon the file path.Linked Issue
Sentry: https://sentry.io/organizations/warpdotdev/issues/7259255054/events/82189ab726af469a990bbe20819b78ca/
Where appropriate, screenshots or a short video of the implementation are included below (especially for user-visible or UI changes).
Testing
cargo checkandcargo clippypass forrepo_metadataandwarp_files.Agent Mode
Conversation: https://staging.warp.dev/conversation/cba99240-32b8-48ea-a4f4-c74fe5ae592c
Run: https://oz.staging.warp.dev/runs/019e8db0-eeaf-7b89-81bf-181eae1835e7
This PR was generated with Oz.