fix: cap recursive inotify watches to prevent multi-GB memory growth#12116
Draft
warp-dev-github-integration[bot] wants to merge 1 commit into
Draft
fix: cap recursive inotify watches to prevent multi-GB memory growth#12116warp-dev-github-integration[bot] wants to merge 1 commit into
warp-dev-github-integration[bot] wants to merge 1 commit into
Conversation
On Linux, the notify crate's inotify backend creates one watch descriptor per directory when using RecursiveMode::Recursive. For users with very large directory trees (deep node_modules, monorepos), this causes the internal HashMap and event buffers to grow to 10+ GB. Add a bounded directory count check before registering recursive watches. When a directory tree exceeds 100,000 subdirectories, the watcher falls back to RecursiveMode::NonRecursive to prevent excessive memory usage. Sentry issue: https://sentry.io/organizations/warpdotdev/issues/7259255054/ Root cause: notify::inotify::EventLoop::add_watch HashMap + Vec<u8> growth (98.4% of 11.22 GB sampled heap in heap-profile.pb) Co-Authored-By: Oz <oz-agent@warp.dev>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cap the number of recursive inotify watches to prevent multi-GB memory growth on Linux.
Root cause: On Linux, the
notifycrate's inotify backend creates one watch descriptor per directory when usingRecursiveMode::Recursive. For users with very large directory trees (deepnode_modules, monorepos with hundreds of thousands of subdirectories), the internalHashMap<PathBuf, (WatchDescriptor, ...)>and associatedVec<u8>event buffers grow to 10+ GB, triggering Sentry memory alerts.Heap profile evidence (event ff1830282d61):
Vec<u8>::cloneinnotify::inotify::EventLoop::add_single_watchHashMaprehash innotify::inotify::EventLoop::add_watchFix: Add a bounded directory count check (
estimate_directory_count) before registering recursive watches. When a directory tree exceeds 100,000 subdirectories, the watcher falls back toRecursiveMode::NonRecursiveto prevent excessive memory usage. The check useswalkdirwith early termination so it doesn't traverse the entire tree.Trade-off: Directories exceeding the threshold will only have top-level change detection (new files/dirs created or deleted directly in the root). Subdirectory changes won't trigger events. This is acceptable because the alternative is a 10+ GB memory spike that can OOM the process.
Linked Issue
linux_stable_release(Linux 6.17.0-1017-aws)Testing
cargo check -p watcherpassescargo clippy -p watcher -- -D warningspasses./script/runAgent Mode
Conversation: https://staging.warp.dev/conversation/6007cbda-5c59-4092-b92c-d1fd7b9070a2
Run: https://oz.staging.warp.dev/runs/019e8b03-c70c-7da9-91e5-8913a6960343
This PR was generated with Oz.