Adding quota to only sweep items based on disk usage - #305
Conversation
Latest tests (via
|
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 1 | 991 | 13 |
| keep_episodes | 10 | 1200 | 13 |
| keep_seasons | 1 | 1100 | 5 |
sweep_until_percent_used: 60
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 1 | ~3000 | 45 |
| keep_episodes | 10 | ~3700 | 79 |
| keep_seasons | 1 | ~3600 | 7 |
sweep_until_gb_free: 14000
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | 95 | 4 |
| keep_episodes | 10 | 681 | 11 |
| keep_seasons | 1 | 613 | 3 |
sweep_until_gb_free: 15500
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | ~1700 | 17 |
| keep_episodes | 10 | ~1800 | 17 |
| keep_seasons | 1 | ~1900 | 11 |
JamesWRC
changed the title
There was a problem hiding this comment.
Pull request overview
Adds “sweep-until” caps to prevent a single cleanup run from queueing more deletions than needed to reach a target disk state, complementing the existing disk-usage-threshold-based cleanup behavior.
Changes:
- Introduces
sweep_until_gb_freeandsweep_until_percent_usedper-library config options and applies them during the mark-for-deletion phase. - Computes live disk usage for library folders (via gopsutil) and limits newly marked items based on estimated bytes freed.
- Documents the new behavior and configuration in the README.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| internal/engine/sweep_until.go | New sweep-until limiting logic based on disk usage stats and estimated freed bytes. |
| internal/engine/engine.go | Stores library folder paths and applies sweep-until limits after filters. |
| internal/config/config.go | Adds two new cleanup config fields for sweep-until targets. |
| README.md | Documents sweep-until options and adds a new “Storage Remaining Example” section. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| sorted := make([]arr.MediaItem, len(entry.items)) | ||
| copy(sorted, entry.items) | ||
| sort.Slice(sorted, func(i, j int) bool { | ||
| return estimateFreedSize(sorted[i], cleanupMode, keepCount) > | ||
| estimateFreedSize(sorted[j], cleanupMode, keepCount) | ||
| }) | ||
|
|
||
| var newlyAccumulated int64 | ||
| included := 0 | ||
| for _, item := range sorted { | ||
| if stats.isTargetMet(libraryConfig, alreadyAccumulated+newlyAccumulated) { | ||
| break | ||
| } | ||
| newlyAccumulated += estimateFreedSize(item, cleanupMode, keepCount) | ||
| result = append(result, item) |
There was a problem hiding this comment.
sort.Slice’s comparator calls estimateFreedSize repeatedly (twice per comparison). For large libraries this can get expensive, especially for TV series where estimateFreedSize walks seasons. Consider precomputing the estimated freed bytes once per item (decorate-sort-undecorate) and sorting on the cached values.
| sorted := make([]arr.MediaItem, len(entry.items)) | |
| copy(sorted, entry.items) | |
| sort.Slice(sorted, func(i, j int) bool { | |
| return estimateFreedSize(sorted[i], cleanupMode, keepCount) > | |
| estimateFreedSize(sorted[j], cleanupMode, keepCount) | |
| }) | |
| var newlyAccumulated int64 | |
| included := 0 | |
| for _, item := range sorted { | |
| if stats.isTargetMet(libraryConfig, alreadyAccumulated+newlyAccumulated) { | |
| break | |
| } | |
| newlyAccumulated += estimateFreedSize(item, cleanupMode, keepCount) | |
| result = append(result, item) | |
| type sweepCandidate struct { | |
| item arr.MediaItem | |
| freed int64 | |
| } | |
| sorted := make([]sweepCandidate, len(entry.items)) | |
| for i, it := range entry.items { | |
| sorted[i] = sweepCandidate{ | |
| item: it, | |
| freed: estimateFreedSize(it, cleanupMode, keepCount), | |
| } | |
| } | |
| sort.Slice(sorted, func(i, j int) bool { | |
| return sorted[i].freed > sorted[j].freed | |
| }) | |
| var newlyAccumulated int64 | |
| included := 0 | |
| for _, cand := range sorted { | |
| if stats.isTargetMet(libraryConfig, alreadyAccumulated+newlyAccumulated) { | |
| break | |
| } | |
| newlyAccumulated += cand.freed | |
| result = append(result, cand.item) |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…RC/jellysweep into feature/sweep_until_quota
|
This is ready for review @jon4hz |
|
Hey @JamesWRC, Thanks for the contribution. I'll try to give you a review soonish. |
| // filesystems would need to match on both capacity and inode count to collide, | ||
| // which is effectively impossible in practice. | ||
| func (s *sweepDiskStats) mountKey() string { | ||
| return fmt.Sprintf("fs:%d:%d", s.totalBytes, s.totalInodes) |
There was a problem hiding this comment.
Is there actually a guarantee that a system can't have two disks with the same amount of bytes and inodes?
Using this as indicator to check if a library actually has the same underlying storage seems very random and unreliable to me.
There was a problem hiding this comment.
Also, I think this PR might not take more complicated filesystem layouts into account. For example, different TV shows (especially large ones) could have their own disk.
How would you handle this?
There was a problem hiding this comment.
Cheers for reviewing.
Yeah youre right it does use the same free/used bytes.
Did not think about setups like 1 show per mount.
Will look into these soon.
|
Please correct me if I'm wrong but with the current implementation the deletion just stops if certain storage targets are triggered, right? How is that reflected on the items that were marked for deletion but actually weren't deleted because the storage target was reached? Will those items be deleted immediately, once the storage fills up to a certain percentage? Or are they removed from the tracking db and must be picked up again to go through the entire deletion process incl. grace period, etc. Option 1 feels like it makes deletion much less predictable which is something that I definitely want to avoid and option 2 will generate more noise as people will be notified over and over that their requested media is scheduled for deletion until it's actually deleted once. Neither of those options seem great to me. |
This acts as an additional filter in the candidate pipeline. Only items within the diskspace quota are saved to the database, items cut off by the limit are simply excluded from that run and re-evaluated on the next scheduled sweep. The normal deletion lifecycle (grace period, notifications, user keep requests) applies as usual, this feature only controls which items become eligible for deletion. |
Introduce named sweep-until quota groups to cap deletions across shared storage pools. Config additions: Config.SweepUntilQuotaGroups map and QuotaGroupConfig (percent_used, gb_free), plus Filter.SweepUntilQuotaGroup to opt libraries into a group; removed per-library sweep_until_* fields. Validation now checks quota group definitions and references and warns about unused groups. Engine changes: aggregate disk usage across unique filesystems for each quota group (deduplicating bind-mounts/volumes), seed group budgets from pending DB items, and enforce group-wide sweep-until targets while preserving original filter ordering. Added robust error handling/logging (skip group on stat failures) and updated helper functions (mount key, disk stat aggregation, pending seeds). README updated with documentation, examples, and guidance for quota groups and multi-disk behavior.
|
The previous implementation was a disk based quota. Now its a per library quota, where all mounts are part of the specified group are summed and used in the quota calculation. |
Ah I see, thanks for the clarification. In this case, the filtering processes shouldn't take place in the |
Re implemented sweep_until as a filter. As that is what it is, so there are now minimal changed to the engine. Updated Filter to be extendable, so filters can get a reference to the engine instance. Added largest_first so that if users prefer (i do), If an item is eligible for deletion, delete the biggest item first - as it leads to less media being removed.
Latest tests (via
|
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 1 | ~1100 | 32 |
| keep_episodes | 10 | ~1500 | 31 |
| keep_seasons | 1 | ~1400 | 21 |
sweep_until_gb_free: 14000
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | 142 | 10 |
| keep_episodes | 10 | 246 | 11 |
| keep_seasons | 1 | 201 | 4 |
sweep_until_gb_free: 15500
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | ~1700 | 45 |
| keep_episodes | 10 | ~2200 | 50 |
| keep_seasons | 1 | ~2200 | 40 |
With largest first on
Latest tests (via jameswrc/jellysweep image):
df (1K blocks) when testing: 43560266240 27810515336 13552065784 68% /mnt/media
Currently 68% full
sweep_until_percent_used: 65
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 1 | ~1000 | 7 |
| keep_episodes | 10 | ~1100 | 9 |
| keep_seasons | 1 | ~1200 | 10 |
sweep_until_gb_free: 14000
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | 231 | 1 |
| keep_episodes | 10 | 231 | 1 |
| keep_seasons | 1 | 231 | 1 |
sweep_until_gb_free: 15500
| Cleanup Mode | Keep Count | Est. GB Freed (via stats dashboard) | Item Count |
|---|---|---|---|
| all | 10 | ~1700 | 20 |
| keep_episodes | 10 | ~2000 | 31 |
| keep_seasons | 1 | ~2000 | 32 |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…aths Fixes from a deep review of the sweep_until_quota feature: Budget accounting: - Count freeable size (keep-mode aware, via Sonarr per-season stats) instead of full series size; store it on new Media.FreeableSize (*int64, nil=unknown) so pending and new items are accounted identically - Store the quota group on new Media.QuotaGroup so the budget seed survives library renames and config regrouping - Stamp both fields in every row-creation path via stampQuotaAccounting, using GetKeepCount() to match the deletion logic - Seed dedupes rows sharing a JellyfinID; re-added items (new arr ID, same Jellyfin ID) are excluded instead of double-charging the budget - DB error during seeding withholds all grouped items instead of treating pending items as zero Filesystem identity: - ZFS datasets keyed by pool: used bytes summed per dataset, pool free space counted once (a dataset's statfs reports its own used but pool-wide free) - Partial disk.Usage failure marks the group broken instead of sweeping against incomplete stats - Symlinks resolved before mount matching; overmounted mountpoints resolve last-wins; st_dev verification splits false key collisions and replaces the unstable Total+Free fallback key on unix Engine integration: - Jellyfin per-library fetch failures no longer wipe pending DB rows (and with them the budget seed) as "missing in Jellyfin" - removeRecentlyPlayedItems runs before marking so doomed rows don't consume budget in the same run Config & ordering: - Quota group names are case-insensitive (viper lowercases map keys) - Default order now preserves the order items were reported eligible; alphabetical behavior available as order: title; deprecated largest_first flag removed (never shipped) - Disabled libraries excluded from group mapping Test suite: injectable disk/DB seams, 50+ tests covering Apply end-to-end, multi-run lifecycle, mergerfs/ZFS layouts, failure modes, and config validation; runs locally with go test ./... -race (no real mounts needed). README: documents the broken-group fail-safe, queue-vs-delete timing, ZFS semantics, env-var limitations, and fixes stale/contradictory text. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Why
Myself and another person (see #158) are keen for this feature. Making it the ideal library management tool.
This filters control which content is eligible for deletion, but nothing previously controlled how much gets queued at once. On a large shared disk this can result in far more content being scheduled than the current disk pressure warrants. These new options let you express a target disk state and have jellysweep stop marking items once that target would be reached.
How it works
applySweepUntilLimitruns after the normal filter chain, before items are written to the database:NOTE:
jameswrc/jellysweep. All other filters and logic like 'Keep'ing an item marked for cleaning up, will grab the next lot of media to satisfy the quota on the next sweep.sweep_until_percent_usedandsweep_until_gb_freevalues have been validated by me, and is as i expect.Some other notes;
dfreports usage as used / (used + available), not used / total — because the kernel reserves some blocks for root that are never shown as usable to normal processes. This implementation uses diskUsed + diskFree (where diskFree = Bavail via gopsutil) as the denominator, so 1sweep_until_percent_used: 67` means precisely the 67% you see in df output.