Skip to content

Library overhaul - #1062

Open
nick2000713 wants to merge 218 commits into
Nezreka:devfrom
nick2000713:library-overhaul
Open

Library overhaul#1062
nick2000713 wants to merge 218 commits into
Nezreka:devfrom
nick2000713:library-overhaul

Conversation

@nick2000713

@nick2000713 nick2000713 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Library Manager v2 — a real, Lidarr-equivalent library manager, built entirely on SoulSync's own pipeline

Update after the full lib2 cutover: this rework was honestly rougher than i expected, and @Nezreka was right to keep asking which database would actually be maintained in the future.

keeping the legacy library and lib2 alive as two equal databases made no sense. every metadata edit, enrichment result, import, repair job and media-server sync would need dual writes and permanent synchronization, and sooner or later both versions would disagree. i stopped trying to maintain both.

all active Library reads, writes, imports, enrichment and catalogue repair paths now use lib2 directly. lib2 is the main Library and the single runtime source of truth. the old tables remain only temporarily for the one-time upgrade import and as an upgrade/rollback safety boundary while this is tested on real large libraries. if that proves solid, the legacy schema can be physically removed in a later update.

this was a much larger and rougher rewrite than i originally planned, but the architecture is finally unambiguous: future Library development and maintenance happens on lib2, not on two databases at once. some older sections below describe the original opt-in/two-library phase and should be read as historical context.

Status: Draft / RFC. This is not ready to merge. I'm opening it early on purpose — to get eyes on the direction, get people testing it on their own libraries, and collect feedback before I lock in decisions that get expensive to change later. Please read the "What's still rough" section before you judge it — I tried to be brutally honest about what's missing.


Why I built this

SoulSync's library page has always just been a read-only mirror of whatever the media server (Plex/Jellyfin/Navidrome) reports back. No "monitor this artist," no missing-tracks visibility, no way to pick a specific release when auto-grab gets it wrong, no repair tools pointed at a single artist. It inherited every blind spot the media server has, and it couldn't do anything Lidarr does.

That always bugged me, because SoulSync already has everything a library manager needs under the hood: a real multi-source search, a download pipeline, quality enforcement, tagging, repair jobs. Lidarr has to shell out to external tools for all of that. We don't. So the actual gap wasn't "we need Lidarr's features" — it was "we need Lidarr's front end and decision layer wrapped around functionality we already have."

That's what this is. Not a reinvention of SoulSync's search/download/tag machinery — a proper Artist → Album → Track library model sitting on top of it, so you can finally manage your library the way you'd manage it in Lidarr, without needing Lidarr installed at all.

I want SoulSync's library management to be genuinely on par with Lidarr — not "good enough," actually equivalent. This PR is the first complete pass at that.


What it is, in one picture

Library v2 is not a second download/search/tagging system living next to the old one. It's a new front end and a new "what's monitored / what's wanted" model that talks to the same pipeline SoulSync already had:

flowchart TB
    subgraph UI["Library v2 UI (opt-in, new)"]
        A[Artist / Album / Track view]
        B[Interactive Search & Manage Tracks]
        C[Quality Profiles & Re-Tag Preview]
    end

    subgraph Bridge["Monitoring mirrors into the existing systems"]
        D[Artist monitor → Watchlist]
        E[Album/Track monitor → Wishlist]
    end

    subgraph Core["SoulSync's existing pipeline — unchanged, reused, not duplicated"]
        F[Multi-source Search]
        G[Download Orchestrator]
        H[Quality Gate + AcoustID]
        I[Tagging / Post-Processing]
    end

    UI --> Bridge --> Core
    Core -. same auto-download machinery keeps running .-> Bridge
Loading

If you never open the Library v2 page, nothing changes — it's dormant behind a feature flag. If you do open it, "Monitor this artist" doesn't invent a new download queue; it just adds the artist to the existing Watchlist, the same one the old page uses. Same for Wishlist. So the whole thing stays compatible with everything else running in the app instead of forking it.


Design rules I wouldn't break, even under pressure

These came out of a lot of back-and-forth with myself about where this could go wrong, and they held up through the whole build:

  • Never media-server-dependent — not even for cover art. Artist/album images come from the file itself (embedded cover) first, then a metadata provider, cached to local disk. A pure SoulSync install with no Plex/Jellyfin/Navidrome attached works exactly the same as one with a media server.
  • Monitoring mirrors the systems you already trust. "Monitor" on an Artist = added to the Watchlist. "Monitor" on an Album/Track = added to the Wishlist. The existing auto-scan/auto-download machinery just keeps working — Library v2 doesn't replace it, it feeds it.
  • One quality-profile table, period. No shadow copy for the new UI. Every piece of the pipeline — search ranking, the AcoustID skip decision, the import quality gate, the upgrade scanner — resolves the same profile live, every time. Change a profile in Settings and it's immediately in effect everywhere, including for things Library v2 didn't queue itself.
  • The database is the source of truth, not your folder layout. Every file's location is tracked per-row, so your library is reconstructable regardless of how you've organized folders.
  • Reuse, don't reinvent. Search, download, tagging, repair jobs, quality logic — all the exact same code the rest of the app uses. This is a front end and a decision layer, not a parallel app.

Feature checklist — where it actually lands vs. Lidarr

Capability Old Library page Library v2 Lidarr
Read-only mirror of the media server ✅ (that's all it is) ❌ (own DB, own model)
Works with zero media server attached
Monitor artist / album / track individually
Missing-tracks view with real titles, not just gaps
Interactive search → pick a specific release ✅ (source-aware, quality preview badges)
Quality Profiles enforced at every pipeline stage partial
Full discography browse + "monitor future releases"
Re-tag preview (diff file tags vs. library metadata)
Manage duplicate single/album versions
Per-artist scoped repair/maintenance jobs ✅ (where job-level support exists)
Playlists visible in the library separate page basic view + trigger (see below) ✅ dedicated
Delete removes the actual files, safely 🚧 DB-only for now

What's actually done (this is the bulk of the PR)

I'm not going to paste the full internal changelog here — it's long — but the shape of it:

  • Full Artist/Album/Track data model with multi-artist credit splitting, single-vs-album linking, and an importer that migrates your existing library into it without duplicating anything.
  • Interactive Search & Grab with source-aware results (Soulseek slots/queue, Usenet grabs, age columns for Usenet/torrent), quality/AcoustID toggle overrides, and profile "preview badges" that tell you before you grab whether a result will meet your quality cutoff.
  • Quality Profiles, for real — per-artist assignment, per-track evaluation against upgrade policy and cutoff, live-resolved everywhere instead of frozen at add-time.
  • Discography browser — full provider catalog per artist, EPs section, bulk monitor/unmonitor, background re-sync job so new releases show up without you clicking anything.
  • Manage Tracks — see single-vs-album duplicate pairs, unlink them, move the file to the "correct" version without touching disk.
  • Re-Tag Preview — a real Lidarr-style diff table (file tags vs. library metadata) before you write anything.
  • Maintenance panel wired to the existing repair jobs (metadata gap fill, unknown-artist fixer, tag consistency, rename/reorganize), scoped to the artist you're looking at where the job supports it.
  • Basic Playlists integration — a read-only view of your existing mirrored playlists (source, owner, artwork, discovery/wanted/library counters) with a button that triggers the existing playlist mirror pipeline. Deliberately minimal — see below.
  • All of the above verified against a real ~285-track library, plus a from-scratch Docker + Playwright pass that actually clicks through the UI in a browser rather than trusting curl/unit tests alone.

Test status at the point of writing this: ~8,200 backend tests green, ~100 frontend tests (Vitest) green, zero lint/typecheck warnings, production build clean.


How to try it

It's off by default. To turn it on:

features.library_v2 = true

Set it either in config/config.json (fresh installs, before the DB has a config row) or directly through the app's config store if you're on an existing install (the DB is the actual source of truth once it exists — editing the JSON file alone won't do anything on an established install). Restart the app afterward — the flag is only read at startup.

Turning it on does not touch your existing library, files, or downloads. The only side effect is that finished downloads start getting mirrored into the new Library v2 tables in the background (so the new view stays in sync even for downloads that went through the old Wishlist/Watchlist flow) — that's bookkeeping only, it never writes to disk and never changes what gets downloaded.


What's still rough — please read this before testing

I'd rather list these myself than have someone find them and assume I didn't notice:

  • Deleting an artist/album only removes database rows — it does not delete files from disk yet. The safe-delete flow (preview what would be removed, journal it, actual recycle/unlink) is designed but not built. Don't expect "Delete" to free up disk space right now.
  • Cover art has a cosmetic bug: the artwork cache always writes .jpg and serves image/jpeg regardless of what format the source image actually was. Usually harmless, occasionally shows a broken image for a non-JPEG provider result.
  • Artist photos currently reuse an embedded album cover rather than always fetching a dedicated artist photo from a provider — so an artist's "photo" can end up being whatever album cover was picked, which isn't always what you'd expect. Worth a real decision before I "fix" it, since it's a deliberate fast/local choice right now, not an oversight.
  • A few UI rough edges: some monitor/save actions can fail silently without a visible error; "Search Monitored" and "Search Upgrades" read as artist-scoped buttons but currently run against the whole library; the interactive search modal doesn't yet expose the same source-priority controls the rest of the app has.
  • Playlists are intentionally shallow right now. You can see them and trigger the existing sync/mirror pipeline, but there's no per-playlist quality profile yet, and no resolution logic for what happens when a track is wanted through a playlist and an artist you've already set a different quality profile for. I want that (track-specific override beats artist-specific beats playlist-default, most specific wins) — but it's a genuinely complicated UX problem given we already have multiple playlist-related pages, so I'm deliberately not rushing it. It's documented and parked, not forgotten.
  • Very large libraries may show their age in a couple of list/detail queries that aren't fully optimized yet (N+1 patterns in a few spots).
  • Scan/Retag hold a database write-lock slightly longer than they need to while reading files off disk — mostly invisible unless you're on a slow network mount, but worth calling out.

None of these are secret landmines — they're all tracked, and none of them touch your files or your existing (non-Library-v2) workflows. But this is genuinely an early, opt-in feature, not a "flip it and forget it" one yet.


What I'm asking for

Try it on your actual library — especially if you're on a setup I can't easily replicate myself: very large libraries, path-mapped Docker volumes, multiple download sources (Usenet + torrent + Soulseek together), multiple user profiles, non-Plex media servers. Tell me what's confusing, what breaks, and what feels like it's missing compared to how you'd expect Lidarr to behave. I'd rather find the rough edges from real libraries now than after this becomes the default.

This has been a big chunk of work and I'm genuinely excited about it — but I also know "big chunk of work I'm excited about" is exactly the kind of PR that needs the most outside eyes before it merges, not the least. So: please poke at it. Break it. Tell me it's wrong. That's what this draft is for.


I screwed up — a force-push (git history cleanup) automatically closed the old PR #1025, and GitHub won't let it be reopened since then ("branch was force-pushed or recreated"). The branch/code is unchanged, only the PR container itself is gone. Here are the comments from last time so nothing gets lost:

@Nezreka (2026-07-14):

I'm excited to test this! Would this replace the entire library page or one of the views that exist in it. Standard view was nothing special but enhanced view was more library focused for manipulation

@nick2000713 (2026-07-14):

It’s still a long way from being finished, so there’s a lot missing and a lot left to implement.

The idea is that it will eventually replace both the standard library and Library Enhanced, although it’s mainly meant as a replacement for Library Enhanced for power users.

If you notice something missing, definitely let me know. There’s still plenty I want to add.

Personal note: I actually started working on this before SoulSync expanded into movies as well. Personally, I’d rather keep focusing on music for now. I feel like the movie/TV (and even YouTube) space is already covered really well by projects like Sonarr, Radarr and TubeSync, which are, in my opinion, still superior in those areas. Because of that, I haven’t really designed or optimized Library v2 with those use cases in mind.

@Nezreka (2026-07-14):

Sounds great! I'm just messing with it now, importing my library. Excited to see how it all works. Yeah you focus on anything you want to, the entire video side is very fresh with more work to be done. Music and video side are fully disconnected though so no worries you will impact anything. I've been daily driving it replacing radarr, sonarr, kometa and ytdl-sub. takes a day or so to enrich the library for the functions to work though.

@Nezreka (2026-07-14):

I'm noticing that the import is taking quite a bit without any live status. Been about 30 min without much happening but it says it is importing. I have a 320,000 track library though.

@nick2000713 (2026-07-14):

Oh Wow, definitely never tested it at 320k tracks 😅 Good news, it's not actually stuck.. the DB import itself is fast, but right after that it silently kicks off an enrichment pass that fetches tracklists + artwork per album from Spotify/Deezer, one by one. At your scale that's tens of thousands of sequential API calls, so itjust grinds for a while.

The actual bug is that none of that shows up in the UI — it just sits on "Importing…" forever with zero
indication anything's happening, even though the backend is tracking real progress under the hood. Logged it as P2-25, will get real progress reporting wired up. For now if you tail the backend logs you should see it steadily crunching through albums.

@nick2000713 (2026-07-14):

To be completely honest, the import still needs quite a bit of work. Several features from the old library haven’t been implemented yet, and not all information that was available in the old library is displayed in Library V2 yet.

If you’d like to try it, feel free to do so, but keep in mind that the import is still a work in progress.

For example, manual artist and track matching, metadata enrichment, and the detailed match overview from the old library are still on the roadmap. Some source information is not shown in Library V2 yet, but I plan to bring all of these features back.

I’m aware that the new library hasn’t reached feature parity with the old one yet, but that’s the goal. I’ll also be revisiting the import process to make it more complete and efficient.

@Nezreka (2026-07-14):

I’m aware that the new library hasn’t reached feature parity with the old one yet, but that’s the goal. I’ll also be revisiting the import process to make it more complete and efficient.

You're killing it, man. I appreciate your work. Take your time, let me know if you ever want me to pop in and check things. Moving SoulSync away from the reliance of media servers is necessary for other things planned down the line anyway.

@nick2000713 (2026-07-16):

Hey @Nezreka, I've worked on optimizing the import pipeline. It is now parallelized (for artwork, tracklists, and tag pre-caching) and runs significantly faster. Several bugs (like track-number healing and missing ReplayGain/Lyrics features after legacy import) have also been resolved. Feel free to try it out and let me know what you think..

@nick2000713 (2026-07-18):

Hey @Nezreka,

I've worked again on the import pipeline and resolved many many bugs.

Several bugs are still known... I still have problems when the artist hasn't the right metadata or has duplicate artists, but I think I have it now somewhat under control.

I also fixed the issue where newly split component artists were automatically set to monitored/watched instead of just the respective track. Now they correctly inherit the monitored status of the parent combined artist.

I would recommend to reset everything and start the import again since many many bugs are now resolved. It's better if you reset and start fresh, maybe now on a smaller library ;)

If you want, you can execute this command in the browser. This is how you can reset it, but be aware it also removes everything on your wishlist/watchlist:

(async () => {
  console.log('Hole Liste aller Künstler...');
  let page = 1;
  let allArtistIds = [];
  
  while (true) {
    const res = await fetch(`/api/library/v2/artists?page=${page}&limit=500`).then(r => r.json());
    if (!res.success || !res.artists || res.artists.length === 0) break;
    allArtistIds.push(...res.artists.map(a => a.id));
    page++;
  }
   
  console.log(`${allArtistIds.length} Künstler gefunden. Löschvorgang gestartet...`);
  
  for (let i = 0; i < allArtistIds.length; i++) {
    const id = allArtistIds[i];
    await fetch(`/api/library/v2/artists/${id}`, { method: 'DELETE' });
    console.log(`[${i + 1}/${allArtistIds.length}] Künstler mit ID ${id} gelöscht`);
  }
   
  console.log('Bibliothek vollständig geleert! Lade Seite neu...');
  location.reload();
})();

After the import and caching is done, I would recommend to manually start the "Reconcile unmapped artists" job. It resolves collaboration names into the real component artists and several other things.

Try it out and tell me what you think, thank you!

@nick2000713 (2026-07-18):

I am now working on the diverse tools that work with the old library, etc. together. For me, every feature I want and the old library had is implemented now. It's easier for me to build solely on the new library (Library v2), so I have to change/modify nearly every tool because it depends on the old library and database tables. For me to develop, it's easier to solely concentrate on Library v2 and migrate every tool over. Therefore, with the new code, the old library will be removed so it makes it easier going forward. If something is missing for you, tell me now...

And another thing is, I have found several bugs anyway and solved them in the download pipeline. Like when you approve a track in the quarantine, it would later be detected as an orphan file because the metadata after a given time weren't present so it wasn't written in the database. Such bugs and several others are also resolved because I stumbled upon them. @Nezreka

@Nezreka (2026-07-19):

Okay, I'm starting to understand this better now that I've got it running on a test library. My big question: what happens to the artist-detail pages?

Right now those are the only thing that lets me pull up an artist's full discography for artists I don't own yet — as far as I can tell Library v2 only works on artists already in my library, so I want to understand how that "find an artist I have nothing by and start monitoring them" flow survives once the old library is gone.

I'll be honest, this is getting a little big lol — scope mostly. But I'm willing to see it through and test as much as you need but also it's early and I haven't seen the full picture yet. I just want us on the same page about what's getting replaced (a page goes away) vs. rewritten (same page, new plumbing underneath) before it gets expensive to change.

@nick2000713 (2026-07-19):

Thanks for testing it so thoroughly. And don't worry, we absolutely don't have to rush this. I completely agree that before something this big gets merged, the architecture has to be right. I'd rather take more time now than end up with something we regret later.

The reason this became much larger than I originally expected is that I started Library V2 as an opt-in feature. Because of that, I introduced separate database tables instead of replacing the existing ones. That worked fine at first, since Library V2 was just a new view sitting on top of what already existed. But once I actually started testing it against a real library, I kept running into background jobs that quietly assumed a track could only ever live in the old tables. As one example: the lyrics-fetch job resolved a track's file path through the old library tables, and once tracks could also exist purely in Library V2's own tables, that same resolver silently returned wrong paths for V2-only tracks. I fixed that, and a few days later hit the exact same class of bug independently in the ReplayGain job, because it had its own separate copy of similar path-resolution logic. And it wasn't just those two — there are 33 separate repair/maintenance tools (quality scanning, tagging, cover art, orphan detection, and so on) that all made the same old-tables assumption in one place or another. At that point I basically had two choices:

  • keep both libraries alive and write synchronization between the old and new tables for every one of those tools, or
  • migrate the tools to Library V2 directly.

I chose the second option because synchronization is much more error-prone than having a single source of truth, and because fixing the same class of bug over and over, one tool at a time, wasn't sustainable. In hindsight, that's also why this PR grew much more than I originally anticipated.

I understand your concern about the artist pages, but I don't think anything you're actually worried about is being removed — this is a wording problem, not a scope problem. To be clear about what's actually going away: it's only the old Library page itself. Search, Discover, the artist-detail page, Watchlist, and Wishlist are separate pages and none of them are being touched by this migration.

Given that, I have to admit I'm not fully sure I still see the problem — you can already search for an artist you don't own today, open their page, browse their discography, and add them to the Watchlist or download straight from there, all without that artist ever being in the library. That flow doesn't sit on the page I'm removing, so it keeps working exactly as it does now. If there's a specific part of it you're worried about beyond the wording, let me know and I'll take a closer look.

The old Library page itself is essentially replaced by Library V2, so I don't see much value in keeping two versions of the same thing around. Everything else stays as is for now.

Also, while working on this I stumbled across a surprising number of unrelated bugs in the existing pipeline. Some of those fixes probably don't even belong in this PR and could be split into separate pull requests to make this one easier to review.

On top of that, the playlist view — we can strike that entirely from scope. I'm not attached to it and it's not something I need in this PR.

@nick2000713 nick2000713 changed the title chore: squash library-overhaul onto dev for a real merge-base Library overhaul Jul 22, 2026
dev added 28 commits July 25, 2026 23:59
Restores a valid merge-base with dev so GitHub/Ultrareview can diff
this branch again. The previous filter-repo history rewrite (blob
cleanup) severed the shared ancestry with dev; content is unchanged,
only the commit history is squashed onto dev's current tip.

Reconciled with the 7 commits dev gained after the previous fix
(mostly the "13-job sweep" adding artist_id to repair-job findings).
Ported that change into the 6 repair-job files this branch still has
(acoustid_scanner, dead_file_cleaner, lossy_converter,
metadata_gap_filler, short_preview_track + its test); the other 6
touched files (album_completeness, duplicate_detector,
mbid_mismatch_detector, quality_upgrade_scanner, single_album_dedup,
test_album_mbid_consistency) are jobs already retired in
RETIRED_JOB_IDS (function moved to native Library-v2 engines), so
upstream's edit to their now-dead code doesn't apply here. Also fixed
two more test fixtures (test_acoustid_scanner.py,
test_lossy_converter_scan.py) whose row tuples needed the same
artist_id-width extension but weren't touched by the auto-merge since
they're unique to this branch.

Single parent, not a merge commit: content is fully reconciled so the
tree matches what a merge would produce, but the PR's commit list
stays at 1.
@nick2000713

Copy link
Copy Markdown
Contributor Author

@Nezreka, i worked through all your points now and honestly this became alot rougher than i expected xD.. you were right to keep asking which database we actually want to maintain in the future. keeping legacy and lib2 alive as two equal libraries makes no sense.. every import, metadata edit, enrichment result and repair job would need to be synced forever, and sooner or later both databases would disagree again.

so i stopped treating lib2 as an optional second library. lib2 is now the main Library and the source of truth for normal runtime. the old tables are only kept for the first upgrade import and as a temporary safety/rollback boundary. i want to remove them in a later update, but only after this survived another real test on your 9gb database.

i also thought alot about why not just put everything back into legacy.. but the old model simply does not fit anymore. one legacy track has one artist_id and one file_path. it cannot properly represent multiple credited artists, multiple files/qualities for one recording, single↔album relationships or mappings from several media servers. to keep legacy i would basically have to rebuild its schema and rewrite every query and worker anyway, while still carrying all those old assumptions. lib2 already models these things properly, so one controlled migration to lib2 is much safer than permanent translation between two libraries.

about the problems you found:

  • migration stuck at 5/7 · 71%: the server now starts first and the large migration continues in the background. it logs the real stage and counter every 30 seconds, including when the counter did not move, so it should finally be possible to see the difference between slow and stuck
  • all the database is locked errors: enrichment workers and the automation engine are paused while the migration owns the writer, then only the workers that were running before are resumed. the large passes commit in batches instead of one huge transaction
  • the WAL growing and getting slower: the migration checkpoints the WAL between batches, so it should not just grow from 85mb to 135mb while throughput keeps dropping
  • restart making it worse: progress is stored with a real resume checkpoint. after a restart it should continue the matching stage instead of blocking database init or starting the complete catalogue walk again
  • no enrichment in lib2: your reading of resync_entity_from_legacy was right at the time.. it existed but was not reliably wired into all worker paths. i moved the provider enrichment to lib2 instead of depending on that bridge forever. artwork, genres, bios, provider ids and provider payloads now end up on the lib2 entity
  • native-only artists getting less enrichment: migrated artists and artists created from featured credits, wishlist/watchlist or discography now use the same native lib2 enrichment path. not having a legacy row should no longer create a second class of "matched" artist with less data
  • /api/library/artists reading legacy: the active Library API/UI now read and write lib2. the old compatibility response is projected from lib2 too, so metadata edits and enrichment should not disappear depending on which endpoint reads them
  • which tables are the final ones: lib2_track_artists and lib2_track_files stay for exactly the reason you mentioned. and lib2_artists, lib2_albums and lib2_tracks are now the maintained catalogue, not temporary copies beside another active library
  • mbid_mismatch_detector: it is now an active lib2 job, not a retired job whose findings get deleted at worker startup

i added a ratchet test for this too.. normal production Library code is not allowed to add new reads or writes to the legacy artist/album/track tables. the one-time upgrade reader is counted separately, so we cannot quietly start depending on legacy again later.

the media-server behaviour is also clearer now:

  • imports always create/update the local lib2 row and do not wait for Plex/Jellyfin/Navidrome to be online
  • a media server does not create the Library anymore. after a successful scan it only adds its own recognition mapping to the existing entity
  • the Library shows a small server/check badge when a server recognised the artist/album/track
  • a failed or interrupted refresh keeps the last good mappings. they are only replaced after the server was read successfully
  • SoulSync DB metadata is not silently pushed into Plex/Jellyfin. the server benefits when SoulSync writes the actual file tags and the server scans the file again

i merged the newest dev too and checked the conflicts one by one so we keep the fixes from there. manual artwork now has a real lock in lib2 and survives provider/server refreshes, the new thumbnail cache is connected to the active lib2 page, genre enrichment scans lib2, reorganize keeps repair findings on the new path and i fixed a crash in the automatic native enrichment sweep that only showed up when starting the real app.

the final merge is f7496e8f0 and includes dev through bbf21fd18. no rebase, no force push and no replaced history. the PR is clean/mergeable now and none of our guide/issue/status markdown files are part of its diff.

if you have the time i would realy like you to test these painful cases again:

  1. upgrade the same pre-lib2 database and check that the server/UI comes up while migration is still running
  2. watch the new counters and WAL through the huge track stage
  3. restart once during migration and check that it resumes without duplicate or missing rows
  4. compare enrichment on one migrated artist and one native-only wishlist/featured artist
  5. edit metadata and artwork, then run enrichment/refresh and confirm the values still stay
  6. import while the media server is offline, then scan it later and check that the badge appears without a duplicate Library row
  7. interrupt a media-server refresh and confirm the previous mappings/badges remain
  8. have a look at reorganize findings, genre enrichment and mbid_mismatch_detector

i ran everything before pushing:

  • backend: 15307 passed, 3 skipped, 2 deselected
  • webui: 325 files / 6724 tests passed
  • browser e2e: 30 passed, 8 skipped
  • production build, ruff, format and type/lint checks passed
  • lib2 legacy/native ratchet: 17 passed

the tests are clean now, but your database is still the test that matters most because it is on a completely different scale than mine.. so pls tell me anything that looks stuck, missing, duplicated or just behaves differently. even one weird provider or badge is useful and i will dig into it..

thx again for pushing on this.. the rewrite was rough, but your question about which database we actually maintain was the right one and the architecture is alot clearer now..

dev and others added 12 commits August 15, 2026 01:17
# Conflicts:
#	core/listening_stats_worker.py
A review of the 59 migration commits since 7aacec8 asked one question of
each ported worker and read site: does it still do what the code it replaced
did? Ten places did not.

- media_mappings: the startup backfill's batch predicate selected rows
  upsert_mapping refuses to write, so one blank server_id spun the loop at
  100% CPU until shutdown. Selection and write now share the condition.
- media_server_sync: the ownership branch wrote track_number/disc_number
  outright while the mapping branch beside it used COALESCE, so every import
  stamped disc 1 over a real disc. New rows keep their default in the INSERT.
- imports: the strict lib2 gate ran before the second catalogue writer, so a
  correctly tagged and placed file whose identity the autolink could not
  derive was reported as failed and skipped the automation event, the history
  row and the wishlist check. The gate now asks the catalogue, after both
  writers.
- imports: the redownload hook guarded on the track id but repoints by path,
  and logged success when it had moved nothing.
- worker_support: provider_id_conflict read the whole artist table on every
  artist match in twelve workers. Narrowed in SQL; _ids still decides.
- worker_queue: the pending query's ORDER BY could not use an index, so every
  worker tick sorted the entire entity table for one row. Split into its two
  disjoint halves, both of which stop early.
- music_database: enrichment coverage divided by every lib2 artist including
  provider-only and discography rows, diluting all nine percentages.
- search: a natively imported track could come back with track_id null and
  record the play against an empty id.
- workers: eight _mark_status docstrings still claimed `error` never retries.
  The auto-retry is the intended contract; the docstrings now say so.
- provider_attempts: the ledger had no delete trigger, and orphans count as
  processed — the progress bar sat at a clamped 100% with work outstanding.

14 regression tests, six verified red against the previous state. Full suite:
15403 passed, 3 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9Rk5htjywarUZeqLyWUsF
The LV2-MIG-10 orphan purge ran the first time the new code started,
including when a bootstrap import was still in flight. Half the catalogue
is imported at that point, so "no entity row" does not yet mean "no
entity" — and the rows it would delete are exactly the history
`backfill_from_legacy` seeds to stop every worker re-asking every
provider about the whole library.

Trigger and purge now wait for the import to settle. Deferring costs
nothing: an import creates entities, it does not delete them, so no
orphan can appear in the window where the trigger is still missing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9Rk5htjywarUZeqLyWUsF
Second review pass over the lib2 cutover, this time every worker and tool
individually plus the upgrade import as its own question. The import itself
came back clean — data-faithful, resumable, and `backfill_from_legacy` does
stop the workers re-asking every provider after an upgrade. Five other
things did not.

- music_database: `get_all_track_ids_for_server` changed from returning
  primary keys to returning server ids but still only guarded IS NOT NULL. A
  server never reports an empty id, so one row stamped `server_id=''` was
  always stale — and detaching by server id turned that single entry into
  every blank-id track of the source. The 50% safety net could not see it:
  it counts distinct ids while the detach removes catalogue rows.
- worker_queue: legacy's tables *were* the owned library. lib2 keeps
  discography, wishlist and provider-only rows beside it, and nothing
  filtered them, so twelve workers enriched — and counted — work legacy
  never did. Same cause in the genre-enrichment repair job.
  `sql_util.owned_sql()` is now the one definition of ownership; it had been
  re-derived in four places and was simply missing from the queue.
- worker_queue: the phases were nested inside the entity types, so a fresh
  album queued behind a backlog of artist retries. Legacy ran priorities 1-3
  before 4-6; phase is now the outer loop again.
- listening_stats / get_track_by_server_id: the authoritative mapping table
  was merged with the compatibility snapshot by UNION+setdefault and by OR.
  Neither has a defined row order, so after a re-match the stale row won and
  play counts landed on the wrong track. Mapping first, snapshot for the
  gaps.

Also closes LV2-MIG-04 properly. Narrowing it in SQL was not enough: the
non-indexable json_extract collapsed the whole predicate to a table scan
(measured, not assumed). Per-service expression indexes make it a
MULTI-INDEX OR; predicate and index are generated from one function because
SQLite only uses an expression index when the text matches.

Queue fixtures had seeded catalogue rows without files, which under the
restored ownership rule correctly means nothing — `tests/lib2_ownership`
gives them the file evidence a real library has.

Full suite: 15410 passed, 3 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Q9Rk5htjywarUZeqLyWUsF
Five conflicts. Three were mechanical: the two dashboard bands kept
upstream's live-refresh behaviour and our import ordering, and
`album-meta-row.tsx` was two blank lines against a file this branch had
already deleted.

The other two were real.

`core/repair_worker.py` — upstream's fix targets
`_track_identity_for_redownload` / `_fix_quality_upgrade`, neither of which
survives here: the native scan raises `quality_below_cutoff` against
`lib2:<id>`, and lib2 rows survive the full refresh that invalidated the
legacy ids in the first place. What DOES survive is migrated pre-V2
`quality_upgrade` findings, which we were refusing outright whenever they
carried no pre-searched match. `_legacy_quality_track_data` now rebuilds the
payload from the finding's own details, reads both key vocabularies, refuses
rather than queueing "Unknown - Unknown", and gives an entity-id-less finding
a stable hashed wishlist id so two of them cannot dedupe each other away.
Upstream's test file was rewritten onto that path rather than deleted.

Reassign an album to a different artist — upstream hung it off the artist
detail page this branch removed, so it had to move, and moving it fixed a
data-loss bug on the way. The service read `tracks`/`albums` and put the
legacy track id into the hint's `replace_track_id`; on this branch
`delete_replaced_track` resolves that id against `lib2_track_files.track_id`.
Both id spaces are bare integers, so "replace the original" would have
deleted a DIFFERENT track's file, silently. The subject is now `lib2:<id>`
and a bare id is refused in preview and apply alike; the local side reads
`lib2_tracks` joined to its active files (one row per track, primary file
first) and resolves the stored path, because staging opens the file for real.
The same-release guard finally covers the source the request names instead of
only Spotify.

The UI moved with it: modal, client half and test now live under
`webui/src/routes/library/-ui/`, reached from "Reassign to another artist…"
in the album overflow menu. The `.reid-*` chassis and upstream's new
`.reassign-*` rules were already in style.css and are reused unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Found by driving the merged flow in the browser. The album overflow menu
offered "Reassign to another artist…" on a pure discography row — a release
that is in the artist's list but owns no files. The preview can only answer
"That album has no files on disk to reassign" there, so the menu was walking
the user into a guaranteed dead end, and the first thing they see is an
error toast for something that was never possible.

The entry is now disabled when the album owns no files, and its tooltip says
why rather than leaving the user to guess which of the two "Memory Reboot"
rows was the wrong one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Until now there were two ways a file left the disk. The delete dialog went
through `delete_entity_files`: preview token, root containment, an operation
row, a per-file result, crash recovery, an entry in the album's History. The
maintenance worker went through a bare `os.remove` behind its own guards —
nothing recorded anywhere, and a process that died mid-run left no state to
recover from.

That gap is survivable only while a human confirms every single deletion. It
stops being survivable the moment a job deletes unattended, which is what this
is preparing for, so it goes first rather than after.

`delete_files_journaled` is the shared primitive: same containment rule, same
journal, same statuses, same recovery, callable with a list of paths instead
of an entity preview. The execution loop is now written once and used by both
entry points. The maintenance fixes (corrupt audio, preview clips, unwanted
content, AcoustID mismatch), the orphan-file delete and the lossy converter's
"delete the lossless original" all route through it, each with its own actor
so the History can say who did it.

What deliberately did NOT change is WHICH files a job may delete. The worker
still decides containment with `fuzzy_resolved_path_is_deletable` — stricter
for a path the resolver guessed, laxer for one the catalogue names exactly —
and calls the journal with require_library_root=False. Enforcing the dialog's
stricter rule here would have silently stopped deletions for every library
whose folders were never listed in `library.music_paths`: a behaviour change
hiding inside a bookkeeping change. Three existing tests caught exactly that
and were right to.

Still on raw unlink, with reasons: `delete_replaced_track` deletes inside the
caller's write transaction, so journalling it needs the unlink hoisted out
first (a second connection there is the self-deadlock from the production
DB-lock investigation); the expired-download cleaner deletes downloads, not
library files; the empty-folder cleaner removes directories.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reported: the AcoustID checker processed everything, and Michael Jackson's
tracks still showed "Not scanned" under Check — after a refresh, and after a
re-scan. They always would have.

The Check column renders `lib2_track_files.acoustid_status`. The scanner never
wrote it. It recorded its verdict in `verification_status` and in
library_history, and left the fingerprint column NULL — so every file it
agreed with fell through to the unscanned branch. In the reporter's database
that is 164 files; the 57 rows that DO carry a status all came from the import
pipeline, never from a scan. Worse at the sharp end: a FAIL on an untagged
file moved no verification status at all, so nothing was written, and a file
whose fingerprint says it is a different recording rendered exactly like one
nothing had ever looked at.

The two columns answer different questions — "does this file stand verified"
and "what did the fingerprint check conclude" — so the scan writes both now,
independently, including `fail`. No AcoustID answer at all still writes
nothing: no verdict is not a verdict.

The files verified before this fix carry no fingerprint status and cannot
honestly be given one, so the badge falls back to `verification_status` =
verified rather than calling them unchecked, and `fail` gets a red "Mismatch"
of its own instead of hiding among the unscanned.

Second half of the report — "it should update by itself". It could not: the
library invalidated its cache on exactly one event, dispatched by Watch All.
A maintenance job that rewrites verification, retags a track or deletes a file
changes what the open page is showing and nothing told it. The signal already
existed app-wide (core.js re-broadcasts the worker's socket frames as
`ss:repair-progress` on every page); `useMaintenanceChanged` listens for a job
LEAVING the running state and invalidates then — not on every frame, which
would refetch the artist view once a second for the length of a scan.

Verified against the real library: "Scream" now shows 12 Verified where it
showed 12 Not scanned, no console errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…, and a full UI pass on track detail/History

Deletion was effectively unusable and its data stale:
- Permanent delete was blocked for any library without library.music_paths
  configured, and one already-missing file blocked deleting the rest.
  Already-gone files are now journaled as such instead of blocking anything.
- The AcoustID checker finished a full run but files still read "Not
  scanned": the scanner never wrote acoustid_status independently of
  verification_status, and the UI never invalidated its cache when a job
  completed. Both fixed; inconclusive scan outcomes (rate limits, missing
  API key) are now distinguished from a genuine no-match instead of both
  silently no-op'ing.
- A complete provider tracklist could only ever raise expected_track_count,
  never correct a stale inflated one downward (DAISIES read "1 missing"
  with nothing missing) — now safety-gated to complete, own-release-id
  fetches only.
- "Missing" counted every absent track regardless of whether the user still
  wanted it; unmonitoring a track you don't want no longer keeps it counted
  forever. Un-materialized provider-promised slots still always count.
- A track's own History tab never showed that its own file had been
  deleted (only the album/artist timelines did) — now scoped correctly.

A follow-up critical UI pass on the same modal:
- The duplicate Check/Verification columns are one column now; its
  "Skipped" verdict splits into "Skipped" (an actual bypass) vs
  "Unverified" (AcoustID ran, found nothing confident) — collapsing both
  lost exactly the distinction a reader needs.
- History gets its own tab (ahead of Metadata, Lyrics demoted), renders as
  a real table like the album/artist History, and its Status column shows
  the same colored verdict badge as the Check column instead of a bare
  "—", with the reasoning in Detail instead of a raw field-name list. The
  legacy track_downloads feed no longer duplicates an acquisition-tracked
  completion as a second "Downloaded" row, and a manual grab now says
  which automatic-gate rejections it overrode.
- Retag/Reorganize preview and Interactive Search: the checkbox/monitor/
  actions columns were `width: auto` under a fixed table layout, which
  grabs an equal share of whatever's left instead of shrinking to
  content — with few other pixel-width columns in those tables, the
  checkbox column ate roughly a third of the width. Pinned to real widths.
- disc/duration/BPM/file-size aligned consistently; the shared dialog
  close button is a thin SVG X instead of a bare "×" in an oversized box;
  the track detail modal is bigger and holds one FIXED size across every
  tab (content-sized height made it visibly jump between tabs); a release
  with everything unmonitored no longer claims "complete" — that is
  nothing being wanted, not completion.
# Conflicts:
#	core/downloads/candidates.py
#	web_server.py
@nick2000713

Copy link
Copy Markdown
Contributor Author

@Nezreka Quick update since the last message:

  • Ported Reassign to a different artist to Library V2 and tested it live in the browser.
  • Kept the branch in sync with dev — just merged your latest 7 commits (concurrent-downloads fix, post-processing queue
    fix, preferred-version matching, Navidrome path resolution, per-item quality-profile filter, radio-edit detection). All
    conflicts resolved, full test suite green after the merge.
  • Kept fixing real bugs found by actually using the app against my own library: permanent file deletion, AcoustID scan
    status not updating live, a few missing-count/history display issues.

No urgency on a reply, just wanted to keep you posted that I'm tracking what you implement upstream, and making sure it lands in Library V2 too...

dev and others added 4 commits August 20, 2026 00:03
43 upstream commits (YouTube Premium quality via cookies.txt, MusicBrainz
artist-alias resolution, collapsible sidebar, wishlist Wing-It guesses,
deezer throttle/genre/sync fixes, soulseek stall-guard, quarantine-wedge
fix, search-punctuation fix, and more).

Conflicts (8, all resolved):
- core/imports/pipeline.py: kept our journal call AND adopted upstream's
  wrapper-level quarantine retry/fail fix (real "wedged in Processing"
  bug upstream root-caused independently of our own downloads work).
- database/music_database.py: adopted upstream's fix removing a
  fabricated-filename fallback that could corrupt file_path on transient
  Subsonic/Navidrome API gaps; kept our lib2-native track upsert over
  upstream's legacy `tracks`-table INSERT/UPDATE (dead path on this
  branch).
- pyproject.toml: merged both branches' pytest markers.
- 5 modify/delete (upstream modified, we'd deliberately deleted):
  the legacy quality_upgrade test (superseded by test_lib2_upgrade_scan)
  and 4 files under the unreachable legacy artist-detail page (§46) —
  kept deleted, consistent with that standing decision.

Post-merge fixes for upstream tests that assumed the legacy
artists/albums/tracks schema instead of this branch's lib2_* catalogue
(runtime legacy SQL is 0 reads/0 writes per the ratchet test — nothing
reads those tables anymore):
- tests/imports/test_import_side_effects.py: read the new opus-bitrate
  assertion through the file's own lib2 _track_row() helper.
- tests/library/test_library_bitrate_estimate.py: reseed via
  seed_artist/seed_album/seed_track instead of raw legacy INSERTs.
- tests/test_fuzzy_search_punctuation.py: reseed both fixtures through
  ensure_library_v2_schema + the lib2 seed helpers, one album per artist
  so the search join (via albums.primary_artist_id) resolves correctly.
- database/music_database.py get_artist_full_detail: the enhanced-view
  track query joined lib2_track_files but never selected its `size`
  column, so opus bitrate estimation silently fell back to a disk
  stat() call instead of the already-fetched value (worked in
  production by luck, broke in tests against a fake path).
- webui account-tabs.test.tsx: added a test for the new
  DEEZER_PLAYLIST_PROGRESS_EVENT export (our own export-coverage guard
  caught it unmentioned).

Verified: full pnpm vitest suite green (6906 tests). Full pytest suite
green when chunked per-directory/batch (a serial full run can wedge on
a pre-existing, unrelated race in utils/async_helpers.py's shared
cross-thread event loop — reproduced identically pre- and post-merge in
isolation; not introduced by this merge, just exposed more often by the
added test volume). Three remaining failures under serial batching are
also pre-existing/artifacts of the batching itself, confirmed by
running each in isolation on both branches.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A five-agent review of `library-overhaul` produced 3,148 lines of findings
(docs/audits/library-overhaul-review-2026-08-20/). This closes all 14 backend
correctness bugs, the two Critical and both High performance findings, nine of
the ten frontend findings, and restores the two features the cutover silently
dropped. `06-remediation.md` records what was measured, what was refuted, and
what was deliberately left.

Correctness — the three that could destroy files
------------------------------------------------
* The ADR-05 delete dialog acted on a resolver GUESS. `resolve_lib2_path`
  suffix-walks the transfer folder FIRST, so a stored path that no longer
  exists (share remounted, folder renamed) resolved onto a freshly downloaded
  replacement awaiting import — and confirming the dialog unlinked that.
  `fuzzy_resolved_path_is_deletable` was written for exactly this hazard and
  applied only by the maintenance path. Now applied here too.
* `move_track_file` read `deleted` tombstones as real files, both directions:
  the target check refused a move forever with "remove or dedup it first" when
  there was nothing left to remove, and a source whose only row was a tombstone
  demonitored the track with USER provenance on the strength of a file that is
  not there.
* Per-artist repair scope was built, passed, logged — and read by nobody.
  `get_scope_file_paths`/`file_path_in_scope` had zero production callers, so
  "run Library Reorganize for this artist" moved the whole library while the API
  answered `scope_files: 180`. Twelve jobs now honour the allowlist; the rest
  declare `supports_file_scope = False` and the endpoint REFUSES a file scope
  rather than silently widening it.

Correctness — the rest
----------------------
* Delete-journal crash recovery was never wired to startup, so a container
  restart mid-delete wedged the operation in `executing` forever while the
  catalogue asserted a file that may already be gone.
* Library Reorganize looked up lib2 track ids in a legacy-keyed map (zero
  findings, one error per track) and refused every album created after the
  cutover — `legacy_album_id` is written only by the one-shot importer, so on a
  fresh install the job emitted one permanently unfixable warning per album.
* `_fix_path_mismatch` read a bare integer as a lib2 id while the sync layer
  read the same id as a legacy back-reference: one track's row repointed at
  another track's file, with a third track's history recording it.
* Every repair fix ran an UNSCOPED full-library `recompute_wanted` — "Fix All"
  over 500 findings held the single SQLite writer for hours.
* One failing `sync_repair_change` discarded the rest of the flush batch: files
  already mutated on disk, nothing rescanned, one error reported for 23 losses.
* `fix_finding` had no atomic claim, so a bulk run and a user's click could both
  execute the same fix. Claimed via a new `fix_claimed_at` column — `status`
  deliberately unchanged, because a finding vanishing from the list mid-fix
  would be the worse bug.
* Batch enrichment dropped the ownership filter and spent API budget on whole
  discographies while the UI read 0 pending / 100%.
* `prune_done` could erase the row that supersedes a stuck outbox op, re-arming
  the dd28-13 wishlist resurrection.
* `repoint_file_path` rewrote deleted tombstones too and returned a rowcount
  that lied to the one caller that reads it.

Performance (measured: 12k artists / 288k tracks / 264k files)
--------------------------------------------------------------
    list_artists(sort="albums")   11,469 ms -> 46 ms
    list_artists(search=...)      21,686 ms -> 10 ms
    list_artists(sort="tracks")   ~5,000 ms -> 47 ms
    list_artists + size roll-up      168 ms -> 44 ms
    list_cutoff_unmet          HTTP 500 above ~33k tracks -> works

Three separate defects, only the first of which the audit found:

* The count sorts were a correlated `ORDER BY` subquery run per artist row.
  Hoisting it into a CTE only reached 4.9 s, because SQLite will not build an
  automatic index on the right-hand side of a `LEFT JOIN` against a CTE — it
  scanned the 12,000-row aggregate once per artist. The same aggregate in an
  indexed table, joined on its primary key, is 3 ms. Hence
  `core/library2/artist_rollup.py`: a SORT KEY only, rebuilt in 374 ms on
  staleness or artist-count drift. The numbers rendered beside each artist
  still come from the exact per-page CTEs, and a test pins that the two agree.
* `list_artists`' size CTE had PERF-03's exact shape in the function PERF-03
  said was already fixed: `EXISTS (...)` over a bare `FROM lib2_track_files`
  scanned all 264k file rows.
* The `iss29-D04` search clause is 21.7 s at 12k artists (the audit measured
  258 ms and passed it): an `OR` across two tables plus a correlated `EXISTS`
  whose index ANALYZE cannot judge selective on a library with few aliases.

Also: `list_cutoff_unmet` materialised the whole library and passed one bind
variable per track (a hard, permanent 500 above SQLite's 32,766 default — which
is what a stock `python:3.x` image ships); the scan/retag loops opened a fresh
connection per file (~3.8 ms of schema parse each, ~19 minutes at 300k files);
`synchronous` was never set, leaving an fsync per commit; the metadata precache
shared the DOWNLOAD concurrency knob at a default of 3.

Frontend
--------
* Export Artists and Watch All Unwatched shipped on upstream/dev and were
  deleted with `library-page.tsx`. Both components survived, imported only by
  their own test — which is why CI stayed green. Rewired: "Monitor All" beside
  Automatic Search, "Export Artists…" under Files/Tools. The monitor action is
  irreversible, library-wide and one pixel from a button people click routinely,
  so it ARMS on the first click and fires on the second. On this page the label
  is Monitor; the API, tables and identifiers stay `watchlist`.
* The blocklist 409 branch was unreachable — ky threw on the status before the
  body was read, so the user got a bare HTTP status instead of being told which
  artist is blocked.
* The monitor-failure message rendered and was clipped out of the viewport by
  an ancestor's `overflow-x: hidden` (which makes BOTH axes clip).
* Ten modals: eight promised `aria-modal="true"` with no focus trap, no Escape
  and no focus restore, and two had no `role="dialog"` at all. One shared hook.
* Two globals named `_escAttr` — the HTML-escaping one won, so the onclick
  builders that needed JS-string escaping got `&Nezreka#39;`, which the attribute
  parser decoded back into a quote that killed the handler.
* `bitrate > 5000`, inlined six times, rendered every hi-res lossless file as
  "9 kbps".

Docker
------
`build_webui_vite_assets` returned "" when the bundle manifest was missing or
its entry key had drifted: the page rendered with no module script, every
vanilla feature kept working, and nothing was logged — so the entire React half
could disappear from an image with no diagnosis available. It now says so, once,
with the fix. (Verified separately that the builder stage's output is
byte-identical to a local build and contains the current UI.)

Coverage and what is left
-------------------------
Backend correctness 14/14. Performance 14/16 (PERF-09 and PERF-13 are both
structural and neither fails). Frontend 9/10 — FE-06 is left: nothing in the app
is virtualized, so a very large discography renders every row, which needs a
windowing library rather than a fix in passing. Dead code: the hazards only
(the tracked MP3 and config backup, the shadowing `docker_resolve_path`
duplicate, the hot-path no-op config reads); the rest costs nothing per run and
would bury this diff.

The `api/library_v2.py` blueprint rebuild and the `library-v2-page.tsx` split
are both correct and both days of pure motion — folding either in would make
this unreviewable. Deleting `legacy_mirror.py` states "there is no rollback
path", which is a product decision; its docstring, which asserted the opposite
of the truth, is fixed instead.

PERF-05 is recorded as REFUTED: the pragma it blamed was simply the first
statement that opens the database file, so it was charged for the schema parse
every caller pays anyway. Measurements are in 06-remediation.md.

Verification
------------
Green: library2 1,760 · repair 213 · downloads 929 · webui 6,914 (334 files),
plus tsc, ruff, oxlint and oxfmt. The perf figures come from a synthetic
12k-artist / 288k-track catalogue driven through the real query functions.

The full `pytest tests/` sweep in deterministic order (`-p no:randomly`) stalls
at 51%, inside `tests/test_bandcamp_client.py`'s threading tests. That is
PRE-EXISTING: a worktree at unmodified HEAD, same command, stalls at the same
point. The file passes in 0.29 s on its own.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It was given `.btnGhost` alone. The size comes from `.automaticSearchButton`,
which constrains the icon to 15x15 and makes the button an inline-flex row —
without it the SVG rendered at its natural size and the button was visibly
taller than the Automatic Search button it sits beside.

Same class pair as that neighbour now, so both share one vertical height. The
class name is historical; the rule itself is the generic "ghost button with a
leading icon" used in this header.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
21 upstream commits (611b64f..250f7f9): curation retention for the Expired
Download Cleaner, live download narration (Nezreka#1156), the MetaSync export endpoint,
atomic staging that works on Docker, Deezer share links, and the Nezreka#1164 sync
matching fixes.

Six conflicts, three of them substantive — all three because upstream hung a new
feature on the legacy tables this branch no longer has.

MetaSync export ported to Library v2
------------------------------------
`api_export_entities` walked `artists`/`albums`/`tracks`. The SoulID worker
writes lib2 here, so the endpoint would have returned zero rows on every
install — silently, with HTTP 200. The legacy-usage ratchet caught it: 0 -> 6
reads, all six in that one function.

Rewritten as `_EXPORT_SPECS`: one projection per entity that serves the lib2 row
under the OLD field names, because those are the wire format the sidecar and its
peers already speak. Two provider ids come from dedicated columns and the rest
from `external_ids`; `record_type` is `album_type`, a track's `year` comes from
its release. The artist name comes from `lib2_albums.primary_artist_id`, never
`lib2_track_artists` — the SoulID worker hashes the album artist, so exporting a
featured credit would describe a row under a name that derives a different key
than the `soul_id` shipped beside it.

lib2 has no `*_match_status` columns. The replacement is the rule the provider
chips already use: a stored id IS the match. Only when there is no id does
`lib2_provider_attempts` answer (`not_found`/`error`/`skipped`); never tried
reads as `pending`, the legacy vocabulary. The provider list is derived from
`match_status.SERVICES` so a new provider cannot go missing here.

lib2 ids are INTEGER, so the keyset cursor now orders numerically and is parsed
as an int on the way in — a malformed token is a 400, not "return the whole
library from row one".

Expired Download Cleaner
------------------------
Upstream's round is a data-loss fix: `play_count = None` means UNKNOWN rather
than zero, a path-suffix fallback covers the Docker/NAS namespace difference,
media-server curation signals protect a track, and anything downloaded before
the last library rebuild is permanently out of scope. Our copy of the file was a
rewrite for the lib2 delete. Resolved by taking upstream's file and putting our
`delete_origin_download` back into it.

That surfaced a real gap: our version REPORTS a lib2 failure in the return value
instead of raising, and upstream's auto-delete branch only counts `except` — so
a run would have claimed a clean sweep while nothing was deleted. It now checks
`res['error']`. The candidate query and the suffix map read
`lib2_track_files.path` + `lib2_tracks.play_count`, and `COALESCE(MAX(...), 0)`
became `MAX(...)` so "unknown" cannot read as "never played" again.

soul_id_path
------------
New column on `lib2_artists` (CREATE + migration); the worker writes it and the
legacy importer carries it across. Only `canonical` is reproducible on another
install — the album fallback depends on what this library owned at the time — so
it cannot be recomputed and has to travel with the id.

Staging guard
-------------
Upstream moved the atomic-publish staging tree to
`<transfer>/.soulsync_atomic_staging` and generalized `skip_deleted_quarantine`
into `is_internal_transfer_dir`, with a suite pinning every transfer walker.
Three entries on that list no longer walk here: `fake_lossless_detector` and
`track_number_repair` take their subjects from the catalogue or from
`filesystem_subjects`, and `quality_upgrade_scanner` is deleted (`lib2_upgrade_scan`
is DB-driven). The list now names `filesystem_subjects.py` — the one shared walk
— instead of the three. Anything staged is by definition not in the catalogue.

Fifteen failing tests, all ours, fixed here too
-----------------------------------------------
The first sweep after the merge failed 15. All 15 failed identically on the
pre-merge tip, and all three affected upstream files pass on upstream/dev — so
none of it came from the merge. It is fallout from 05f4597 (the five-agent
audit remediation), i.e. our debt, and it is closed in this commit.

* `test_repair_stale_findings` (4) — the fixture builds its own repair_findings
  DDL, and the audit added `fix_claimed_at` to production but not to the copy.
  fix_finding's claiming UPDATE raised "no such column", the outer handler
  swallowed it, and every assertion read the pre-fix row. Column added.

* `test_repair_worker_path_mismatch` (4) — the audit put `_fix_path_mismatch`
  behind `_stale_legacy_subject`: a bare integer subject is a LEGACY
  back-reference by contract (T-12), and applying it would move a file on
  behalf of a different track. The test still passed '10'/'20'/'11'/'12'. Now
  `lib2:<id>`, plus a new test pinning the refusal itself.

* `test_script_split_integrity` (1) — `_escAttr` was deduped by the frontend
  audit (FE-07) but still listed in KNOWN_CROSS_FILE_DUPES, and the test fails
  on a STALE entry by design. Moved to the resolved block with its cause.

* `test_cross_batch_dedup` (6) — the file landed as 191 new lines without the
  feature. Implemented:
    - `_find_owning_sibling` finds the task in ANOTHER batch that already owns
      this recording. Only completed/post_processing/already_owned count —
      searching/downloading is a promise, not a file, and standing down against
      a peer that later fails strands the track. Cross-batch only: a task with
      no batch is a hand-started download (a re-rip, say), and answering that
      with "you already have it" would undo a deliberate action.
    - `download_track_worker` short-circuits before searching: status
      already_owned, inherits the owner's verification/quality/path/source (a
      blank badge reads as a failure for a track that is present and fine).
    - lifecycle counts `already_owned` as finished — without it a deduped task
      held its batch in 'downloading' forever. Both completion checks now read
      one `_FINISHED_TASK_STATUSES`.
    - Identity comes from `status.download_identity` /
      `status.normalize_track_fields`, the same triple the downloads view dedups
      on, so the UI and the dedup cannot disagree about what "the same song" is.
      That normalization was inline in build_unified_downloads_response; it is a
      shared function now.

Verification
------------
Legacy ratchet back to 0 reads / 0 writes. Full sweep GREEN: 16,711 passed,
3 skipped, 0 failed. webui vitest 6946 across 336 files, oxlint type-check
clean, oxfmt clean, ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@nick2000713

Copy link
Copy Markdown
Contributor Author

@Nezreka synced with dev again (up to 250f7f9), conflicts all resolved on my side.

also ported the new work over to library v2 here, metasync export and the expired cleaner queries and soul_id_path. i'll keep doing that with every sync, it just means the PR keeps growing a bit each round.

i'll keep doing it, but the PR gets bigger the longer this goes on.. splitting it isn't really an option anymore either, the cutover is one piece since i ported every worker...da

dev and others added 3 commits August 21, 2026 21:29
…ogue

Two halves. First, everything in LIBRARY_V2_FINDINGS.md that exists because of
the Library v2 overhaul: the ten findings it caused outright, plus the lib2
halves of five findings whose dev-side fix cannot reach this branch. Each
carries its regression test. The six dev-side fixes themselves live on
`fix/dev-review-findings-2026-08-21` so they can be merged upstream first,
independently of this branch.

Second, the legacy catalogue schema goes — the last section.

== the ten overhaul findings ==

L2-003 cross-batch dedup: post_processing no longer counts as ownership (its
import gates can still fail the owner), the requesting side builds its identity
through track_info_identity like the sibling side does (collaborations were a
false negative), provider ids decide in both directions inside one namespace
with duration/disc/track backing the metadata fallback, and an owner must have
a file and the same quality profile before a task stands down against it.

L2-004 manual match: set_library_v2_match syncs lib2_provider_attempts in the
same transaction. A PUT settles the queue entry, a DELETE removes it so the
entity is selectable again without waiting out a retry window.

L2-006 watchlist import: the watchlist row's quality_profile_id is carried onto
the lib2 artist (validated, and only when duplicate rows agree) instead of
being reset to the global default.

L2-007 wishlist import: spotify_data is a provider-neutral payload. The
provider is resolved from the payload/source_info and non-Spotify ids go to
external_ids under their own namespace, with provider-keyed album/track lookup
so an existing recording is reused instead of duplicated. The namespace
sanitizer now covers lib2_tracks too.

L2-008 reset intent: the pre-wipe album-intent snapshot is committed to
lib2_import_intent_snapshot in the same transaction as the deletes, reloaded on
resume, folded into a later rebuild, and cleared only after a successful
restore.

L2-009 manual import: holds a ClaimKeepalive for the whole run like the
automatic bootstrap does, and treats mark_done()==False as a hard failure
instead of reporting done and starting artwork.

L2-012 full refresh: clear_server_data no longer stamps library_rebuilt_at.
The lib2 detach keeps the catalogue and play_count, so nothing needs the
amnesty, and stamping it grandfathered every older download permanently while
each further refresh pushed the boundary forward again. An existing stamp from
a genuinely destructive older build is still honoured.

L2-013 media-server mappings: backfill_legacy_mappings runs as an import
finalize step, so mapping-only consumers work after the first successful
upgrade instead of only after the next restart.

L2-015 repair scope: build_artist_file_scope resolves the whole alias group,
includes track-only credits and takes only active files, so a run started from
a merged artist page covers what that page shows.

L2-016 worker queue: status_counts applies the same ownership predicate as
next_pending, so provider-only rows stop showing as pending work.

== the lib2 halves the dev fix cannot reach ==

L2-005 (lib2): every enrichment worker here imports honor_stored_match from
core/library2/worker_support, not from core/enrichment, so the dev fix reaches
none of them. Same three-state contract: a stored id the provider could not
confirm is kept rather than released to a fuzzy name search, and the failure is
written to lib2_provider_attempts so the queue backs off instead of handing the
entity straight back.

L2-002 (lib2): the atomic publish repoints lib2_track_files, and
repoint_file_path already reports a rowcount precisely so a caller can tell
"the catalogue did not know this file" from "done". The lifecycle threw it
away; it is returned now, which is what lets the (dev-side) publish treat a
zero for an audio file as a failed publish.

L2-010/L2-011 (lib2): api_export_entities was rewritten for lib2 on this
branch, so the dev fix does not apply to it. Same treatment: `since` is
normalised to UTC once and compared through julianday() instead of as text, the
filter spans each row's parents, and the lib2 writers that change an exported
field (soul_id, canonical claims, and a provider-attempt status change) now
move updated_at. record_attempt touches only on a real status change, because
it runs on every provider cycle.

L2-014 (lib2): lib2_artists has its own soul_id_path column with the same gap.
The new versioned migration proves each path locally by recomputing it and
records anything it cannot prove as `unknown` rather than assuming canonical.

== the legacy catalogue schema is gone ==

Nothing has read or written `artists`/`albums`/`tracks` at runtime for a while —
`tests/library2/test_legacy_usage_ratchet.py` pins that at reads: 0, writes: 0.
What was still there was the machinery that maintained them, so it goes now. A
fresh install stops creating 3 tables / 215 columns / 85 indexes that nothing
touches; an install that already has them keeps them until
`core/library2/importer.py` has migrated its rows.

Deleted outright: `core/library2/legacy_mirror.py`, whose own docstring already
said it was not installed and had no caller, and everything in
`core/library2/enrich.py` reachable only from it — the resync, the divergence
audit, the handover bookkeeping. That module keeps the one thing the upgrade
importer reads (`_ENRICHMENT_PAYLOAD` + `enrichment_columns`) and goes 748 → 131
lines. `core/library2/native_enrich.py` — what the Enrich buttons and the
enrichment sweep actually call — is untouched.

`database/music_database.py` loses 19 migrations that existed only to grow
columns on those three tables: the per-provider `_add_*_columns` set, the
art-lock and core-media backstops, the genre-format one-shot, the match-status
backfill, and the three `CREATE TABLE`s with their indexes. Two are kept trimmed
because their non-legacy half is live — `_add_musicbrainz_cache` (the API result
cache `core.metadata` reads) and `_add_metadata_match_provenance`.

Because the tables are no longer created, a fresh install reports zero source
rows because they are ABSENT, and the importer's projection would raise on a
table that has none of its required columns. `bootstrap.run` therefore settles
an empty source into `waiting_for_source` — a state already written and checked
for, but which `mark_waiting_for_source` had no caller to set. It is pinned to
the watermark, so an upgrade whose rows do exist still runs normally.

Two bugs the deletion uncovered, both fixed here:

- `metadata_match_provenance` could not record a Library-v2 match at all. Its
  CHECK listed only the three legacy entity spellings while every lib2 writer
  and reader uses `lib2_artist`/`lib2_album`/`lib2_track`, so each write was
  rejected by the constraint and swallowed by the `except` around it — no manual
  match has ever been stored, and no chip has ever carried a `last_attempted`.
  The constraint is widened and an existing table is rebuilt once, carrying its
  rows over. (The triggers that used to fill this table automatically off the
  legacy `<provider>_match_status` columns had the same namespace split, so they
  had been writing rows nothing could read long before they were removed.)
- The public REST API served `repair_status` and `repair_last_checked` as
  permanently NULL: they were legacy `tracks` columns that nothing had written
  since the repair worker went native. `_api_project_lib2` answers them from
  `repair_findings` instead — the newest open finding's type, and when repair
  last had something to say about the row.

Frontend: the pre-V2 artist-grid island (`library-artist-card`,
`-library.helpers`, `-library.types` and their tests) had no importer outside
itself. `-library.export`, `-library.watch-all` and the two modals stay — the V2
page imports them.

Tests move with their subjects rather than being dropped: the ReplayGain,
similar-artists and JioSaavn fixtures seed lib2 rows, the verification-status
migration test points at `lib2_track_files` (where two files for one track can
be verified independently), and the stale-finding tests keep the refusal they
were written for without a legacy twin to leave untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
38 upstream commits, 250f7f9..5ae0a0c. Fifteen conflicts, and the
shape of nearly all of them is the same one: PR Nezreka#1169 landed our dev-side
findings on the LEGACY tables, and this branch already carries the lib2
twin. Those took ours. The rest needed real work.

Kept ours (lib2 twin already present)
- the eight provider workers: upstream's honor_stored_match grew
  mark_status_fn/status_column to record an UNAVAILABLE attempt;
  core.library2.worker_support already writes that to the attempt ledger.
- soulid_worker: ten hunks, all lib2_* vs legacy table names, incl. the
  new _migrate_artist_soul_id_paths (PATH_MIGRATION_VERSION stays
  'lib2_v1').
- repair_worker: upstream's L2-011 updated_at fix lands in
  _fix_unknown_artist, which this branch retired (unknown_artist_fixer is
  in RETIRED_JOB_IDS).
- MetaSync export: _EXPORT_SPECS/_EXPORT_CHANGED_AT already do what
  upstream's inline specs/effective dicts do, on lib2. Took upstream's
  `from None` on the since-parser (B904, which took ruff red on dev).

Kept both
- imports/guards: upstream's Nezreka#652 quarantine source block AND our
  acquisition/manual-grab pipeline callbacks, in that order.
- downloads/lifecycle: _FINISHED_TASK_STATUSES (ours) and
  _wake_waiting_batches (upstream's global concurrency cap).

Ported upstream's new work onto lib2
- the owned count. Upstream moved in_library off the join and onto a flag
  the sync matcher writes, keeping the join as the fallback for playlists
  nobody has synced since. The fallback arrived reading legacy tracks/
  artists; it reads lib2 here, id-first, and every hit must own an ACTIVE
  file — a provider-only discography row is known, not owned.
- get_mirrored_playlist_status_counts, the single-playlist variant, now
  reads flag-first too. It feeds the sync history entry while the batched
  one renders the card, and one playlist reporting two different
  ownership figures is worse than either being a little wrong.
- test_soulid_path_migration seeds lib2_artists/lib2_albums.
- test_atomic_publish_wiring: the DB stubs answer a rowcount, because the
  repoint runs through library2.track_files.repoint_file_path and a stub
  that cannot say how many rows moved now reads as "matched no row",
  which is a FAILED publish (L2-002). The real-sqlite rollback test seeds
  lib2 and reads lib2_track_files back.
- test_metasync_export gains upstream's minting test, on lib2.

Fixed
- atomic publish iterated staged files in os.walk order, which is the
  filesystem's business: on this box btrfs hands back 02 before 01, so a
  failure on the second track rolled back a different track than it did
  on ext4. Sorted. Upstream's own
  test_a_file_that_will_not_roll_back_keeps_its_db_row could not pass
  here until it was, and its sibling passed vacuously.
- ran oxfmt over the 42 files upstream added unformatted; npm run check
  is a PR gate here and dev pushes do not run it.

Verified: legacy-usage ratchet still 0 reads / 0 writes, 16,809 python
tests pass (2 failures, both pre-existing: test_confirmed_search_route
fails identically on the pre-merge tip, and the wishlist transient test
is the known -n 8 isolation flake and passes serially), 7,290 vitest
pass, oxfmt/oxlint clean, vite build clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…acks GET

Driven by a full production wishlist report (611 rows, 3.2.3, Navidrome with
media_server_connected=false). Every number below was reproduced from that
report before anything was changed.

Album covers (373/373 Library-v2 rows had none)
  `track_wishlist_payload` never built an `images` array, and the UI reads
  `spotify_data.album.images[0].url` — as does the import pipeline, for
  cover.jpg and embedded art. New `core/library2/wishlist_art.py` answers
  "what image URL do I hand the browser for this entity" once, in the same
  order the Library v2 pages use: the local artwork endpoint first (the
  long-term truth, already on disk, no media server involved), the provider
  CDN url after it as the stand-in for a cold build. The write path fills it
  for new rows, the read path backfills existing ones without rewriting a
  stored payload.

  Two consequences of that order, both required or the fix trades one bug for
  two: the import pipeline downloads from this list and cannot resolve a
  relative URL, so it asks for the first FETCHABLE entry instead of indexing
  `[0]`; and the wishlist UI had no "paint the CDN url while the local build
  runs" behaviour, so `WishlistCover` adds it. That is deliberately not the
  library page's `Artwork` component — importing it would pull the entire
  library page into the wishlist bundle.

Artist photos (218/218 distinct URLs 404'd in 12-94 ms)
  They were resolved by exact name match against `lib2_artists.image_url`,
  which on a media-server install is a `/rest/..` path — rebuilt into an
  authenticated Navidrome URL that nobody can load when the server is down.
  Resolution now goes through the Library-v2 identity (lib2_track_id ->
  primary artist, else a folded name_key match) and never returns a
  media-server path or a provider placeholder — not even as the stand-in, since
  a broken stand-in is worse than none. The Last.fm default star
  (129 rows / 99 artists) is now rejected centrally instead of in two
  private copies of the same hash set.

Image cache (857 rows, 602 of them pending and never served)
  `normalize_image_url` minted a fresh random Subsonic salt per call, so the
  same cover produced a new URL — and a new cache key — on every render. The
  salt is now derived from (password, path): stable per image, still rotating
  with the password, and leaking nothing the URL did not already carry.
  Separately, a `pending` row is a registration, not an image: it was written
  with `expires_at = 0` and `prune()` only deleted `expires_at > 0`, so with
  the size cap disabled nothing could ever reclaim it. Pending rows now have
  their own TTL and old zero-expiry rows drain on `last_accessed`.

Provider identity (20 album groups shared across distinct entities)
  "Memory Reboot" and "Memory Reboot (Slowed)" shared one Spotify album id
  because album titles are folded with `normalize_artist_name`, which deletes
  exactly the words that separate a variant from its original.
  `titles_are_same_release()` compares the variant signature (slowed/sped/
  remix/live + a trailing sequence number) while deliberately still folding
  edition words (remastered, deluxe) so matching coverage is unchanged.
  `set_library_v2_match()` additionally refuses a provider id another entity
  already holds; a manual match moves it instead. Recording-keyed track ids
  (MusicBrainz, Last.fm, AudioDB, Genius) may legitimately be shared and are
  exempt. Pre-existing conflicts are reported, never auto-merged.

Count vs. list (614 vs 611)
  `GET /api/wishlist/tracks` opened by deleting rows, which is wrong on its
  own terms and was the only difference between it and `/api/wishlist/stats`
  — so the gap could never be attributed. The GET is read-only now (cleanup
  still runs in the maintenance automation and the processing cycle), the
  response reports `stored_rows`/`hidden_rows`/`duplicates_found`, unreadable
  rows are logged by id, and `scripts/diagnose_wishlist_visibility.py` names
  the dropped rows per stage against a read-only copy of the database.

Upgrades are not duplicates
  343 of the 373 rows are quality upgrades of files already on disk. The
  policy is untouched; the list view now marks them so a profile change does
  not read as the wishlist duplicating the library.

Also: `/api/image-cache/<key>` reports why it failed in a header instead of
returning an indistinguishable empty 404.

Also carried in this commit: retired provenance triggers are dropped before
the provenance table is rebuilt, so the rebuild no longer fails against a
database that still has them.

Merged upstream/dev 3.2.4 into the same commit (5ae0a0c..163d1ed, 13
commits). Two of them are this fork's own PRs coming back (Nezreka#1172 publish
order + suite, Nezreka#1173 quality-profile save), so most of the overlap resolves
to "we already have it".

Eleven conflicts, none of them semantic:

  Seven are modify/delete on files this branch removed on purpose — the
  legacy artist-detail enhanced view (bb03187) and library-artist-card
  (754cbdc). Upstream's bba7c0c fixes two defects in that view: a stats
  bar and a section list that classified releases separately, and buckets
  beyond album/ep/single never rendering. Neither reaches Library v2 —
  `artist_detail()` classifies once, with `else: albums.append(...)` as the
  catch-all, so a compilation is listed AND counted, and `album_type` is
  lowercased on write (importer, materialize, provider adapters), so there
  is no strict-vs-folded comparison to disagree with. Deletions kept.

  `iter_staged_files` sorted its walk on both sides. Upstream's version is
  the better one — `_publish_order_key` reads digit runs as numbers, so a
  boxset's track 100 no longer sorts in among the ones. Took theirs whole;
  our plain `out.sort()` was the same fix, coarser.

  The reassign module and modal only conflicted because this branch moved
  them out of artist-detail and upstream reformatted them in place (oxfmt,
  8e389bd). Kept ours. The listening-history-band mock took upstream's
  optional `_album` parameter — the required one is a TS2493 in every
  `mock.calls[0][3]` and oxlint --type-check gates CI on it.

Nothing needed porting. The Prowlarr search throttle lands inside
`ProwlarrClient.search`/`search_each_indexer`, so the lib2 manual-grab and
acquisition paths inherit the pacing without a call-site change; the FLAC
oversize-art fix, the no-op sync-history record and the `sync_type` filter
all sit on tables Library v2 never replaced. The legacy-usage ratchet is
still green (14 passed, no new reads of the retired catalogue), which is the
check that would have caught an upstream feature hung on the dead tables.

Suite after the merge: 16,957 passed / 3 failed, and the same three fail on
the pre-merge tip in a worktree — test_confirmed_search_route's materialize
guard plus two candidate_store tests that hang on the shared async loop under
-n 8 and pass serially. Frontend: 7,304 tests, `npm run check` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants