feat(delete-visibility): --diff / --deleted / --snapshot — deleted-file search + forensic tombstone read - #559
Merged
Merged
Conversation
Add `file_ref: u64` to `CompactRecord` — the NTFS File Reference `(sequence_number << 48) | frs`. The FRS (low 48 bits) is the MFT slot; the sequence number (high 16 bits) disambiguates slot reuse, so together they uniquely identify a *file incarnation*. This is the identity key the planned delete-diff needs (to tell a modified file from a recycled slot) and the FRS the forensic view needs — both first-class, hot-path consumers, so the reference is stored inline rather than in a cold sidecar. `frs` + `sequence_number` already come out of the parser (they live on `MftIndex`'s `FileRecord`) and were simply discarded; `build_compact_index` now packs them via `CompactRecord::pack_file_reference`. The USN live-create path carries FRS only (USN strips the sequence number), so a freshly-created record gets seq 0 until the next full read stamps the real generation. ADS/hardlink expansions share the base record's reference. Cost: the row grows 80 -> 88 bytes (+10% index memory, ~+38 MB per 5M files). An A/B micro-bench (5M-record scan, min-of-12 x3 on macOS) shows no measurable scan regression — sequential scan within +/-2% noise, random-access walk equal-or-better — because a simple scan is not purely bandwidth-bound at this row size. Build cost is negligible (one packed field per record in the existing bulk loop; the MFT parse dominates). Cache format bumped v11 -> v12 (the bulk `bytemuck` row-size change invalidates older caches at the header check; the daemon rebuilds).
Phase 2 of the delete-visibility slice: a pure, deterministic `diff_records(baseline, current) -> DeltaReport` that set-differences two compact record arrays by File Reference and classifies every real row as added, deleted, or modified. - **Deleted** — reference in baseline, absent from current. - **Added** — reference in current, absent from baseline. - **Modified** — same reference, changed `size` or `modified` timestamp. Keying on the full File Reference `(seq << 48) | frs` — not FRS alone — is what makes a delete-then-reuse of the same MFT slot classify correctly: a bumped sequence number yields a different reference, so the pair is a delete plus an add, never a false "modify". That case is the anchor test. Synthetic rows (`file_ref == 0`: aggregate rollups, `default()` placeholders) are excluded from both sides — a zero reference is not a unique identity, and no real file can carry it (FRS 0 is the `$MFT` metafile, already dropped from the compact index at build time). Hard links / ADS share one reference, so each of a file's names is reported at row granularity — the caller resolves each row index to a path against the correct side (deleted → baseline, added/modified → current). `DeltaReport` carries row indices, not resolved strings, because path reconstruction needs the whole index (the parent-chain walk); keeping the engine index-only makes it allocation-bounded and trivially unit-testable against synthetic index pairs. 10 tests cover add / delete / modify (size and mtime) / slot-reuse / synthetic-skip / hard-link fan-out / mixed. `diff_indexes(&DriveCompactIndex, ..)` is the thin wrapper over loaded indexes. The CLI/daemon surface (Phase 3) consumes this.
Adds the presentation layer on top of the diff engine: `resolve_delta`
runs `diff_indexes`, then walks each classified row's parent chain
(`resolve_path`) to a full `C:\…` path with its size + mtime, returning a
`ResolvedDelta { added, deleted, modified, truncated }` of `DeltaEntry`.
Adds resolve against the current index, deletes against the baseline (each
side owns its rows), and `limit` caps every class independently, flagging
`truncated` when it drops rows so a caller never mistakes a capped list for
the whole delta. This is the exact shape the daemon RPC serialises.
Two tests over minimally-built resolvable drives pin it: add/delete/modify
each resolve to the right full path with the right side's metadata (delete
keeps the baseline size, modify the current size), and the limit path caps
+ sets `truncated`.
Phase 3: wire the delete-visibility engine to a user-facing command. `uffs --diff <BASELINE> --drive <D>` loads a baseline MFT capture and diffs it against the drive's LIVE in-memory index — the deletion-visible companion to `--newer`, which is structurally blind to deletes. Runs against the hot index the daemon already serves searches from, so it needs no cold rebuild. Layering (each layer verified where it can be): - uffs-core: `resolve_delta` already resolves the delta to full paths. - uffs-client: `diff_wire` (DiffParams / DiffResultWire / DiffEntryWire) wire types + a `UffsClientSync::diff` RPC helper. Round-trip tested. - uffs-daemon: `handle_diff` (own sibling file, 800-LOC policy) + `IndexManager::diff_snapshot` — snapshot the registry for the live current side, load the baseline off-thread, run the engine, map to the wire result. `DriveNotLoaded` -> ERR_NOT_READY, baseline-load failure -> ERR_INTERNAL. The pure wire-mapping is unit tested; the live path is Windows/daemon-only (no small MFT fixture exists to drive it on the host). - uffs-cli: `--diff` command (thin client) with arg parsing, a grouped human table (deleted rows carry their last-known size — the "what did I lose" figure), `--limit N`, and `--json`. Arg parsing fully tested. The `--diff` token is disjoint from every search flag, so `uffs diff` (bare) still searches for "diff"; only `--diff` as the first token is the command.
The DeltaReport doc linked `crate::tree::resolve_path`; the item lives at `crate::search::tree::resolve_path`. Trips the rustdoc broken-intra-doc-links gate (host + CI). No code change.
…eted files) Mechanism 2 of the delete-visibility slice: surface recently-deleted files with NO baseline required. When NTFS deletes a file it clears the record's in-use flag but leaves the bytes (name, parent, timestamps) until the MFT slot is reused; forensic parsing keeps those not-in-use records, and this command reconstructs each path by walking the still-present parent chain. `uffs --deleted --mft-file <PATH> [--drive D] [--limit N] [--json]` — client-side, no daemon: reads the MFT capture (`load_raw_mft` forensic mode), forensic-parses every slot, and classifies the `is_deleted` records. Path reconstruction walks parent FRS through a whole-MFT map, so a delete still resolves through intermediate deleted directories; a parent whose slot was already reused yields a `…`-flagged partial path. Honest about its limits (in output + --help): best-effort (only deletes whose slot has not recycled are visible), the timestamp is the file's own last-write, not the deletion time, and reused-parent paths are unreliable. The classification + path resolution (`collect_tombstones` / `resolve_deleted_path`) are pure and unit-tested (deleted-only selection, resolution through a deleted parent, missing-parent → partial path, limit + truncation). Reuses `output::format_filetime_local` for wall-clock display. Live `--drive` (raw volume read via MftReader / broker) is a follow-up; the `--mft-file` path works on any offline capture today and is host-testable.
The first cut materialized every parsed record (4.7M rich `ParsedRecord`s) plus a whole-MFT frs->name map on top of the 4.8 GB raw buffer — many GB of RAM, enough to get OOM-killed on a real volume. Rewrite to keep only the *deleted* records (a small fraction) in one pass, then resolve each parent on demand from the raw buffer with a small memoized cache. Peak memory is now ~the raw MFT plus the deleted subset, not a multiple of it. Path resolution is factored behind a `lookup` closure so it stays unit-tested with a plain map (no file I/O), and one reused fixup buffer replaces the per-record allocation.
…eted files) Replaces the bespoke `--diff` output with a search over the baseline's DELETED set, so deleted files are filterable/sortable by every criterion a normal search supports — pattern, `--ext`, `--newer`/`--older`, `--min-size`, `--type`, sort, projection, and every output format (table/csv/json). The key insight: the old MFT snapshot is a full capture, so each deleted file is a complete record. The mechanism, high-reuse: - uffs-core: a `SearchFilters.deleted` filter (mirrors `malformed`) that matches the `DELETED` tombstone bit (0x8000) on `CompactRecord.flags`. - uffs-client: `SearchParams.diff_baseline` + a `--diff <BASELINE>` search flag (so `uffs --diff C_old.bin --drive C '*.txt' --newer 30d` just works); a bare diff defaults the pattern to `*` (list all deleted). - uffs-daemon: `search()` splits into a thin entry + `run_search_over(params, override_index)`. `diff_search` loads the baseline off-thread, diffs it against the live index by File Reference (`diff_indexes`), marks the vanished rows `DELETED`, wraps the baseline as a `DriveIndex`, and runs the normal pipeline over it with the deleted-only filter forced on. The registry warm-up / dispatch-accounting is skipped for the (non-shard) baseline. Setup errors (no drive / unloaded / bad baseline) map to JSON-RPC errors. `--diff` is now a search flag, not a command, so `uffs diff` (bare) still searches for "diff". Retires the old bespoke diff surface (diff_wire wire types, the `diff` RPC + client helper, the `--diff` command). The `deleted` filter, `diff_baseline` parsing (compose + bare-star default), and the DELETED-tombstone match are unit-tested; the live round-trip is Windows-only.
Closes the two capture-side gaps so the delete-visibility workflow is first-class end to end on Windows. - `uffs --snapshot --drive C --out C_baseline.bin` captures the live MFT to a baseline file (the step 1 that `--diff` needs). Thin wrapper over the same proven `MftReader::open` + `save_raw_to_file` the `uffs-mft save` diagnostic uses; `--no-compress`, `--compression-level`, `--raw` mirror it. - `uffs --deleted --drive C` now scans the LIVE volume (in addition to `--mft-file`): reads the raw MFT via the fast parallel reader (`MftReader::read_raw`) into an in-memory `RawMftData`, then runs the exact same forensic tombstone pass. `--mft-file` XOR `--drive` is required. The live MFT read is Windows + elevated; both commands return an actionable error on other platforms (and point at the `--mft-file` offline path). Arg parsing for both is host-unit-tested; the Windows I/O compiles clean under xwin clippy and reuses library primitives already shipping in `uffs-mft`.
- forensics-diagnostics.md: new "Delete Visibility (uffs CLI)" section covering the three commands, both mechanisms, and their honest limits. - cli-overview.md: --diff as a forensic filter (§2) + --snapshot / --deleted commands (§6), cross-linked to the forensic guide. - delete-visibility-snapshot-diff.md: status → implemented; note which phases shipped and what's still deferred (slice-8.6 reconciliation, baseline-retention ring). No new forensic doc created — engine/12-forensics-diagnostics.md already existed and is the right home.
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Jul 14, 2026
… deletes The compact `file_ref` is the NTFS File Reference `(sequence_number << 48) | frs`, but the parsers never set `FileRecord.sequence_number` — so `file_ref` degraded to the FRS (slot number) alone. MFT slot numbers are stable across delete-then-reuse, so `diff_indexes` found the same set of `frs` values in baseline and current and reported **0 deletions**. The `ntfs` module even noted "only consumes the FRS half; the sequence-number extraction is [not done]". Unit tests passed because they built synthetic records with `sequence_number` already set, hiding the gap. Fix: the two parsers that populate the compact index now copy the base record's header sequence number onto the record — - `process_record` (unified single-pass parser used by both the live read and the offline `.bin` load), and - `parse_record_to_index` (the USN live-update path). Cache version 12 -> 13: v12 caches were written with `file_ref == frs` (seq 0); rejecting them forces a rebuild that captures the real sequence number, so live diffs work without a manual cache clear. Regression test pins that `process_record` persists the header seq.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Full delete-visibility capability for UFFS: find what was deleted on a drive, filterable like any other search.
Two mechanisms (design:
docs/architecture/delete-visibility-snapshot-diff.md):1. Snapshot diff —
uffs --diff <BASELINE>(primary)--diffis a search flag: it diffs a baseline MFT capture against the drive's live in-memory index by NTFS File Reference(seq << 48) | frs(so a delete-then-reuse of an MFT slot is a delete + an add, never a false "modify"), marks the vanished rows, and runs the full search pipeline over the deleted set. Deleted files therefore filter/sort by every criterion a normal search has:2. Tombstone read —
uffs --deleted(no baseline)Surfaces not-in-use MFT records via forensic parsing, reconstructing each path from the surviving parent chain. Live volume or offline capture:
How
file_refstored inline onCompactRecord(row 80→88 B, cache v11→v12);uffs_core::diff::diff_indexes(File-Reference keyed); aSearchFilters.deletedfilter matching theDELETEDtombstone bit.SearchParams.diff_baseline+--diffsearch flag (bare diff defaults pattern to*).search()split into a thin entry +run_search_over(params, override_index);IndexManager::diff_searchloads the baseline off-thread, diffs vs the live index, marks the deleted rows, and runs the normal pipeline with a forced deleted-only filter (registry warm-up skipped for the non-shard baseline).--snapshot(thin wrapper over the provenMftReader::open+save_raw_to_file), live/offline--deleted(streams only the deleted records — memory-bounded),--diffrouted through the search path.Testing
file_refbench: no scan regression, +10% index memory. All host-runnable gates green: 2344 workspace tests, lint-prod, lint-tests, rustdoc, Windows/xwin clippy (the live-read paths compile clean). The engine,deletedfilter,--diffcompose/parse, and tombstone resolution are unit-tested.Not verified from CI's mac host: the live Windows reads — snapshot capture, live
--diffround-trip through the daemon, live--deleted— need a real elevated Windows box (they reuse library primitives already shipping inuffs-mft). Docs updated:engine/12-forensics-diagnostics.md,user-manual/cli-overview.md, and the design note's status.