Handle-Based I/O and Path Caching

[vbp1]

Summary

Handle-Based I/O, FIFO operation ordering, and materialize_on_read policy.

• Introduce handle-based pread/pwrite I/O to reduce repeated file opens
• Add sequence-based barrier mechanism (PendingOps) for FIFO write ordering
• Implement materialize_on_read policy to keep diff small for read-heavy workloads
• Fix PostgreSQL "read only 0 of 8192 bytes" error with sparse diff files
• Remove attr_cache to eliminate race conditions with async writes
• Add memory cleanup via forget() to prevent accumulation during long sessions

Key Changes

1. Handle-Based I/O (fuse.rs)

• open() creates persistent file handles stored in handles HashMap
• read()/write() use pread/pwrite via handles instead of repeated fs::read/fs::write
• For pg_datafile: handles are NOT created — reads always go through overlay.read_range() to correctly handle sparse diff files and block-level materialization

2. FIFO Operation Ordering (pending.rs)

• New PendingOps module with sequence-based barrier mechanism
• Guarantees READ sees all preceding WRITEs without waiting for concurrent ones
• Uses DashMap + parking_lot::Condvar for efficient per-inode synchronization
• wait_barrier() in open/read/getattr ensures consistency
• wait_for_preceding() in write workers ensures FIFO execution order

3. Materialize-on-Read Policy (overlay.rs)

• PBKFS_MATERIALIZE_ON_READ env var controls block materialization behavior
• Default false: reads from pg_datafile do NOT copy blocks to diff
• true: eager materialization (legacy behavior, useful for debugging)
• New read_range_nonmaterializing() path for non-materializing reads

4. Correctness Fixes

• Remove attr_cache — was invalidated BEFORE async writes completed, causing stale sizes
• Remove logical_len/diff_len caching from BlockCacheEntry — file sizes always read fresh
• Parse n_blocks/full_size from backup_content.control for accurate incremental backup lengths
• Add forget() to clean up PendingOps state when kernel forgets inode

5. Metrics & Observability (logging/mod.rs)

• OverlayIoSnapshot struct for cache/handle metrics
• log_overlay_io_metrics() emits periodic snapshots
• Track handle_hits/handle_misses and cache_hits/cache_misses

Files Changed

• src/fs/pending.rs — NEW: FIFO ordering mechanism
• src/fs/fuse.rs — Handle management, barrier integration, forget()
• src/fs/overlay.rs — materialize_on_read policy, remove size caching
• src/fs/mod.rs — Export pending module
• src/cli/mount.rs — Debug logging for layer order
• src/logging/mod.rs — New metrics types
• Cargo.toml — Add dashmap, parking_lot dependencies

Post-Implementation Notes

• Attr cache removed: Correctness > performance (~1-5μs per stat is acceptable)
• File size caching removed: Fresh fs::metadata() prevents "unexpected data beyond EOF"
• No file handles for pg_datafile: Sparse diff files require overlay.read_range() for correct block sourcing