Make index/centroids updates crash-safe and expose update progress in health

[DeeJayPee]

This PR fixes a production failure mode where next-plaid-api could crash with SIGBUS during a query while a bulk update was rewriting index files.

The root issue was that critical files such as centroids.npy were rewritten in place with File::create(...), which truncates the file immediately. If another request was reading the old mmap-backed file at the same time, Linux could raise SIGBUS; if the process died before the rewrite completed, the index was left with a zero-byte centroids.npy, causing restart failures like:

Index load failed: NPY file too small

What Changed

• Writes critical index files through atomic temp-file writes followed by rename.
• Covers centroids.npy, ivf_lengths.npy, metadata.json, cluster_threshold.npy, IVF files, chunk files, and index creation/update paths.
• Keeps the previous loaded index available to readers until the updated index has been written and successfully reloaded.
• Improves NPY load errors to include the file path and observed byte size.
• Adds in-memory update progress tracking.
• Extends /health with an updates field showing active and recent update status.
• Tracks stages such as queued, encoding, batching, centroid_expansion, kmeans, index_write, metadata_write, reload, complete, and failed.

Example /health Output

{
  "status": "healthy",
  "version": "1.3.1",
  "loaded_indices": 1,
  "index_dir": "/var/lib/next-plaid/indices",
  "memory_usage_bytes": 2312192000,
  "indices": [],
  "updates": [
    {
      "index": "my_test_index",
      "status": "running",
      "stage": "centroid_expansion",
      "queued_documents": 135500,
      "processed_documents": 500,
      "started_at": "2026-05-22T00:01:27Z",
      "updated_at": "2026-05-22T00:05:21Z",
      "elapsed_ms": 235491,
      "message": "finding embeddings outside existing centroids"
    }
  ],
  "model": null
}

Validation

• cargo check -p next-plaid-api --locked
• cargo test -p next-plaid atomic_write_failure_preserves_original_file --locked
• cargo test -p next-plaid-api health --locked
• cargo test -p next-plaid-api update_health_status_tracks_active_and_completed_updates --locked

[raphaelsty]

Thanks for the MR @DeeJayPee! Really clean read on the SIGBUS root cause, and the atomic_write_file mechanism is implemented carefully (same parent directory for the temp file, create_new to dodge the TOCTOU race, sync_all on both the file and the parent dir, cleanup on failure). The /health progress will save real time the next time someone files an indexer ticket. Left a few inline thoughts, nothing blocking apart from the Windows path.

[raphaelsty]

Worth keeping the upfront unload on Windows. On POSIX the atomic rename works because the old mmap holds the unlinked inode, but on Windows fs::rename cannot replace a file while it is currently mapped (returns ERROR_SHARING_VIOLATION). A cfg gate keeps the new behaviour on Linux and macOS while preserving the previous Windows workaround:

#[cfg(target_os = "windows")]
state_clone.unload_index(&name_inner);

Same pattern would help in process_batch if Windows is still a target.

[raphaelsty]

If operation() panics, the thread_local stays populated. Since tokio reuses spawn_blocking workers, the next update on this thread inherits a stale callback pointing into a freed closure. A small RAII guard makes it panic safe:

struct ProgressGuard;
impl Drop for ProgressGuard {
    fn drop(&mut self) {
        UPDATE_PROGRESS.with(|slot| *slot.borrow_mut() = None);
    }
}

Then let _g = ProgressGuard; before let result = operation();.

[raphaelsty]

Heads up: this takes a write lock on every /health call to prune. With monitoring plus load balancer polling, that contends with update workers writing progress. Two cheap options: try_write and skip pruning when contended, or fold pruning into the record_update_* paths so reads only need a read lock.

[DeeJayPee]

Good catches, agreed on all three. I dont use Windows... I’ll keep the POSIX behavior to preserve mmap readers during atomic rename, restore the upfront unload behind #[cfg(target_os = "windows")] in both update paths, make the thread-local progress callback panic-safe with an RAII guard, and move pruning out of /health reads so health only takes a read lock.

[raphaelsty]

Hi @DeeJayPee thank you for this MR, merging it :)