Vector Set recall smoke tests + diskann-garnet 4.0.2 - #1990
Merged
Badrish Chandramouli (badrishc) merged 9 commits intoJul 30, 2026
Merged
Vector Set recall smoke tests + diskann-garnet 4.0.2#1990Badrish Chandramouli (badrishc) merged 9 commits into
Badrish Chandramouli (badrishc) merged 9 commits into
Conversation
Adds VectorSetRecallSmokeTests: recall-oriented smoke tests that stress a
DiskANN Vector Set graph across the physical storage configurations that
matter for disk-tiered operation:
* quantization mode (NOQUANT / Q8 / BIN)
* larger-than-memory: records spill to the storage tier during load, or
are evicted to disk after an in-memory build
* read-cache vs copy-reads-to-tail vs neither
* save -> restart -> recover, optionally re-evicting the recovered graph
The invariant under test is physical robustness: a graph that answers
queries well in memory must keep answering them well once the same records
are served from disk or recovered from a checkpoint. Each test builds a
small, well-clustered, deterministic graph (Dim=32, fixed seed, single
connection, sequential VADD), measures recall@10 against a brute-force
ground truth, applies a physical stressor, and asserts recall does not
collapse. Spill is forced and verified with DEBUG FLUSHANDEVICT plus the
store head/tail addresses. The graphs are intentionally tiny so the whole
suite runs in ~30s.
Known-failing cases: the Q8 cases that read from disk (FlushAndEvict,
larger-than-memory load, and recover) currently fail against
diskann-garnet 4.0.0. The per-dimension Q8 quantization table is native
in-memory-only state that is lost when the index is recreated after
eviction/recovery and cannot be rebuilt, so stored codes are decoded with a
missing table and recall collapses (~1.0 -> ~0.03-0.33). NOQUANT and BIN
(table-free) are robust across every configuration. These failures are
deliberate regression guards for that bug, not flaky tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recovering a checkpoint onto a "smaller box" (a much smaller log) must keep the Vector Set working correctly. Reworks RecallSurvivesSaveAndRecover so the `recoverIntoSmallerLog` case recovers into a 64k log (down from 8m) while holding the page size fixed at a valid 16k, so the recovered graph no longer fits in memory and is served from disk. Asserts the records actually spilled and that recall is preserved. Verified: NOQUANT and BIN preserve recall (1.0) when recovered into the smaller log; only Q8 collapses, which is the known quantization-table bug. (An earlier iteration recovered with the lowMemory helper, which forces a 4k page size below the 16k minimum the server enforces for Vector Sets; that is an unsupported geometry and is not what "recover to a smaller log" means.) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
VectorSessionFunctions.CopyUpdater copies the full existing value into the
new record (so the DiskANN read-modify-write callback can read it back)
before the callback runs. The new record was sized from
GetRMWModifiedFieldInfo to only WriteDesiredSize + ValueAlignmentBytes, which
is smaller than the existing value when a record is rewritten to a smaller
size. In that case oldValueAligned.CopyTo(newValueAligned) overflows the
destination and throws ArgumentException ("Destination is too short"),
crashing the server. This only occurs on the pending (disk) CopyUpdater path
— in-memory updates use InPlaceUpdater, which does not copy — so it surfaces
under heavy eviction (e.g. a small page size / large graph).
Size the destination to max(WriteDesiredSize + ValueAlignmentBytes,
existingValue.Length) so the copy can never overrun the destination,
regardless of whether the update grows or shrinks the record. The common
(growing / same-size) case is unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ecords" This reverts commit 9a71407. The fix guarded a condition that does not occur. Instrumented investigation (per-(namespace,key) write-size tracking with liveness confirmation, plus a disk-chain-walk namespace/key match check, run isolated and across the full smoke suite at 4 KB pages under CPU stress) established: - DiskANN neighbor/quant/FSM RMWs never change a record's size. Exactly two copy/inplace shapes are produced and each is perfectly constant across a build (NeighborList 68 B inline; the 8192 B overflow record). Zero size changes were ever observed for any key. - The pending (disk) CopyUpdater always matches the correct key AND namespace for DiskANN records (zero mismatches). GarnetKeyComparer's generic path compares NamespaceBytes + KeyBytes and correctly distinguishes the two namespaces, so a small-record RMW never copy-updates a larger record. Given the source value equals WriteDesiredSize + ValueAlignmentBytes for a correctly-matched, size-stable record, oldValueAligned.Length always equals newValueAligned.Length, so the CopyTo cannot overflow in normal operation. max(WriteDesiredSize + ValueAlignmentBytes, value.Length) therefore protects against nothing that happens; worse, if ValueSpan.Length were ever transiently corrupt (large), it would attempt a huge allocation (OOM) instead of failing fast. Any future "Destination is too short" here must be root-caused as a race/corruption, not sized around. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
diskann-garnet 4.0.2 fixes the Q8 quantization-table-lost-on-recreate bug: the per-dimension scalar-quantization table is now preserved (or rebuilt) when a Vector Set index is recreated after eviction/recovery, so codes read back from disk decode correctly instead of collapsing recall. With 4.0.2 all 18 VectorSetRecallSmokeTests pass on net8.0 and net10.0 (was 12 pass / 6 fail, where the 6 failures were exactly the Q8 disk cases: FlushAndEvict Q8 x2, larger-than-memory Q8, recover Q8 x2, plus the Q8 read-cache case). The Q8 disk cases now stand as regression guards rather than known-failing. Refresh the smoke-test doc comment and the Q8 assertion messages to reflect that the bug is fixed in 4.0.2 (previously 4.0.0/4.0.1). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81de0152-2aa0-47b5-bf27-f0358619d578
Keep the doc comment, the recall-budget comment, and the Q8 assertion messages precise and current-tense: state what each config guards rather than which diskann-garnet version regressed or fixed it, and remove a non-essential aside. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81de0152-2aa0-47b5-bf27-f0358619d578
Badrish Chandramouli (badrishc)
marked this pull request as ready for review
July 30, 2026 17:40
Copilot started reviewing on behalf of
Badrish Chandramouli (badrishc)
July 30, 2026 17:40
View session
Cut a new release that ships the diskann-garnet 4.0.2 upgrade (Q8 vector-set disk-recall fix). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81de0152-2aa0-47b5-bf27-f0358619d578
kevin-montrose
approved these changes
Jul 30, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
Adds recall-focused smoke tests for Vector Sets to validate DiskANN index “physical robustness” (recall stability) across eviction/spill and recovery scenarios, and bumps the diskann-garnet dependency to 4.0.2 to cover the Q8-on-disk fix described in the PR.
Changes:
- Added a new
VectorSetRecallSmokeTestssuite covering quantization modes and key physical stressors (flush+evict, larger-than-memory build, save/restart/recover). - Updated the centrally-managed NuGet version for
diskann-garnetfrom4.0.0to4.0.2.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| test/standalone/Garnet.test.vectorset/VectorSetRecallSmokeTests.cs | New deterministic recall smoke suite for DiskANN vector sets across disk-tiered configs and recovery. |
| Directory.Packages.props | Bumps diskann-garnet package version to 4.0.2. |
- TearDown no longer swallows exceptions from server.Dispose(), so a failing cleanup surfaces instead of being hidden. - Assert DEBUG FLUSHANDEVICT returns OK before relying on address-based eviction checks, so a rejected/errored command is diagnosed directly. - Use GetServers().Single() instead of [0] so an unexpected multi-endpoint multiplexer fails loudly rather than silently picking one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 81de0152-2aa0-47b5-bf27-f0358619d578
kevin-montrose
approved these changes
Jul 30, 2026
Badrish Chandramouli (badrishc)
merged commit Jul 30, 2026
96d3194
into
main
314 of 315 checks passed
Badrish Chandramouli (badrishc)
deleted the
badrishc/vector-set-recall-smoke-tests
branch
July 30, 2026 18:54
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Recall-oriented smoke tests for Garnet Vector Sets that stress a DiskANN graph across the physical storage configurations that matter for disk-tiered operation.
Smoke test coverage
Invariant
Physical robustness: a graph that answers queries well while resident in memory must keep answering them well once the same records are served from disk or recovered from a checkpoint.
Each test builds a small, well-clustered, deterministic graph (Dim=32, 16 clusters, M=16, EF=64, fixed seed, single connection, sequential
VADD), measures recall@10 against a brute-force ground truth, applies a physical stressor, and asserts recall does not collapse (MaxRecallDrop = 0.20). Spill is forced and verified withDEBUG FLUSHANDEVICT+ the store head/tail addresses. Everything is single-threaded and the graphs are intentionally tiny, so the whole suite runs in ~30s.Tests (18 cases)
RecallSurvivesFlushAndEvict(quant, {none/readcache/copytotail})RecallSurvivesLargerThanMemoryLoad(quant)RecallSurvivesSaveAndRecover(quant, recoverIntoSmallerLog)Status: all green on diskann-garnet 4.0.2
Requires diskann-garnet 4.0.2 (bumped in this PR from 4.0.0). All 18 cases pass on net8.0 and net10.0.
The Q8 cases that read from disk (FlushAndEvict, larger-than-memory load, and recover) previously failed against diskann-garnet 4.0.0/4.0.1: the per-dimension Q8 quantization table was native in-memory-only state that was lost when the index was recreated after eviction/recovery, so stored codes were decoded with a missing table and recall collapsed (~1.0 → ~0.02–0.11). 4.0.2 fixes this (the table is preserved/rebuilt on index recreate), so the Q8 disk cases now pass and stand as regression guards. NOQUANT and BIN (table-free) were always robust.
Result on
main+ diskann 4.0.2: 18 pass / 0 fail (was 12 / 6 on 4.0.0).Notes
Garnet.test.vectorsetconventions (TestBase,TestUtils.CreateGarnetServer,VADD/VSIMvia StackExchange.Redis, binary int element ids).Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com