Skip to content

using cagraHnswLayers > 1 and IVF-PQ with cagraIntermediateGraphDegree >= 128 results in extra slow performance #163

Description

@nvzm123

Reproduction steps:

  1. run SearchScale benchmark script on the latest 26.08.0 main branches of cuvs and cuvs-lucene
  2. Need to use a dataset with at least 5M vectors in order to trigger IVF-PQ
  3. Need to use a dataset with a distribution that can handle setting cagraHnswLayers above 1 (e.g. 3)
  4. run exactly two tests where for one of them you set cagraIntermediateGraphDegree=126 and another one where you set cagraIntermediateGraphDegree=128
  5. Observe the results

Results:
cagraGraphDegree = 66, cagraIntermediateGraphDegree = 126:
Total Index Build Time: 580 seconds

cagraGraphDegree = 66, cagraIntermediateGraphDegree = 128:
Total Index Build Time: 1058 seconds

Note:
Reproducing this may require the PR to enable out-of-core index-building #141
Alternatively, if your dataset can fit on your GPU, then this is reproducible on multiple GPUs (e.g. I have reproduced it on both an A10-G and an L40S)

Please see attachment for detailed metrics:

metrics_ySiC8I_GPU__10M-1536d__1-seg_3-hnsw-layers__corner-case-128.csv

Note: I do not see this to be reproducible when using NN_DESCENT or when setting cagraHnswLayers=1. This is an issue because cuVS limits us to cagraIntermediateGraphDegree <= 512, and sometimes we may need to set cagraHnswLayers > 1 to achieve sought recall. This is 100% reproducible

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions