Duplicate Items in Shared Dataset MSSPACE1B - Bug or Intended Design?

[juelinl]

Description: I noticed that the dataset shared with the SPTAG repository contains many duplicate items / vectors with identical column-wise values. This behavior raises questions about whether it is a bug. The presence of duplicates could potentially impact the performance or results when using SPTAG for nearest neighbor search tasks.

One such example is the row 258036 (0-indexed). It is replicated 33 times in the first 1M rows. See attached images for details.

Expected Behavior: Datasets used for nearest neighbor search typically do not include duplicates unless explicitly stated, as duplicates can skew the performance and evaluation metrics. If duplicates are included by design, would you kindly provide documentation or comments clarifying their purpose.

Thank you for taking time considering this!