1. why disable stats? the stats task shoulnd't not be disabled
2. why we set segment size to that large? We recommend segment size to be no more than 8G
3. no compaction will be triggered if segment index is not done
4. need more logs for details.

also 128G memory may not be enough for hosting 50M 768 dim data.
if you need more help please contact us [email protected]

/assign @gland1

1. why disable stats? the stats task shoulnd't not be disabled
2. why we set segment size to that large? We recommend segment size to be no more than 8G
3. no compaction will be triggered if segment index is not done
4. need more logs for details.

also 128G memory may not be enough for hosting 50M 768 dim data. if you need more help please contact us [email protected]

1. we disable stats because when importing a large dataset it takes much longer. we do all search by vectors so we dont see why we need stats.
2. We work with large dataset (largest 1B). with 8G segments using DISKANN we get sub 100 QPS.so we need large segments
3. I thought compaction is about segments not indices. why wait for indexing on small segments , only to destroy them and create final indices on the larger segments?
4. I'll attach logs.

We want to import a dataset , compact it to desired number of segments and only then create an index.
This used to work in 2.4

0.log.gz
Attaching datacoord logs

/assign @czs007
please help to explain the reason that Milvus does not do compaction if the segment is not built index.

1. why disable stats? the stats task shoulnd't not be disabled
2. why we set segment size to that large? We recommend segment size to be no more than 8G
3. no compaction will be triggered if segment index is not done
4. need more logs for details.

also 128G memory may not be enough for hosting 50M 768 dim data. if you need more help please contact us [email protected]

1. we disable stats because when importing a large dataset it takes much longer. we do all search by vectors so we dont see why we need stats.
2. We work with large dataset (largest 1B). with 8G segments using DISKANN we get sub 100 QPS.so we need large segments
3. I thought compaction is about segments not indices. why wait for indexing on small segments , only to destroy them and create final indices on the larger segments?
4. I'll attach logs.

We want to import a dataset , compact it to desired number of segments and only then create an index. This used to work in 2.4

2. large diskann index build is fairly slow, could be even more than 1 hour.
3. compaction need to wait on index build done., this is the current rule. The reason is to avoid we do cascade compaction and no segment with index can be loaded, cause slow search.

you can tune dataCoord.compaction.indexBasedCompaction to be false if you just do one time ingestion.

indexBasedCompaction

Thanks I can confirm this solves the issue