feat: #220 add index creation logic#221
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| logging.info(f"Creating IVF_FLAT index: {partitions} partitions, ~{estimated_total} total vectors") | ||
|
|
||
| index = vs.ingest( | ||
| index_type="IVF_FLAT", |
There was a problem hiding this comment.
out of curiosity, did you try VAMANA in the past?
https://cloud.tiledb.com/academy/structure/ai-ml/vector-search/foundation/data-model/#vamana
There was a problem hiding this comment.
Never, I have no idea what this thing is. Maybe I should learn more about it!
There was a problem hiding this comment.
not necessary now, you can make a new issue! just wondering if it might have some advantages / disadvantages compared to IVF_FLAT. but IVF_FLAT is working fine currently.
| for i in range(total_batches): | ||
| df = _read_csv_gz(f"{embeddings_prefix}/batch_{i}.csv.gz") | ||
| dfs.append(df) |
There was a problem hiding this comment.
Don't know if you need a speedup here: you may not... but if you do, multiprocessing / multithreading (I forget which) can make this kind of multi-csv-read pretty fast
Might not be worth it at all if this is not a bottleneck
There was a problem hiding this comment.
I've run into situations in the past where sequential reading of these was like 40 mins and parallel reading was like 5 mins
There was a problem hiding this comment.
oooooh nevermind, you're breaking this loop after a certain size
sjfleming
left a comment
sjfleming
left a comment
There was a problem hiding this comment.
I think it looks good. I have just a couple comments and questions. I had initially recommended some kind of parallel .csv.gz file reading, but I don't know if it actually makes sense in this case, since you're only reading a certain smaller number of cells upfront to create the index, and then you're (necessarily sequentially) ingesting more cells into the index after that.
|
3 participants
This pr introduces the script that creates vsindex. It was in cellarium-cas-workflows previously, but I wanted to move it here, so it can live outside of the pipeline and be just imported there.
Closes #220