Langchain integration

• langchain release vlite will be available under langchain-community
• set_batch to add many embeddings/texts in batches, helpful for existing data import
• optimized retrieval doing less than 1.1 seconds for retrieving 500k documents, minor bug fixes

Pure binary embeddings with MRL + hybrid PyTorch approach

Uses binary embeds with MRL, no rescoring and a new architecture for CTX files (renamed from omom), logging is rebuilt, for more stability. Removes usage of llama.cpp due to memory leaking.

vlite2: fastest retrieval vector database & new file format for context storage

1. OMOM Storage Format: vlite now utilizes the OMOM (Optimized Memory-Mapped Objects) file format for efficient storage and retrieval of embeddings and associated data. OMOM acts like a browser cookie for user embeddings, providing fast and memory-efficient storage.

2. llama.cpp Accelerated Embedding Generation: We have integrated llama.cpp to accelerate the generation of embeddings, significantly reducing the time required for indexing and retrieval operations.

3. Binary and INT8 Embedding Rescoring: vlite now supports binary and INT8 embedding rescoring, enabling the fastest retrieval in memory vector databases. This enhancement provides a substantial performance boost compared to previous versions.

4. Expanded Data Type Support: In addition to text, vlite now supports various data types, including PDF, CSV, PPTX, and webpages. This allows you to store and retrieve a wide range of data formats seamlessly.

5. Metadata Support: vlite introduces metadata support, enabling you to associate additional information with your stored items. Metadata provides flexibility in organizing and filtering your data based on specific criteria.

6. Chunking and Fast Chunking: We have implemented chunking and fast chunking options to efficiently handle large texts. This feature optimizes memory usage and improves performance when dealing with extensive datasets.

7. PDF OCR Support: vlite now comes with built-in PDF OCR support, allowing you to extract text from scanned PDFs. This feature enhances the versatility and usability of vlite in various scenarios.

8. Performance Improvement: vlite 0.2.0 delivers exceptional performance, with indexing speeds up to 77.95% faster than Chroma, a popular vector database. This substantial improvement enables you to process and store large volumes of data efficiently.

Full Changelog: v0.2.0...v0.1.0

v1: simple vector db built with numpy

v0.1.0

Stop tracking tests/notebook.ipynb