Add methods to update/delete documents in LocalIndex

[vitonsky]

Currently the hermes-wasm package have a LocalIndex that does not expose any methods to manage documents.
We can add document or a few at once, but we cannot delete or update a documents.

Let's introduce some API to maintain the index content

[ppodolsky]

This is not on the table for now — update and delete cannot be easily implemented due to the write-once segment architecture. Documents are immutable once committed to a segment; true update/delete would require either segment rewriting or a deletion bitmap with merge-time compaction, both of which are significant changes to the core.

[vitonsky]

@ppodolsky this limitation in design is clear from a technical perspective, but it seems as a significant problem in terms of use the Hermes in a real applications.

Let's say we have privacy focused note taking app that uses Hermes. User imports 1k notes, we index all of that via Hermes and then search. That works very nice. Then user updates some notes, and we have to update the index, but we can't. In results of search user will not to see an updated notes.

We could re-build whole index when one note is changed but it would affect the UX, user would feel the slow response as a "freezes".

We could just add a new text in index, but it have a lot of problems

• The search strategy becomes complex. We should merge results with the same note id, since we have a few versions of a note, or we could use some additional filters like "updatedAt" with exact time of note version.
• The Hermes database still contains a previous versions of that note. In case user had some sensitive text in previous versions, the Hermes may compromise the user
• Disk space will be bloated, especially in case we would extract embeddings for text segments for each version and keep them in Hermes

Could you elaborate what exactly you mean when say it is not on the table for now.

Is it a critical misfit with the Hermes design and mission?

[ppodolsky]

Could you elaborate what exactly you mean when say it is not on the table for now.

This index is designed for an “index-once” pattern, like most segment-based search engines. Whether you take Tantivy, Elasticsearch, or any other search index, they do not support real deletion. Deletion or updating is implemented by marking the row with a tombstone, but it has the same performance caveats as the manual updatedAt approach you described above.

Every search query still hits the dead row - they’re just filtered out on a later steps of the search process - but they still take up space and processing time. And it will happen in any search engine that you could take, not just Hermes. So, any deletion there is mostly syntactic sugar.

Real deletion does happen, but only during the compaction step, which is essentially a full “rebuild” of the segment which "freezes" your index in the same way you have described it. Having this kind of syntactic sugar is nice for library users, but it doesn't provide any theoretical performance benefits. And it won't solve privacy issues, deleted rows will stay within your database until compaction happens.

[ppodolsky]

Moreover, if privacy is at the center of your product, manual lifecycle management of the database is preferred, as it is the best way to guarantee that data has truly been deleted. The compaction process is usually an internal database concept and is not exposed for management, so you cannot know for sure whether a row has been physically deleted. This is also true for classic databases like MySQL or PostgreSQL.

[vitonsky]

@ppodolsky it does make sense.

What about some API to run the compaction with no ingestion the original data?
I mean if we already have the data in Hermes, we could save a time on transferring data from original sources like databases.

Let's say we have a Hermes instance with 10k documents, and (separately, or maybe even in Hermes) we have the list of dead rows ids.

On client side we use filters, maybe built-in mustNot feature to filter out a deleted rows.

Then, according to policy defined on client side (for example once per day or every N deleted rows, or when user click some button) we want to compact the DB.

How we could implement the compaction with no transfer all data from DB into Hermes? I mean what is the best way to do it in terms of performance.

The instinct says me it would be better to do it on Hermes side because it would save us against round trips between WASM-Client-WASM. For clients who runs Hermes not in browser and operates over gigabytes/terabytes it may be even more critical.

---

I imagine such API as something like compact(ids) that clients will call and pass some ids to delete/keep, then Hermes would atomically rebuild the index in 3 steps

• build new chunks
• toggle inner state to use new data storage
• delete previous state and any intermediate files (like other unsuccessfull migrations)