fix(rag-worker): use deterministic embedding point ids to avoid orphaned duplicate vectors

[serhiizghama]

Problem

Closes #603.

The RAG worker writes embeddings to Qdrant and then updates the document record in MongoDB as two separate network calls. Point IDs are generated with crypto.randomUUID() on every run:

const points = embeddings.map((embedding, i) => ({
    id: crypto.randomUUID(),
    // ...
}));
await qdrantClient.upsert("embeddings", { points });
await dataSourceDocsRepository.updateByVersion(/* ... */);

If the Qdrant upsert succeeds but a later step fails (MongoDB timeout, connection drop, version conflict), the worker throws and the document becomes eligible for retry. On retry, fresh random UUIDs are generated, so a new set of points is inserted while the originals remain in the collection — the app no longer holds any reference to them. Over time this accumulates orphaned duplicate vectors that:

• return duplicate chunks during vector search,
• waste context space and degrade retrieval quality,
• grow unbounded with no tracking.

The same pattern is present in all three pipelines: runProcessFilePipeline, runScrapePipeline, and runProcessTextPipeline.

Solution

Derive each point ID deterministically from the document id and chunk index instead of generating a random UUID per run:

id: embeddingPointId(doc.id, i),

embeddingPointId() produces an RFC 4122 v5 (name-based) UUID from ${docId}-chunk-${chunkIndex} under a fixed namespace. Because the same chunk always maps to the same id, a retry upserts over the existing points rather than inserting duplicates — the operation becomes idempotent. The return value is a valid UUID, which EmbeddingRecord.id (z.string().uuid()) requires.

The helper is dependency-free (uses the already-imported Node crypto module), so no new packages are added. Deletion (runDeletionPipeline) filters by the docId payload field rather than by point id, so it is unaffected.

Testing

The repo has no test harness, so I verified the helper's behavior in isolation:

• Deterministic — embeddingPointId("doc123", 0) returns the same id across calls.
• Unique per chunk / per doc — different chunk index or doc id yields a different id.
• Valid v5 UUID — output matches ^[0-9a-f]{8}-[0-9a-f]{4}-5[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$, satisfying EmbeddingRecord.

Re-running any pipeline for the same document now overwrites the existing points in Qdrant instead of adding a duplicate set.