Summary
The full rebuild ingestion pipeline (kubeflow-pipeline.py) correctly inserts new documents and refreshes vectors for documents that still exist in the source repository. However, it does not remove vectors for documentation files that have been deleted or renamed.
As a result, Milvus can accumulate orphaned vectors that no longer correspond to any document in the GitHub repository.
Where the issue occurs
The issue occurs in the Full rebuild pipeline (kubeflow-pipeline.py), specifically during the Milvus storage stage.
During a full rebuild, the pipeline:
- Downloads the current documentation from GitHub.
- Chunks and embeds the documents.
- Deletes old chunks only for files that are present in the current ingestion run.
- Inserts the newly generated chunks.
The pipeline never checks whether documents that existed in previous runs have disappeared from the source repository.
Current behavior
Suppose the repository initially contains:
content/en/docs/
├── installation.md
├── architecture.md
└── deployment.md
After running the pipeline, Milvus contains vectors for all three files.
Later, the repository changes to:
content/en/docs/
├── installation.md
└── deployment.md
architecture.md has been deleted.
When the full rebuild pipeline runs again:
installation.md is refreshed.deployment.md is refreshed.architecture.md is never processed because it no longer exists in GitHub.
Since the delete logic only removes chunks for files that are currently being ingested, the vectors for architecture.md remain in Milvus indefinitely.
The same issue occurs when a file is renamed:
architecture.md
↓
system-architecture.md
The new file is inserted, but the vectors belonging to the old filename remain in the collection.
Expected behavior
During a full rebuild, Milvus should exactly reflect the current state of the indexed GitHub directory.
Before inserting new vectors, the pipeline should:
- Collect the files currently stored in Milvus for the indexed repository/directory.
- Collect the files produced by the current ingestion run.
- Compute the difference:
existing_files_in_milvus
−
current_files_from_github
- Delete vectors belonging to those orphaned files.
- Continue with the normal refresh/insert process.
This ensures that additions, updates, deletions, and renames are all handled correctly.
Why this matters
Leaving orphaned vectors in Milvus can result in:
- Retrieval results for documentation that no longer exists.
- Duplicate search results after file renames.
- Gradual accumulation of stale embeddings over multiple full rebuilds.
- An index that no longer reflects the actual GitHub repository.
Proposed solution
Add a reconciliation step during the Milvus storage stage of the full rebuild pipeline.
The reconciliation should compare the current GitHub snapshot with the existing indexed files in Milvus, remove vectors belonging to files that no longer exist, and then continue with the normal update/insert workflow.
This keeps the Milvus collection synchronized with the source repository while preserving the existing ingestion flow.
Validation
I reproduced this locally by running the full ingestion pipeline against a test repository and verified the following scenarios:
- ✅ New files are inserted.
- ✅ Existing files are refreshed.
- ✅ Deleted files are removed from Milvus.
- ✅ Renamed files no longer leave stale vectors behind.
A pull request implementing this behavior has been opened alongside this issue.
Summary
The full rebuild ingestion pipeline (
kubeflow-pipeline.py) correctly inserts new documents and refreshes vectors for documents that still exist in the source repository. However, it does not remove vectors for documentation files that have been deleted or renamed.As a result, Milvus can accumulate orphaned vectors that no longer correspond to any document in the GitHub repository.
Where the issue occurs
The issue occurs in the Full rebuild pipeline (
kubeflow-pipeline.py), specifically during the Milvus storage stage.During a full rebuild, the pipeline:
The pipeline never checks whether documents that existed in previous runs have disappeared from the source repository.
Current behavior
Suppose the repository initially contains:
After running the pipeline, Milvus contains vectors for all three files.
Later, the repository changes to:
architecture.mdhas been deleted.When the full rebuild pipeline runs again:
installation.mdis refreshed.deployment.mdis refreshed.architecture.mdis never processed because it no longer exists in GitHub.Since the delete logic only removes chunks for files that are currently being ingested, the vectors for
architecture.mdremain in Milvus indefinitely.The same issue occurs when a file is renamed:
The new file is inserted, but the vectors belonging to the old filename remain in the collection.
Expected behavior
During a full rebuild, Milvus should exactly reflect the current state of the indexed GitHub directory.
Before inserting new vectors, the pipeline should:
This ensures that additions, updates, deletions, and renames are all handled correctly.
Why this matters
Leaving orphaned vectors in Milvus can result in:
Proposed solution
Add a reconciliation step during the Milvus storage stage of the full rebuild pipeline.
The reconciliation should compare the current GitHub snapshot with the existing indexed files in Milvus, remove vectors belonging to files that no longer exist, and then continue with the normal update/insert workflow.
This keeps the Milvus collection synchronized with the source repository while preserving the existing ingestion flow.
Validation
I reproduced this locally by running the full ingestion pipeline against a test repository and verified the following scenarios:
A pull request implementing this behavior has been opened alongside this issue.