Skip to content

Commit

Permalink
lancedb-fix-phi-2404 (#1787)
Browse files Browse the repository at this point in the history
## Description

Lancedb insert bug fix 

Fixes #1780

## Type of change

Please check the options that are relevant:

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Model update
- [ ] Infrastructure change

## Checklist

- [ ] My code follows Phidata's style guidelines and best practices
- [ ] I have performed a self-review of my code
- [ ] I have added docstrings and comments for complex logic
- [ ] My changes generate no new warnings or errors
- [ ] I have added cookbook examples for my new addition (if needed)
- [ ] I have updated requirements.txt/pyproject.toml (if needed)
- [ ] I have verified my changes in a clean environment

## Additional Notes

Include any deployment notes, performance implications, or other
relevant information:
  • Loading branch information
ysolanky authored Jan 14, 2025
1 parent 3da1084 commit 1a179cc
Showing 1 changed file with 10 additions and 5 deletions.
15 changes: 10 additions & 5 deletions phi/vectordb/lancedb/lance_db.py
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ def doc_exists(self, document: Document) -> bool:
Args:
document (Document): Document to validate
"""
if self.table:
if self.table is not None:
cleaned_content = document.content.replace("\x00", "\ufffd")
doc_id = md5(cleaned_content.encode()).hexdigest()
result = self.table.search().where(f"{self._id}='{doc_id}'").to_arrow()
Expand Down Expand Up @@ -164,11 +164,16 @@ def insert(self, documents: List[Document], filters: Optional[Dict[str, Any]] =
)
logger.debug(f"Inserted document: {document.name} ({document.meta_data})")

if self.table:
self.table.add(data)
logger.debug(f"Upsert {len(data)} documents")
else:
if self.table is None:
logger.error("Table not initialized. Please create the table first")
return

if not data:
logger.debug("No new data to insert")
return

self.table.add(data)
logger.debug(f"Inserted {len(data)} documents")

def upsert(self, documents: List[Document], filters: Optional[Dict[str, Any]] = None) -> None:
"""
Expand Down

0 comments on commit 1a179cc

Please sign in to comment.