-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Refactor uploading system deduplication using hash #320
base: main
Are you sure you want to change the base?
Conversation
…ture/uploading-system
TODO: troubleshoot i.e. updating the
|
Solution: |
Problem
What is the problem this work solves, including
closes #316
Solution
What I/we did to solve this problem
used
hashlib
to generate a unique hashes for each doc (any changes will produce a new hash)removed the
should_write
method from each doc classupdated unit tests
with @ascibisz
introduced new field
dedup_hash
Type of change
Please delete options that are not relevant.
Steps to Verify:
pytest