feat(koplugin): bulk sync annotations#84
Conversation
This reverts commit f7ec420.
We do not need to query the statistics database for book md5 values. We can just use the ones from the sidecar files, which we already have anyways.
…ionReader.cleanAnnotations
|
Hm, need to look into why the test was failing. Not sure why any of the changes I made here affected ths. |
|
The test is flaky :/ I hit this earlier this morning. |
|
@tku137 can we chat directly on another platform? Discord or something else? |
Sure, I have a Discord account. I could send you a mail with my handle if you like? Or just send me an invite. |
Yup, perfect! Shoot an email at georgi@gar.dev |
|
The current sync method also bulk syncs all books, it just doesn't bulk sync with highlights, correct? I'm wondering if we should be keeping both of the syncs - seems kinda confusing. If this implementation bulk syncs all books with annotations, we can probably keep only that and remove the old one? Also, maybe it's a nice idea to sync only the current book on suspend? So maybe it's worth implementing a "Sync current book" alongside "Sync all books with all highlights"? |
…for bulk operations There were too many requires inside functions all over the place. Also, I saw a nicer Koreader API to get doc_settings. But I encountered an error once, turns out that after rebooting KoReader, there is a chance that ReaderUI is not yet there, failing requires at module level. I used lazy loading, which seems to run just fine. Its still nicer to require once, not for every function call, ESPECIALLY during bulk sync! This prevents crashes when requiring the module outside the reader, preserves live in-memory settings during normal usage, and makes bulk annotation reads safe and non-destructive.
This is an excellent idea. I think we should have a single "Synchronize data" button in the menu that syncs statistics (only available in bulk) and annotations in bulk, while keeping a "sync statistics as usual and annotations for opened book" version in the sync-on-suspend path. I am still somehow worried about the performance of bulk-syncing annotations, but given how "bulky" a statistics sync already is and how basically instantaneous an annotations sync is, I guess even for a couple hundred books, this still is pretty fast, since we utilize the Koreader history of opened books without crawling through a huge library. |
…tistics in full sync - rename sync functions for clarity: sync() → syncCurrentBook(), bulkSync() → syncAllBooks() - merge "Bulk Sync All Books" into single "Synchronize data" menu option - fix syncAllBooks() to include full statistics sync from database (was only syncing annotations) - sync-on-suspend now explicitly use syncCurrentBook() for fast, targeted sync Manual sync now properly syncs all statistics + all annotations. Suspend sync remains fast, syncing all statistics + current book only. I am unsure about the performance impact on syncing all anotations actually is though...
GeorgeSG
left a comment
GeorgeSG
left a comment
There was a problem hiding this comment.
Thanks again, this looks very good! Added some questions, this is more for my understanding of what's going on :D
|
|
||
| if not book_md5 then | ||
| logger.warn("[KoInsight] Could not determine MD5 for current book, skipping annotations") | ||
| return annotations_by_book |
There was a problem hiding this comment.
we should return nil or {} to be sure this is hardcoded empty (just in case something somewhere pushes data to annotations_by_book.
|
We should also probably bump the plugin version |
|
Wrap up of recent decelopments:
|
GeorgeSG
left a comment
GeorgeSG
left a comment
There was a problem hiding this comment.
Not sure I won't get too lazy maintaining the changelog file, but looks good! 😅
3 participants
This adds the possibility to bulk-sync annotations. It is probably a rarely used feature, I used it to have an initial sync of all book annotations without the need to open each book individually.
Unfortunately, getting annotations is only available for the currently opened book in koreaders api, see #79. Crawling the whole library directory for sidecar files would be pretty heavy. So in this PR, I use koreaders history api, which gives a list of books opened in the past. This includes read, reading, paused and so on books. So my thinking is: for a book to have highlights, it must have been opened at one point and thus is listed in the history.
Having a list of books from koreaders history, we then can loop over all these books and sync their annotations one by one.
Several improvements have been made along the way, for example: