Added code for Pageview Update with Single Transaction #810
Conversation
…unction - Added explicit transaction handling using wp10db.begin() to ensure all updates are treated as a single unit. - Implemented error handling with rollback to maintain data integrity in case of errors during the update process. - Introduced a batch commit strategy to commit changes after processing a specified number of rows (commit_after) to optimize performance and reduce overhead. - Updated logging to reflect the number of committed updates.
- Atomic Update: The transaction is started with wp10db.begin() before processing any rows and committed only once after the loop completes. This ensures that the entire update is applied as a single atomic unit. - Error Handling & Rollback: If an error occurs during processing, the except block rolls back the entire transaction, maintaining data integrity. - Removal of Batch Commits: The previous batch commit strategy (committing after every 50,000 rows) has been removed. As noted by @audiodude, committing in batches would break the requirement to update the entire dataset in one transaction.
|
Hey @audiodude I restored the 2-space indentation. Really sorry for the delay. |
There was a problem hiding this comment.
This looks good, but please install the pre-commit hooks (https://github.com/openzim/wp1?tab=readme-ov-file#pre-commit-hooks) or otherwise run yapf to fix the spacing. I see lots of unrelated whitespace changes in this PR based on what looks like manually reducing spacing to two spaces even when the formatter inserts more.
EDIT: fixed wrong link
|
Hi @harsh-0409: I'm going to close this PR in favor of #815. No offense to you, this is a decent approach, but I feel the other PR is more sound overall. As for GSoC, I consider that you are still eligible having gone through this process, even though your PR wasn't merged. To be clear, you have submitted (for review) a well thought out PR that solves a specific bug, and provided good and thoughtful communication. I consider the Kiwix PR requirement satisfied and will advocate for you as such. EDIT: Remember, GSoC participants are selected based on the strength of their project proposals, not their success in merging PRs in the pre-period. |
|
Hi @audiodude : Regarding GSoC, I’m particularly interested in the idea "Project B, Scheduled selections/ZIMs" from the idea list. I’d love to explore this further—what’s the best way to get in touch with you to discuss it? |
Mention me in the Kiwix #gsoc channel on Slack and we can discuss there. |
Single Commit:
Remove the batch commit strategy (i.e., committing after every 50,000 rows) and instead commit only once after processing all rows. This guarantees that either all pageview updates are applied or none are, maintaining data integrity.