feat(database): add WAL checkpoint on shutdown and startup recovery#3353
Open
Mustaqeem66 wants to merge 1 commit into
Open
feat(database): add WAL checkpoint on shutdown and startup recovery#3353Mustaqeem66 wants to merge 1 commit into
Mustaqeem66 wants to merge 1 commit into
Conversation
This fix addresses .forge.db corruption issues in ForgeCode by: 1. Startup WAL Recovery: - Checkpoints any leftover WAL from previous crashed sessions - Runs database integrity check on startup - Ensures data is recovered before new session starts 2. Auto-Checkpoint Threshold Reduced: - Changed from 1000 to 100 frames (~5MB max instead of ~50MB) - Prevents massive WAL files during long sessions 3. Async Checkpoint Method: - Added checkpoint_async() for graceful shutdown scenarios - Uses pool-based connection (async-safe) 4. Drop Checkpoint: - Checkpoints WAL when DatabasePool is dropped - Logs warnings if fails (expected on force-kill) 5. Comprehensive Tests: - test_checkpoint_method_exists - test_drop_calls_checkpoint - test_in_memory_pool_has_checkpoint - test_checkpoint_truncates_wal - test_wal_recovery_on_startup - test_async_checkpoint_method - test_autocheckpoint_threshold_reduced Fixes tailcallhq#3260 related corruption issues by preventing WAL accumulation and ensuring data integrity on startup. Co-authored-by: Mustaqeem66 <ageisnode@gmail.com>
github-actions
Bot
added
the
type: feature
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This fix addresses
.forge.dbcorruption issues that cause ForgeCode to get stuck or lose data.Changes
Startup WAL Recovery - Checkpoints any leftover WAL from previous crashed sessions and runs database integrity check on startup
Auto-Checkpoint Threshold Reduced - Changed from 1000 to 100 frames (~5MB max instead of ~50MB), preventing massive WAL files during long sessions
Async Checkpoint Method - Added
checkpoint_async()for graceful shutdown scenarios using pool-based connection (async-safe)Drop Checkpoint - Checkpoints WAL when
DatabasePoolis dropped to prevent data lossComprehensive Tests - Added 7 tests covering all new functionality
Motivation
This addresses issues similar to #3260 where:
Testing
All existing tests pass plus 7 new tests:
test_checkpoint_method_existstest_drop_calls_checkpointtest_in_memory_pool_has_checkpointtest_checkpoint_truncates_waltest_wal_recovery_on_startuptest_async_checkpoint_methodtest_autocheckpoint_threshold_reduced