[MySQL] Extend replication keep alive mechanism. #314
+251
−143
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…kage directly Added check for tablemap events
…tener class. Introduced a mechanism to limit the maximum size of the binlog processing queue, thus also limiting memory usage. This maximum processing queue size is configurable
# Conflicts: # modules/module-mysql/package.json
Cleaned up BinLogStream logs a bit
Simplified BinLogListener stopping mechanism
Added a few more defensive stopped checks to the binlog listener
… memory usage rather than number of events
… memory usage rather than number of events. Introduced a maximum timeout that the binlog processing queue can be paused before auto-resuming. This is to prevent the replication connection timing out.
Made SourceTable implement SourceEntityDescriptor interface
…trics instead of ignoring them. SourceTable. Moved MySQL table detail retrieval logic to utility function.
# Conflicts: # modules/module-mongodb-storage/src/storage/implementation/MongoBucketBatch.ts # modules/module-mongodb-storage/src/storage/implementation/MongoSyncBucketStorage.ts # modules/module-mongodb/src/replication/ChangeStream.ts # modules/module-mysql/package.json # modules/module-mysql/src/replication/BinLogStream.ts # modules/module-mysql/src/replication/zongji/BinLogListener.ts # modules/module-mysql/src/replication/zongji/zongji-utils.ts # modules/module-mysql/test/src/BinLogListener.test.ts # modules/module-postgres-storage/src/storage/PostgresSyncRulesStorage.ts # modules/module-postgres/src/replication/WalStream.ts # packages/service-core/src/storage/SourceTable.ts # pnpm-lock.yaml
Improved binlog table filtering Added extended type definitions for node-sql-parser package
Cleaned up MySQL tests in general and added a few new test utils
… on the mysql database yet.
TableFilter creation is now internally handled in the BinLog listener Pause/unpause binlog listening now uses the same stop start functionality used for schema change handling.
…on-keepalive-mechanism
Added tests for keepalive Moved common test functions to utils
# Conflicts: # modules/module-mysql/src/replication/BinLogStream.ts # modules/module-mysql/test/src/BinLogListener.test.ts # modules/module-mysql/test/src/util.ts
🦋 Changeset detectedLatest commit: 62090fd The changes in this PR will be included in the next version bump. This PR includes changesets to release 11 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Rentacookie
changed the title
Rentacookie
changed the title
…the LSN stayed the same.
Added a few more comments for keepalives Updated to released Zongji package
Rentacookie
changed the title
Rentacookie
commented
Aug 6, 2025
| @@ -134,7 +134,7 @@ export async function getSyncRulesStatus( | |||
| }) | |||
| ); | |||
|
|
|||
| if (live_status && status?.active && sourceConfig.type != 'mysql') { | |||
| if (live_status && status?.active) { | |||
There was a problem hiding this comment.
@rkistner I think this was the only thing to re-enable the replication lag check for mysql?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the source MySQL database is idle for long periods of time, write checkpoints in powersync can be sparse.
We use hearbeat events on the BinLog to keep the replication connection alive. But now we also latch onto these events to update the keep alive timestamp on the sync rules.