feat(spanner): use single change stream reader via partition queue - #70
Draft
thetechnick wants to merge 2 commits into
Draft
feat(spanner): use single change stream reader via partition queue#70thetechnick wants to merge 2 commits into
thetechnick wants to merge 2 commits into
Conversation
Replace per-partition goroutine spawning with a serialised partition queue so that at most one Spanner change stream read is active at any time, keeping the process well within the 20-reader-per-stream limit. Previously, handleChildPartitionsRecord spawned a new goroutine (and a new b.client.Single().Query) for every child partition produced by a split or merge. Multiple concurrent readers of the same change stream cause the Spanner emulator to distribute writes non-deterministically across partitions, making events visible to one reader but not another. Changes in broadcaster.go: - Add partitionWork struct and buffered partitionQueue channel (cap 128). - Add runPartitionLoop: the single goroutine that dequeues and calls readChangeStream one at a time. - readChangeStream loops continuously, re-issuing the query on iterator. Done with no children (common on the emulator when no data is buffered yet). The emulator buffers writes even when no query is active, so events are never lost between re-queries. - readChangeStream returns only when sawChildren == true (partition ended due to split/merge), so runPartitionLoop can dequeue and read the children that handleChildPartitionsRecord enqueued. - handleChildPartitionsRecord enqueues child partitions into partitionQueue (without holding the lock) instead of spawning goroutines. enqueuedChildren set replaces spawnedChildren to prevent duplicate enqueues when the same token appears more than once. Changes in broadcaster_test.go: - Rename TestBroadcaster_SingleReader -> TestBroadcaster_PartitionManagement and update it to verify the queue-based bookkeeping (enqueuedChildren, pendingChildren) directly. - In TestBroadcaster_DeleteEvent and TestBroadcaster_UpdateEvent, move Subscribe before Create so the subscriber is always registered before any events are processed, eliminating a race where the CREATE could be broadcast before the subscriber existed. - Increase eventTimeout from 10s to 30s: the Spanner emulator takes up to ~20 seconds to deliver DELETE and UPDATE events through the change stream, so a 10-second timeout caused reliable test failures. Signed-off-by: Nico Schieder <nschieder@redhat.com>
|
Skipping CI for Draft Pull Request. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: thetechnick The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
…d loop Extract a synchronous discoverPartitions method that queries the root (nil-token) partition to obtain the initial set of child partition tokens before starting the reader loop. Previously, newSpannerBroadcaster seeded the partition queue with the root partition and let readChangeStream handle both discovery and steady-state reading in the same code path. If the change stream was unavailable (missing DDL, connection error), the broadcaster would silently retry with exponential backoff in the background while returning a nil error to the caller. Now the initialisation is two-phase: 1. discoverPartitions issues a short-lived root query (1s heartbeat) and collects all ChildPartitionsRecord entries. If the query fails, newSpannerBroadcaster returns the error immediately. 2. The discovered partitions are enqueued into partitionQueue and runPartitionLoop is started. If the root partition has not been split (common on production Spanner with small tables), discoverPartitions returns a single root partition work item so the reader loop reads the un-split stream directly. This follows the same two-phase pattern used by spanner-etcd's queryInitialPartitions. Signed-off-by: Nico Schieder <nschieder@redhat.com>
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace per-partition goroutine spawning with a serialised partition queue so that at most one Spanner change stream read is active at any time, keeping the process well within the 20-reader-per-stream limit.
Previously, handleChildPartitionsRecord spawned a new goroutine (and a new b.client.Single().Query) for every child partition produced by a split or merge. Multiple concurrent readers of the same change stream cause the Spanner emulator to distribute writes non-deterministically across partitions, making events visible to one reader but not another.
Changes in broadcaster.go:
Changes in broadcaster_test.go: