Skip to content

feat(spanner): use single change stream reader via partition queue - #70

Draft
thetechnick wants to merge 2 commits into
openshift-online:mainfrom
thetechnick:spanner-reader-refactoring
Draft

feat(spanner): use single change stream reader via partition queue#70
thetechnick wants to merge 2 commits into
openshift-online:mainfrom
thetechnick:spanner-reader-refactoring

Conversation

@thetechnick

Copy link
Copy Markdown
Contributor

Replace per-partition goroutine spawning with a serialised partition queue so that at most one Spanner change stream read is active at any time, keeping the process well within the 20-reader-per-stream limit.

Previously, handleChildPartitionsRecord spawned a new goroutine (and a new b.client.Single().Query) for every child partition produced by a split or merge. Multiple concurrent readers of the same change stream cause the Spanner emulator to distribute writes non-deterministically across partitions, making events visible to one reader but not another.

Changes in broadcaster.go:

  • Add partitionWork struct and buffered partitionQueue channel (cap 128).
  • Add runPartitionLoop: the single goroutine that dequeues and calls readChangeStream one at a time.
  • readChangeStream loops continuously, re-issuing the query on iterator. Done with no children (common on the emulator when no data is buffered yet). The emulator buffers writes even when no query is active, so events are never lost between re-queries.
  • readChangeStream returns only when sawChildren == true (partition ended due to split/merge), so runPartitionLoop can dequeue and read the children that handleChildPartitionsRecord enqueued.
  • handleChildPartitionsRecord enqueues child partitions into partitionQueue (without holding the lock) instead of spawning goroutines. enqueuedChildren set replaces spawnedChildren to prevent duplicate enqueues when the same token appears more than once.

Changes in broadcaster_test.go:

  • Rename TestBroadcaster_SingleReader -> TestBroadcaster_PartitionManagement and update it to verify the queue-based bookkeeping (enqueuedChildren, pendingChildren) directly.
  • In TestBroadcaster_DeleteEvent and TestBroadcaster_UpdateEvent, move Subscribe before Create so the subscriber is always registered before any events are processed, eliminating a race where the CREATE could be broadcast before the subscriber existed.
  • Increase eventTimeout from 10s to 30s: the Spanner emulator takes up to ~20 seconds to deliver DELETE and UPDATE events through the change stream, so a 10-second timeout caused reliable test failures.

Replace per-partition goroutine spawning with a serialised partition
queue so that at most one Spanner change stream read is active at any
time, keeping the process well within the 20-reader-per-stream limit.

Previously, handleChildPartitionsRecord spawned a new goroutine (and a
new b.client.Single().Query) for every child partition produced by a
split or merge. Multiple concurrent readers of the same change stream
cause the Spanner emulator to distribute writes non-deterministically
across partitions, making events visible to one reader but not another.

Changes in broadcaster.go:
- Add partitionWork struct and buffered partitionQueue channel (cap 128).
- Add runPartitionLoop: the single goroutine that dequeues and calls
  readChangeStream one at a time.
- readChangeStream loops continuously, re-issuing the query on iterator.
  Done with no children (common on the emulator when no data is buffered
  yet). The emulator buffers writes even when no query is active, so
  events are never lost between re-queries.
- readChangeStream returns only when sawChildren == true (partition
  ended due to split/merge), so runPartitionLoop can dequeue and read
  the children that handleChildPartitionsRecord enqueued.
- handleChildPartitionsRecord enqueues child partitions into
  partitionQueue (without holding the lock) instead of spawning
  goroutines. enqueuedChildren set replaces spawnedChildren to prevent
  duplicate enqueues when the same token appears more than once.

Changes in broadcaster_test.go:
- Rename TestBroadcaster_SingleReader -> TestBroadcaster_PartitionManagement
  and update it to verify the queue-based bookkeeping (enqueuedChildren,
  pendingChildren) directly.
- In TestBroadcaster_DeleteEvent and TestBroadcaster_UpdateEvent, move
  Subscribe before Create so the subscriber is always registered before
  any events are processed, eliminating a race where the CREATE could be
  broadcast before the subscriber existed.
- Increase eventTimeout from 10s to 30s: the Spanner emulator takes up
  to ~20 seconds to deliver DELETE and UPDATE events through the change
  stream, so a 10-second timeout caused reliable test failures.

Signed-off-by: Nico Schieder <nschieder@redhat.com>
@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 11, 2026
@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: thetechnick

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 11, 2026
…d loop

Extract a synchronous discoverPartitions method that queries the root
(nil-token) partition to obtain the initial set of child partition
tokens before starting the reader loop.

Previously, newSpannerBroadcaster seeded the partition queue with the
root partition and let readChangeStream handle both discovery and
steady-state reading in the same code path. If the change stream was
unavailable (missing DDL, connection error), the broadcaster would
silently retry with exponential backoff in the background while
returning a nil error to the caller.

Now the initialisation is two-phase:

1. discoverPartitions issues a short-lived root query (1s heartbeat)
   and collects all ChildPartitionsRecord entries. If the query fails,
   newSpannerBroadcaster returns the error immediately.

2. The discovered partitions are enqueued into partitionQueue and
   runPartitionLoop is started.

If the root partition has not been split (common on production Spanner
with small tables), discoverPartitions returns a single root partition
work item so the reader loop reads the un-split stream directly.

This follows the same two-phase pattern used by spanner-etcd's
queryInitialPartitions.

Signed-off-by: Nico Schieder <nschieder@redhat.com>
@openshift-ci openshift-ci Bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 11, 2026
@openshift-ci

openshift-ci Bot commented Aug 11, 2026

Copy link
Copy Markdown

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant