SubscribeToTask live events are process-local and can be missed across pods

[xu-shanshan]

Summary

SubscribeToTask appears to rely on process-local subscriber channels, so a subscriber connected to one server instance can miss task events produced on another instance.

This is not a task snapshot persistence issue: the folded task state may still be readable later via GetTask. The risk is specifically live event delivery for an active subscription when producer and subscriber are on different processes/pods.

Current behavior

From ChannelEventNotifier, subscribers are stored in an in-process dictionary of channels.

In a multi-pod/server deployment:

1. Pod B accepts SubscribeToTask and creates a local subscriber channel.
2. Pod A applies a later task event.
3. Pod A persists the updated task snapshot.
4. Pod A notifies only subscribers registered in pod A's process memory.
5. The subscriber on pod B does not receive the live event.

Expected behavior

For deployments with more than one server instance, SubscribeToTask should either:

• deliver task events to subscribers regardless of which instance produced the event, or
• document that subscription delivery requires routing affinity / single-instance topology / an application-provided distributed notifier.

Why this matters

Without this constraint being explicit, a service can pass local or single-pod tests but still hang or miss events under load-balanced production routing.

The key distinction is:

• GetTask: can be satisfied by the persisted snapshot.
• SubscribeToTask: needs live event fan-out to the active subscriber connection.

Possible directions

• Document the topology requirement for the current ChannelEventNotifier.
• Expose or document an extension point for a distributed notifier/backplane.
• Add a test that demonstrates same-notifier delivery succeeds while cross-notifier delivery does not, so the limitation is visible.

Minimal repro idea

Create two notifier instances to simulate two pods:

1. Register a subscriber for taskId on notifier A.
2. Call Notify(taskId, event) on notifier B.
3. Assert notifier A's subscriber does not receive the event.
4. Register and notify on the same notifier, and assert the event is received.

That would make the current process-local behavior deterministic without requiring multi-pod infrastructure in the test.