fix: scope reminder test clock to reminder service - #10317
Conversation
There was a problem hiding this comment.
Pull request overview
Scopes the reminders test-time clock to the reminders subsystem’s keyed TimeProvider so that advancing reminder time does not inadvertently advance unrelated silo background timers (membership/health/etc.), preventing functional reminder test hangs.
Changes:
- Register
ReminderTestClock’sFakeTimeProvideras a keyedTimeProviderunderReminderTimeProviderNames.Remindersinstead of replacing the unkeyedTimeProvider. - Update
ReminderTestClockdocumentation to reflect keyed scoping and reduced side effects. - Add a functional test asserting the unkeyed silo
TimeProviderremainsTimeProvider.Systemwhile the reminders provider is separately keyed.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Reminders.Tests/TimerTests/ReminderTests_TableGrain.cs | Adds coverage verifying reminder time is keyed/scoped and the unkeyed silo clock remains TimeProvider.System. |
| src/Orleans.Testing.Reminders/ReminderTestClock.cs | Changes the test clock registration to a reminders-specific keyed TimeProvider and updates remarks accordingly. |
Copilot's findings
- Files reviewed: 2/2 changed files
- Comments generated: 0
0dbc548 to
e7a79b6
Compare
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (1)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:889
- The new
IsReminderWithinLoadingWindow(..., DateTime? nextTickTime)overload skipsentry/Periodvalidation whennextTickTimeis provided (becauseCalculateNextTickTimeis bypassed). This can allow invalid reminder entries to pass silently in the keyed-path calls. Consider validatingentryandentry.Periodregardless of whethernextTickTimeis supplied.
return (nextTickTime ?? CalculateNextTickTime(entry, now)) <= now.AddClamped(loadingWindow);
- Files reviewed: 4/4 changed files
- Comments generated: 0 new
There was a problem hiding this comment.
Copilot's findings
Suppressed comments (2)
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:598
- With
shouldLoadcurrently depending onnextTick > now, tombstones are treated specially at exact cadence boundaries. IfshouldLoadis made inclusive (to allow due-now loads), this condition should instead explicitly guard tombstones usingnextTick <= nowto prevent resurrecting an occurrence which just fired.
if (!isWithinLoadingWindow || (state is LocalReminderState.Tombstone && !shouldLoad))
src/Orleans.Reminders/ReminderService/LocalReminderService.cs:567
shouldLoadisfalsewhenCalculateNextTickTimereturnsnow(exact cadence boundary). For entries which are not yet inlocalReminders, this means the service will skip loading them from storage even though they are within the loading window, delaying delivery until a later refresh. If the intent is only to avoid resurrecting tombstones at exact boundaries, consider allowing due-now loads here and handling the tombstone case separately.
This issue also appears on line 598 of the same file.
var nextTick = CalculateNextTickTime(entry, now);
// Keep distant schedules in storage, and skip exact-due table loads to simplify fake-time tests.
var isWithinLoadingWindow = nextTick <= now.AddClamped(reminderOptions.ReminderLoadingWindow);
var shouldLoad = nextTick > now && isWithinLoadingWindow;
- Files reviewed: 3/3 changed files
- Comments generated: 0 new
Register the fake clock using the reminder subsystem's keyed TimeProvider so advancing reminder time does not synchronously drive unrelated silo background timers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Run exact local reminder lifecycle assertions with a single owner so queued refreshes from another silo cannot span large fake-time advances and produce duplicate transient activity. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Preserve the computed following tick on outside-window tombstones so a refresh at the same timestamp cannot reload and redeliver the occurrence which just completed. Retire the marker once time advances past that occurrence, and restore multi-silo test coverage. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Run deterministic stale-refresh tests with one reminder owner so the blocked range read and direct mutation share the same local sequence. Multi-owner routing is independent of the local reconciliation behavior under test. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Load table entries only when their next tick is in the future and within the loading window. This deliberately skips an exact-boundary table load so frozen fake time cannot reload an occurrence which just fired. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
c8e8309 to
81bbd18
Compare
| // Keep distant schedules in storage, and skip exact-due table loads to simplify fake-time tests. | ||
| var isWithinLoadingWindow = nextTick <= now.AddClamped(reminderOptions.ReminderLoadingWindow); | ||
| var shouldLoad = nextTick > now && isWithinLoadingWindow; |
There was a problem hiding this comment.
This is an intentional trade-off for deterministic fake-time behavior: a table entry whose next occurrence is exactly now is skipped, so it will resume at the following cadence boundary. I expanded the inline comment to make that semantic caveat explicit.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
ReminderTestClock replaced the unkeyed TimeProvider, so Orleans' keyed fallback routed unrelated silo background timers through FakeTimeProvider. Advancing reminder time could therefore synchronously execute membership, health, and other timer callbacks and hang functional reminder tests.
Register the fake provider specifically for ReminderTimeProviderNames.Reminders and verify that unkeyed silo time remains TimeProvider.System.
The scoped clock also makes table refreshes deterministic at exact reminder cadence boundaries. Load table entries only when their next tick is in the future and within the loading window, deliberately skipping an exact-due table load instead of reloading an occurrence which just fired. The controlled stale-read tests use one reminder owner so their blocked table read and direct local mutation share the sequence whose ordering they validate.