Skip to content

fix: scope reminder test clock to reminder service - #10317

Merged
ReubenBond merged 6 commits into
dotnet:mainfrom
ReubenBond:reubenbond-investigate-reminder-hang
Aug 4, 2026
Merged

ReubenBond merged 6 commits into
dotnet:mainfrom
ReubenBond:reubenbond-investigate-reminder-hang

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Aug 1, 2026

Copy link
Copy Markdown
Member

ReminderTestClock replaced the unkeyed TimeProvider, so Orleans' keyed fallback routed unrelated silo background timers through FakeTimeProvider. Advancing reminder time could therefore synchronously execute membership, health, and other timer callbacks and hang functional reminder tests.

Register the fake provider specifically for ReminderTimeProviderNames.Reminders and verify that unkeyed silo time remains TimeProvider.System.

The scoped clock also makes table refreshes deterministic at exact reminder cadence boundaries. Load table entries only when their next tick is in the future and within the loading window, deliberately skipping an exact-due table load instead of reloading an occurrence which just fired. The controlled stale-read tests use one reminder owner so their blocked table read and direct local mutation share the sequence whose ordering they validate.

Copilot AI review requested due to automatic review settings August 1, 2026 01:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Scopes the reminders test-time clock to the reminders subsystem’s keyed TimeProvider so that advancing reminder time does not inadvertently advance unrelated silo background timers (membership/health/etc.), preventing functional reminder test hangs.

Changes:

  • Register ReminderTestClock’s FakeTimeProvider as a keyed TimeProvider under ReminderTimeProviderNames.Reminders instead of replacing the unkeyed TimeProvider.
  • Update ReminderTestClock documentation to reflect keyed scoping and reduced side effects.
  • Add a functional test asserting the unkeyed silo TimeProvider remains TimeProvider.System while the reminders provider is separately keyed.
Show a summary per file
File Description
test/Orleans.Reminders.Tests/TimerTests/ReminderTests_TableGrain.cs Adds coverage verifying reminder time is keyed/scoped and the unkeyed silo clock remains TimeProvider.System.
src/Orleans.Testing.Reminders/ReminderTestClock.cs Changes the test clock registration to a reminders-specific keyed TimeProvider and updates remarks accordingly.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

Copilot AI review requested due to automatic review settings August 1, 2026 04:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings August 1, 2026 05:58
@ReubenBond
ReubenBond force-pushed the reubenbond-investigate-reminder-hang branch from 0dbc548 to e7a79b6 Compare August 1, 2026 05:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings August 1, 2026 06:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings August 1, 2026 06:51

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (1)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:889

  • The new IsReminderWithinLoadingWindow(..., DateTime? nextTickTime) overload skips entry/Period validation when nextTickTime is provided (because CalculateNextTickTime is bypassed). This can allow invalid reminder entries to pass silently in the keyed-path calls. Consider validating entry and entry.Period regardless of whether nextTickTime is supplied.
            return (nextTickTime ?? CalculateNextTickTime(entry, now)) <= now.AddClamped(loadingWindow);
  • Files reviewed: 4/4 changed files
  • Comments generated: 0 new

Copilot AI review requested due to automatic review settings August 1, 2026 15:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

Suppressed comments (2)

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:598

  • With shouldLoad currently depending on nextTick > now, tombstones are treated specially at exact cadence boundaries. If shouldLoad is made inclusive (to allow due-now loads), this condition should instead explicitly guard tombstones using nextTick <= now to prevent resurrecting an occurrence which just fired.
            if (!isWithinLoadingWindow || (state is LocalReminderState.Tombstone && !shouldLoad))

src/Orleans.Reminders/ReminderService/LocalReminderService.cs:567

  • shouldLoad is false when CalculateNextTickTime returns now (exact cadence boundary). For entries which are not yet in localReminders, this means the service will skip loading them from storage even though they are within the loading window, delaying delivery until a later refresh. If the intent is only to avoid resurrecting tombstones at exact boundaries, consider allowing due-now loads here and handling the tombstone case separately.

This issue also appears on line 598 of the same file.

            var nextTick = CalculateNextTickTime(entry, now);
            // Keep distant schedules in storage, and skip exact-due table loads to simplify fake-time tests.
            var isWithinLoadingWindow = nextTick <= now.AddClamped(reminderOptions.ReminderLoadingWindow);
            var shouldLoad = nextTick > now && isWithinLoadingWindow;
  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new

Register the fake clock using the reminder subsystem's keyed TimeProvider so advancing reminder time does not synchronously drive unrelated silo background timers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Run exact local reminder lifecycle assertions with a single owner so queued refreshes from another silo cannot span large fake-time advances and produce duplicate transient activity.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Preserve the computed following tick on outside-window tombstones so a refresh at the same timestamp cannot reload and redeliver the occurrence which just completed. Retire the marker once time advances past that occurrence, and restore multi-silo test coverage.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Run deterministic stale-refresh tests with one reminder owner so the blocked range read and direct mutation share the same local sequence. Multi-owner routing is independent of the local reconciliation behavior under test.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Load table entries only when their next tick is in the future and within the loading window. This deliberately skips an exact-boundary table load so frozen fake time cannot reload an occurrence which just fired.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Copilot AI review requested due to automatic review settings August 4, 2026 15:51
@ReubenBond
ReubenBond force-pushed the reubenbond-investigate-reminder-hang branch from c8e8309 to 81bbd18 Compare August 4, 2026 15:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 1

Comment on lines +565 to +567
// Keep distant schedules in storage, and skip exact-due table loads to simplify fake-time tests.
var isWithinLoadingWindow = nextTick <= now.AddClamped(reminderOptions.ReminderLoadingWindow);
var shouldLoad = nextTick > now && isWithinLoadingWindow;

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an intentional trade-off for deterministic fake-time behavior: a table entry whose next occurrence is exactly now is skipped, so it will resume at the following cadence boundary. I expanded the inline comment to make that semantic caveat explicit.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: a6e333e6-0d83-496f-94a5-cd181d30a30b
Copilot AI review requested due to automatic review settings August 4, 2026 19:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot's findings

  • Files reviewed: 3/3 changed files
  • Comments generated: 0 new

@ReubenBond
ReubenBond merged commit 671c5ca into dotnet:main Aug 4, 2026
67 checks passed
@ReubenBond
ReubenBond deleted the reubenbond-investigate-reminder-hang branch August 4, 2026 20:13
This was referenced Aug 28, 2026
@github-actions github-actions Bot locked and limited conversation to collaborators Sep 4, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants