Create a Reminder TestKit with model-based provider conformance tests

[ReubenBond]

Summary

Create a reusable Microsoft.Orleans.Reminders.TestKit, analogous to Microsoft.Orleans.Persistence.TestKit, which defines and validates the correctness guarantees of reminder table providers and the reminder service integration.

This is a follow-up to #947. Storage-side reminder paging and future reminder-table evolution require one executable contract shared by every provider instead of correctness logic distributed across provider-specific suites.

Motivation

Reminder tests are currently spread across:

• test/Orleans.Runtime.Internal.Tests/RemindersTest/ReminderTableTestsBase.cs;
• test/Orleans.Reminders.Tests/TimerTests/ReminderTestsBase.cs and ReminderTests_TableGrain.cs;
• provider suites for ADO.NET, Azure Table, Cosmos DB, DynamoDB, Redis, and Firestore;
• provider-specific serialization, schema, configuration, and standalone cluster tests.

ReminderTableTestsBase already provides a small common suite, but the observable IReminderTable contract, ETag ordering, range semantics, concurrency behavior, reconciliation guarantees, and service-level behavior are not expressed as one reusable conformance package. This makes it difficult to verify that providers behave equivalently and to safely introduce capabilities such as stable, bounded due-window paging.

The persistence subsystem provides a useful precedent: Orleans.Persistence.TestKit combines direct reusable tests with Microsoft.Accordant-generated model tests and runs them independently against each provider.

Proposed work

1. Add a reusable Reminder TestKit

Add a test-kit project following the Orleans.Persistence.TestKit structure, with reusable runners and fixtures for testing:

• IReminderTable directly;
• the reminder service through InProcessTestCluster;
• provider-specific implementations independently, without requiring unrelated provider tests.

The TestKit should own provider-neutral data generation, assertions, cleanup, diagnostics, and failure messages. Provider suites should supply construction/configuration and retain only tests for provider-specific schema, serialization, configuration, migration, and external-service behavior.

2. Define the reminder table correctness contract

Express the observable guarantees for:

• StartAsync/StopAsync lifecycle behavior;
• upsert, point read, grain read, hash-range read, conditional removal, and clear;
• identity as (GrainId, ReminderName);
• ETag creation, replacement, stale-ETag rejection, and failed-removal behavior;
• updates which change StartAt, Period, or future scheduling/index fields;
• inclusive/exclusive hash-range boundaries and wrap-around ranges;
• isolation between grains, reminder names, services, and clusters where applicable;
• deterministic handling of duplicate and concurrent operations;
• cancellation, retry, and failure behavior which is part of the provider contract;
• stable page/continuation semantics when bounded reminder enumeration is introduced;
• explicit update/deletion observations so absence from a due-time page is never interpreted as durable deletion.

Document each guarantee in the TestKit and encode it as executable tests.

3. Add Microsoft.Accordant model-based tests

Create a behavioral model and generated conformance runner similar to GrainStorageModelBasedTestRunner.

The model should generate valid and invalid sequences containing operations such as:

• upsert a new reminder;
• update an existing reminder using its current ETag;
• point-read or grain-read reminders;
• enumerate normal and wrap-around hash ranges;
• remove with current, stale, missing, and mismatched ETags;
• move a reminder between scheduling windows or index buckets;
• repeat reads, writes, and removals;
• clear and re-register reminders.

The modeled state should track reminder identity, persisted schedule, current and previous ETags, existence, hash ownership, and scheduling/index placement. Generated failures must report the minimized/reproducible operation sequence, provider name, inputs, expected state, and observed result.

Use targeted tests alongside the sequential Accordant model for concurrency and distributed behaviors which are not safely represented as a sequential state machine.

4. Add an idealized reminder-table oracle

Implement a deterministic, strongly consistent, in-memory reminder table specifically for the TestKit. It should be an independent reference implementation of the documented contract, not a wrapper around a production reminder provider.

The oracle should support:

• exact ETag and range semantics;
• deterministic time and ordering;
• complete introspection of persisted records and operations;
• controlled blocking, stale snapshots, retries, cancellation, and injected failures;
• future bounded-page and continuation behavior;
• invariant checks which fail at the operation that violates the model.

Run the direct and model-based TestKit against this oracle first. Include negative tests using intentionally invalid implementations so the suite proves that it detects contract violations.

5. Integrate the oracle with InProcessTestCluster

Add a supported testing hook or builder extension which installs the oracle reminder table into every silo in an InProcessTestCluster, similar to how the in-process cluster installs its custom grain directory implementation.

The cluster fixture should expose the shared oracle and test controls so service-level tests can validate:

• registration, update, lookup, enumeration, and unregister;
• loading-window entry and eviction;
• exact-due and late recovery behavior;
• stale refresh ordering and tombstones;
• ownership transfer during silo joins, departures, and restarts;
• registration received by a stale/non-owner silo;
• storage outages and recovery;
• concurrent registration, update, removal, and refresh;
• future due-window page boundaries and continuation behavior.

The idealized table and fake reminder clock should provide deterministic synchronization barriers instead of sleeps or timing ranges.

6. Consolidate existing test logic

Inventory all reminder suites and move provider-neutral scenarios into the TestKit. Every built-in provider should run the same direct and model-based conformance suite wherever its capabilities permit:

• in-memory/grain-based;
• ADO.NET providers;
• Azure Table Storage;
• Cosmos DB;
• DynamoDB;
• Redis;
• Firestore.

Capability differences must be explicit TestKit options with documented semantics, rather than tests being silently omitted. Keep provider-specific tests only where behavior is intentionally provider-specific.

Acceptance criteria

• A dedicated Reminder TestKit project provides reusable direct, cluster, and Accordant model-based runners.
• The reminder table and reminder service correctness guarantees are documented and executable.
• The idealized oracle passes the full suite, while intentionally faulty implementations demonstrate that key violations are detected.
• InProcessTestCluster can be configured to use and expose the oracle reminder table and its deterministic controls.
• All built-in reminder providers run the common direct conformance suite independently.
• Providers which support the same capabilities run identical model-generated sequences and service-level scenarios.
• Existing provider-neutral test logic is removed from provider suites rather than duplicated.
• Test failures identify the provider, operation sequence, reminder identity, expected result, observed result, ETags, ownership/range, and relevant time/window state.
• The TestKit has usage documentation showing how an external reminder provider adopts the suite.
• The suite establishes the baseline needed to specify and validate storage-side due-window paging for Reminders scalability issues #947.