diff --git a/evals/cosmosdb-best-practices/tasks/model-ttl-expiration.yaml b/evals/cosmosdb-best-practices/tasks/model-ttl-expiration.yaml new file mode 100644 index 0000000..0bd73eb --- /dev/null +++ b/evals/cosmosdb-best-practices/tasks/model-ttl-expiration.yaml @@ -0,0 +1,14 @@ +id: model-ttl-expiration +name: Data Modeling - Use TTL for automatic data expiration +description: | + Test that the skill recommends Cosmos DB TTL instead of scheduled cleanup + jobs for data with a natural retention window. +tags: + - modeling + - ttl + - retention +inputs: + prompt: "I store short-lived session tokens and temporary cache entries in Cosmos DB. Should I run a nightly job to delete expired records, or is there a built-in way to expire them automatically?" +expected: + outcomes: + - type: task_completed diff --git a/skills/cosmosdb-best-practices/AGENTS.md b/skills/cosmosdb-best-practices/AGENTS.md index 8bcd21a..42fcc29 100644 --- a/skills/cosmosdb-best-practices/AGENTS.md +++ b/skills/cosmosdb-best-practices/AGENTS.md @@ -29,7 +29,8 @@ Performance optimization and best practices guide for Azure Cosmos DB applicatio - 1.8 [Reference Data When Items Grow Large](#18-reference-data-when-items-grow-large) - 1.9 [Use ID references with transient hydration for document relationships](#19-use-id-references-with-transient-hydration-for-document-relationships) - 1.10 [Version Your Document Schemas](#110-version-your-document-schemas) - - 1.11 [Use Type Discriminators for Polymorphic Data](#111-use-type-discriminators-for-polymorphic-data) + - 1.11 [Use TTL for Automatic Data Expiration](#111-use-ttl-for-automatic-data-expiration) + - 1.12 [Use Type Discriminators for Polymorphic Data](#112-use-type-discriminators-for-polymorphic-data) 2. [Partition Key Design](#2-partition-key-design) — **CRITICAL** - 2.1 [Plan for 20GB Logical Partition Limit](#21-plan-for-20gb-logical-partition-limit) - 2.2 [Distribute Writes to Avoid Hot Partitions](#22-distribute-writes-to-avoid-hot-partitions) @@ -1317,7 +1318,89 @@ Always increment version when: Reference: [Schema evolution in Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/modeling-data) -### 1.11 Use Type Discriminators for Polymorphic Data +### 1.11 Use TTL for Automatic Data Expiration + +**Impact: MEDIUM** (removes stale data without custom cleanup jobs) + +## Use TTL for Automatic Data Expiration + +Use Azure Cosmos DB time to live (TTL) for data that has a natural retention window, such as session tokens, event logs, temporary cache entries, invitations, one-time codes, or short-lived processing state. TTL lets the service expire items automatically instead of requiring a scheduled cleanup job that scans and deletes old records. + +TTL is configured in seconds. The expiration countdown is based on the item's last modified timestamp (`_ts`), so updating an item resets its TTL window. + +**Incorrect (scheduled cleanup job scans and deletes expired items):** + +```csharp +// Anti-pattern: periodic cleanup query scans old items and deletes them one by one. +var query = new QueryDefinition( + "SELECT * FROM c WHERE c.type = 'session' AND c.expiresAt < @now") + .WithParameter("@now", DateTimeOffset.UtcNow); + +using var iterator = container.GetItemQueryIterator(query); +while (iterator.HasMoreResults) +{ + foreach (var session in await iterator.ReadNextAsync()) + { + await container.DeleteItemAsync( + session.Id, + new PartitionKey(session.UserId)); + } +} +``` + +```json +{ + "id": "session-123", + "userId": "user-42", + "type": "session", + "expiresAt": "2026-06-11T13:00:00Z" +} +``` + +**Correct (enable TTL and set per-item expiration):** + +```csharp +// Enable TTL on the container without a default expiration. +// Items expire only when they include their own ttl value. +await database.DefineContainer("sessions", "/userId") + .WithDefaultTimeToLive(-1) + .CreateAsync(); + +var session = new +{ + id = "session-123", + userId = "user-42", + type = "session", + ttl = 3600, // Expire one hour after the item is created or last modified. + createdAt = DateTimeOffset.UtcNow +}; + +await container.CreateItemAsync(session, new PartitionKey(session.userId)); +``` + +```json +{ + "id": "session-123", + "userId": "user-42", + "type": "session", + "ttl": 3600, + "createdAt": "2026-06-11T12:00:00Z" +} +``` + +Use the right TTL mode for the retention pattern: + +- Leave container TTL unset or `null` when items should never expire automatically. +- Set container `DefaultTimeToLive` to a positive number when every item should expire after the same retention period. +- Set container `DefaultTimeToLive` to `-1` when TTL should be enabled but only specific items should expire. +- Set item-level `ttl` to a positive number to override the container default for that item. +- Set item-level `ttl` to `-1` for items that should not expire in a TTL-enabled container. + +TTL is best for automatic retention, not exact scheduling. Expired items stop appearing in query results after the TTL expires, but physical deletion happens asynchronously in the background. In provisioned throughput accounts, TTL deletes use leftover RUs; in serverless accounts, deletes are charged like delete item operations. + +Reference: [Time to Live in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/time-to-live) + +### 1.12 Use Type Discriminators for Polymorphic Data **Impact: MEDIUM** (enables efficient single-container design) diff --git a/skills/cosmosdb-best-practices/rules/model-ttl-expiration.md b/skills/cosmosdb-best-practices/rules/model-ttl-expiration.md new file mode 100644 index 0000000..6b63538 --- /dev/null +++ b/skills/cosmosdb-best-practices/rules/model-ttl-expiration.md @@ -0,0 +1,84 @@ +--- +title: Use TTL for Automatic Data Expiration +impact: MEDIUM +impactDescription: removes stale data without custom cleanup jobs +tags: model, ttl, expiration, retention, cleanup +--- + +## Use TTL for Automatic Data Expiration + +Use Azure Cosmos DB time to live (TTL) for data that has a natural retention window, such as session tokens, event logs, temporary cache entries, invitations, one-time codes, or short-lived processing state. TTL lets the service expire items automatically instead of requiring a scheduled cleanup job that scans and deletes old records. + +TTL is configured in seconds. The expiration countdown is based on the item's last modified timestamp (`_ts`), so updating an item resets its TTL window. + +**Incorrect (scheduled cleanup job scans and deletes expired items):** + +```csharp +// Anti-pattern: periodic cleanup query scans old items and deletes them one by one. +var query = new QueryDefinition( + "SELECT * FROM c WHERE c.type = 'session' AND c.expiresAt < @now") + .WithParameter("@now", DateTimeOffset.UtcNow); + +using var iterator = container.GetItemQueryIterator(query); +while (iterator.HasMoreResults) +{ + foreach (var session in await iterator.ReadNextAsync()) + { + await container.DeleteItemAsync( + session.Id, + new PartitionKey(session.UserId)); + } +} +``` + +```json +{ + "id": "session-123", + "userId": "user-42", + "type": "session", + "expiresAt": "2026-06-11T13:00:00Z" +} +``` + +**Correct (enable TTL and set per-item expiration):** + +```csharp +// Enable TTL on the container without a default expiration. +// Items expire only when they include their own ttl value. +await database.DefineContainer("sessions", "/userId") + .WithDefaultTimeToLive(-1) + .CreateAsync(); + +var session = new +{ + id = "session-123", + userId = "user-42", + type = "session", + ttl = 3600, // Expire one hour after the item is created or last modified. + createdAt = DateTimeOffset.UtcNow +}; + +await container.CreateItemAsync(session, new PartitionKey(session.userId)); +``` + +```json +{ + "id": "session-123", + "userId": "user-42", + "type": "session", + "ttl": 3600, + "createdAt": "2026-06-11T12:00:00Z" +} +``` + +Use the right TTL mode for the retention pattern: + +- Leave container TTL unset or `null` when items should never expire automatically. +- Set container `DefaultTimeToLive` to a positive number when every item should expire after the same retention period. +- Set container `DefaultTimeToLive` to `-1` when TTL should be enabled but only specific items should expire. +- Set item-level `ttl` to a positive number to override the container default for that item. +- Set item-level `ttl` to `-1` for items that should not expire in a TTL-enabled container. + +TTL is best for automatic retention, not exact scheduling. Expired items stop appearing in query results after the TTL expires, but physical deletion happens asynchronously in the background. In provisioned throughput accounts, TTL deletes use leftover RUs; in serverless accounts, deletes are charged like delete item operations. + +Reference: [Time to Live in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/time-to-live)