diff --git a/evals/cosmosdb-best-practices/tasks/security-client-encryption.yaml b/evals/cosmosdb-best-practices/tasks/security-client-encryption.yaml new file mode 100644 index 0000000..d45c644 --- /dev/null +++ b/evals/cosmosdb-best-practices/tasks/security-client-encryption.yaml @@ -0,0 +1,19 @@ +id: security-client-encryption +name: Security - Client-Side Encryption for Sensitive Fields +description: | + Test that the skill recommends Always Encrypted with Azure Key Vault for + sensitive fields instead of relying on encryption at rest, and calls out + that encrypted fields cannot be indexed or freely queried. +tags: + - security + - happy-path + - always-encrypted +inputs: + prompt: | + We store patient records in Cosmos DB and need to be HIPAA compliant. + Each document has an SSN and a diagnosis field. Azure says data is + encrypted at rest by default, so is that enough, or do we need to do + something extra for these fields? +expected: + outcomes: + - type: task_completed diff --git a/skills/cosmosdb-best-practices/AGENTS.md b/skills/cosmosdb-best-practices/AGENTS.md index b617766..a8666ab 100644 --- a/skills/cosmosdb-best-practices/AGENTS.md +++ b/skills/cosmosdb-best-practices/AGENTS.md @@ -1,6 +1,6 @@ # Azure Cosmos DB Best Practices -**Version 1.0.0** +**Version 1.1.0** CosmosDB Agent Kit January 2026 @@ -144,6 +144,20 @@ Performance optimization and best practices guide for Azure Cosmos DB applicatio - 11.4 [Configure Vector Indexes in Indexing Policy](#114-configure-vector-indexes-in-indexing-policy) - 11.5 [Normalize Embeddings for Cosine Similarity](#115-normalize-embeddings-for-cosine-similarity) - 11.6 [Implement Repository Pattern for Vector Search](#116-implement-repository-pattern-for-vector-search) +12. [Full-Text Search](#12-full-text-search) — **HIGH** + - 12.1 [Add Full-Text Index in the Indexing Policy](#121-add-full-text-index-in-the-indexing-policy) + - 12.2 [Define Full-Text Policy on the Container](#122-define-full-text-policy-on-the-container) + - 12.3 [Enable Full-Text Search Capability on Account](#123-enable-full-text-search-capability-on-account) + - 12.4 [Combine FTS predicates with range or equality filters for hybrid queries](#124-combine-fts-predicates-with-range-or-equality-filters-for-hybrid-queries) + - 12.5 [Use FullTextContains for keyword matching on indexed text fields](#125-use-fulltextcontains-for-keyword-matching-on-indexed-text-fields) + - 12.6 [Use FullTextScore with ORDER BY RANK for BM25 relevance ranking](#126-use-fulltextscore-with-order-by-rank-for-bm25-relevance-ranking) +13. [Security](#13-security) — **HIGH** + - 13.1 [Encrypt sensitive fields with Always Encrypted client-side encryption](#131-encrypt-sensitive-fields-with-always-encrypted-client-side-encryption) + - 13.2 [Enable Continuous Backup for Point-in-Time Restore](#132-enable-continuous-backup-for-point-in-time-restore) + - 13.3 [Disable Local Authentication (Keys)](#133-disable-local-authentication-keys-) + - 13.4 [Use Managed Identity with DefaultAzureCredential](#134-use-managed-identity-with-defaultazurecredential) + - 13.5 [Restrict Network Access](#135-restrict-network-access) + - 13.6 [Assign Minimum RBAC Roles with Narrow Scope](#136-assign-minimum-rbac-roles-with-narrow-scope) --- @@ -13488,6 +13502,801 @@ const results = await documentRepo.vectorSearch(embedding, { --- +## 12. Full-Text Search + +**Impact: HIGH** + +### 12.1 Add Full-Text Index in the Indexing Policy + +**Impact: HIGH** (without the index, FTS functions fall back to a full scan) + +## Add Full-Text Index in the Indexing Policy + +**Impact: HIGH (without the index, FTS functions fall back to a full scan)** + +The `fullTextIndexes` array in the `indexingPolicy` tells Cosmos DB to build an inverted index for the corresponding path. This is separate from the range index — a field can have both. Fields covered by a full-text index should **not** also appear in `excludedPaths`. + +**Incorrect (field excluded from range index but no FTS index — slow scan):** + +```bicep +excludedPaths: [ + { path: '/description/?' } // excluded from range index... +] // ...but no fullTextIndexes entry → full scan +``` + +**Correct (Bicep):** + +```bicep +indexingPolicy: { + indexingMode: 'consistent' + includedPaths: [ + { path: '/name/?' } + { path: '/userid/?' } + ] + excludedPaths: [ + { path: '/*' } // root wildcard + // description NOT listed here — managed by FTS index below + ] + #disable-next-line BCP037 + fullTextIndexes: [ + { path: '/description' } // inverted index — case-insensitive, tokenized + ] +} +``` + +> A field under `fullTextIndexes` incurs **extra write RU** for index maintenance. Only index fields that are actually queried with `FullTextContains` or `FullTextScore`. + +Reference: [Indexing policy for full-text search](https://learn.microsoft.com/azure/cosmos-db/gen-ai/full-text-search) + +### 12.2 Define Full-Text Policy on the Container + +**Impact: HIGH** (required for tokenizer and stop-word configuration) + +## Define Full-Text Policy on the Container + +**Impact: HIGH (required for tokenizer and stop-word configuration)** + +The `fullTextPolicy` declares which paths are full-text searchable and their language. Supported languages: `en-US`, `de-DE` (preview), `fr-FR` (preview), `it-IT` (preview), `pt-BR` (preview), `pt-PT` (preview), `es-ES` (preview). Language codes are **case-sensitive** — use the exact casing shown (e.g., `en-US` not `en-us`). + +**Incorrect (wrong language casing causes ARM BadRequest):** + +```bicep +fullTextPolicy: { + defaultLanguage: 'en-us' // ❌ lowercase — rejected by ARM + fullTextPaths: [ + { path: '/description', language: 'en-us' } // ❌ + ] +} +``` + +**Correct (Bicep):** + +```bicep +#disable-next-line BCP037 +fullTextPolicy: { + defaultLanguage: 'en-US' // ✅ exact casing required + fullTextPaths: [ + { + path: '/description' + language: 'en-US' // ✅ + } + ] +} +``` + +**Correct — Java SDK (container creation):** + +```java +FullTextPolicy ftsPolicy = new FullTextPolicy() + .setDefaultLanguage("en-US") + .setFullTextPaths(List.of( + new FullTextPath().setPath("/description").setLanguage("en-US") + )); + +CosmosContainerProperties props = new CosmosContainerProperties("videos", "/videoid"); +props.setFullTextPolicy(ftsPolicy); +database.createContainerIfNotExists(props).block(); +``` + +Reference: [Configure full-text policy](https://learn.microsoft.com/azure/cosmos-db/gen-ai/full-text-search) + +### 12.3 Enable Full-Text Search Capability on Account + +**Impact: HIGH** (prerequisite — FTS SQL functions fail without it) + +## Enable Full-Text Search Capability on Account + +**Impact: HIGH (prerequisite — FTS SQL functions fail without it)** + +Full-text search is an opt-in account-level capability. The SQL functions `FullTextContains`, `FullTextContainsAll`, `FullTextContainsAny`, and `FullTextScore` all return an error if this capability is not enabled. + +**Incorrect (capability absent — FTS queries fail at runtime):** + +```sql +-- This query fails with "Function 'FullTextContains' is not supported" +-- when EnableNoSQLFullTextSearch capability is missing on the account +SELECT * FROM c WHERE FullTextContains(c.description, 'cosmos') +``` + +**Correct — enable via Azure CLI:** + +```bash +az cosmosdb update \ + --resource-group \ + --name \ + --capabilities EnableNoSQLFullTextSearch +``` + +**Correct — enable via Bicep (account resource):** + +```bicep +resource cosmosAccount 'Microsoft.DocumentDB/databaseAccounts@2024-05-15' = { + name: cosmosAccountName + properties: { + // ... other properties ... + capabilities: [ + { name: 'EnableNoSQLFullTextSearch' } + ] + } +} +``` + +> **Note:** As of Bicep type library v0.41, `fullTextIndexes` and `fullTextPolicy` may emit `BCP037` warnings. Suppress with `#disable-next-line BCP037` — the properties are valid at the ARM REST API level. + +Reference: [Full-text search in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/gen-ai/full-text-search) + +### 12.4 Combine FTS predicates with range or equality filters for hybrid queries + +**Impact: MEDIUM** (avoids full-container scans when combined with equality/range filters) + +## Combine FTS with Range Filters for Hybrid Queries + +**Impact: MEDIUM (avoids full-container scans when combined with equality/range filters)** + +FTS predicates can be combined with standard SQL predicates. Cosmos DB uses the most selective predicate first. Put the most restrictive filter (e.g., equality on a high-cardinality property) before the FTS predicate to reduce the candidate set. + +**Incorrect (FTS-only query — no range filters, scans all partitions):** + +```sql +-- ❌ No equality filter — Cosmos DB must scan every partition before ranking +SELECT * FROM c +WHERE FullTextContains(c.description, @q) +ORDER BY RANK FullTextScore(c.description, @q) +``` + +**Correct — filter by partition + FTS:** + +```sql +SELECT * FROM c +WHERE c.type = 'video' + AND c.userid = @userid + AND FullTextContains(c.description, @q) +ORDER BY RANK FullTextScore(c.description, @q) +``` + +```java +// Hybrid: exact field filters narrow partition, FTS ranks within results +String sql = "SELECT * FROM c " + + "WHERE c.type = 'video' " + + "AND FullTextContains(c.description, @q) " + + "ORDER BY RANK FullTextScore(c.description, @q)"; + +CosmosQueryRequestOptions opts = new CosmosQueryRequestOptions(); +// enableCrossPartitionQuery is true by default for FTS ORDER BY RANK + +return container.queryItems( + new SqlQuerySpec(sql, new SqlParameter("@q", term)), + opts, Video.class +).byPage(pageSize).next().toFuture(); +``` + +**Fields that should NOT use FTS:** +- Short identifiers (`id`, `userid`) — use point read or range index equality +- Numeric fields — use range index with `=`, `>`, `<` +- Array elements already indexed with `[]/?` — `CONTAINS(LOWER(t), @q)` via EXISTS is fine + +Reference: [Full-text search queries](https://learn.microsoft.com/azure/cosmos-db/gen-ai/full-text-search) + +### 12.5 Use FullTextContains for keyword matching on indexed text fields + +**Impact: HIGH** (replaces expensive CONTAINS(LOWER(...)) string scans with O(log n) inverted index lookup) + +## Use FullTextContains for Keyword Matching + +**Impact: HIGH (replaces expensive CONTAINS(LOWER(...)) string scans with O(log n) inverted index lookup)** + +`FullTextContains(path, term)` performs a single-keyword lookup against the inverted index and is case-insensitive by design. It is dramatically faster than `CONTAINS(LOWER(c.field), @q)` on large containers because it does an `O(log n)` index lookup instead of a full document scan. + +**Incorrect (scan-based — avoid for long text fields with FTS index):** + +```sql +-- Full document scan, case folding at query time +SELECT * FROM c +WHERE CONTAINS(LOWER(c.description), @q) +``` + +```java +String sql = "SELECT * FROM c WHERE CONTAINS(LOWER(c.description), @q)"; +``` + +**Correct:** + +```sql +-- Inverted index lookup — no LOWER() needed, FTS tokenizer handles casing +SELECT * FROM c +WHERE FullTextContains(c.description, @q) +``` + +```java +// Java SDK — parameterized query with FullTextContains +String sql = "SELECT * FROM c WHERE c.type = 'video' " + + "AND (CONTAINS(LOWER(c.name), @q) " + // short field — range index OK + "OR FullTextContains(c.description, @q) " + // long text — FTS index + "OR EXISTS(SELECT VALUE t FROM t IN c.tags WHERE CONTAINS(LOWER(t), @q)))"; + +SqlQuerySpec querySpec = new SqlQuerySpec(sql, + new SqlParameter("@q", query.trim().toLowerCase())); + +return container.queryItems(querySpec, opts, Video.class) + .byPage(continuationToken, pageSize) + .next() + .map(page -> new ResultListPage<>(page.getResults(), page.getContinuationToken())) + .toFuture(); +``` + +**Variants:** +- `FullTextContains(path, term)` — document contains the term +- `FullTextContainsAll(path, term1, term2, ...)` — document contains ALL terms (AND) +- `FullTextContainsAny(path, term1, term2, ...)` — document contains ANY term (OR) + +Reference: [FullTextContains function](https://learn.microsoft.com/azure/cosmos-db/nosql/query/fulltextcontains) + +### 12.6 Use FullTextScore with ORDER BY RANK for BM25 relevance ranking + +**Impact: MEDIUM-HIGH** (enables BM25-based ranked results instead of arbitrary order) + +## Use FullTextScore for Relevance Ranking + +**Impact: MEDIUM-HIGH (enables BM25-based ranked results instead of arbitrary order)** + +`FullTextScore(path, term)` returns a BM25 relevance score. Use it in `ORDER BY` to surface the most relevant documents first. It **requires** `FullTextContains` in the WHERE clause on the same path. + +**Incorrect (FullTextScore without FullTextContains — parse error):** + +```sql +SELECT * FROM c +ORDER BY FullTextScore(c.description, 'cosmos') -- ❌ missing WHERE FullTextContains +``` + +**Correct:** + +```sql +SELECT c.name, c.description, c.addedDate +FROM c +WHERE FullTextContains(c.description, @q) +ORDER BY RANK FullTextScore(c.description, @q) +``` + +```java +String sql = "SELECT c.name, c.description, c.addedDate FROM c " + + "WHERE FullTextContains(c.description, @q) " + + "ORDER BY RANK FullTextScore(c.description, @q)"; + +SqlQuerySpec querySpec = new SqlQuerySpec(sql, new SqlParameter("@q", searchTerm)); +``` + +> `RANK FullTextScore(...)` is cross-partition — Cosmos DB merges and re-ranks results from all partitions before returning the page. + +Reference: [FullTextScore function](https://learn.microsoft.com/azure/cosmos-db/nosql/query/fulltextscore) + +--- + +## 13. Security + +**Impact: HIGH** + +### 13.1 Encrypt sensitive fields with Always Encrypted client-side encryption + +**Impact: MEDIUM** (sensitive fields stay unreadable to anyone without key access, including operators with full account permissions) + +## Encrypt Sensitive Fields with Always Encrypted Client-Side Encryption + +**Impact: MEDIUM (sensitive fields stay unreadable to anyone without key access, including operators with full account permissions)** + +Cosmos DB encrypts all data at rest by default, but that only protects the storage media. Anyone with account keys, RBAC read access, or portal access still sees documents in plaintext. For regulated data such as PII, health records, or payment details, use Always Encrypted: marked fields are encrypted inside the client SDK before the document leaves your process, and the service only ever stores ciphertext. Decryption requires access to the key in Azure Key Vault, which you control separately from database access. + +**When to use it:** + +- Compliance requirements like GDPR, HIPAA, or PCI DSS +- Fields such as national IDs, card numbers, salaries, or diagnoses +- Separating duties: database operators can administer data they cannot read + +**Limitations to plan around:** + +- Encrypted paths are not indexed; randomized encryption cannot be filtered on at all, deterministic encryption supports equality filters only +- The `id` and partition key paths cannot be encrypted +- The encryption policy is set at container creation and cannot be changed later, so decide the paths up front +- Key wrap and unwrap calls add latency on first use per key (cached afterwards) + +**Incorrect (plaintext PII, relying on encryption at rest):** + +```csharp +// Encryption at rest is always on, but it is transparent: anyone who can +// read the container sees the SSN in plaintext, portal included +var patient = new Patient { Id = "p1", HospitalId = "h1", Ssn = "123-45-6789" }; +await container.CreateItemAsync(patient, new PartitionKey(patient.HospitalId)); +``` + +**Correct (client-side encryption backed by Azure Key Vault):** + +```csharp +// Package: Microsoft.Azure.Cosmos.Encryption +using Azure.Identity; +using Azure.Security.KeyVault.Keys.Cryptography; +using Microsoft.Azure.Cosmos.Encryption; + +var credential = new DefaultAzureCredential(); +var client = new CosmosClient(endpoint, credential) + .WithEncryption(new KeyResolver(credential), KeyEncryptionKeyResolverName.AzureKeyVault); + +// One-time setup: a data encryption key, wrapped by a key you own in Key Vault +var database = client.GetDatabase("hospital"); +await database.CreateClientEncryptionKeyAsync( + "patient-cek", + DataEncryptionAlgorithm.AeadAes256CbcHmacSha256, + new EncryptionKeyWrapMetadata( + KeyEncryptionKeyResolverName.AzureKeyVault, + "cosmos-kek", + "https://myvault.vault.azure.net/keys/cosmos-kek", + EncryptionAlgorithm.RsaOaep.ToString())); + +// One-time setup: declare which paths are encrypted, and how +var paths = new List +{ + new ClientEncryptionIncludedPath + { + Path = "/ssn", + ClientEncryptionKeyId = "patient-cek", + EncryptionType = EncryptionType.Deterministic, // allows equality filters + EncryptionAlgorithm = "AEAD_AES_256_CBC_HMAC_SHA256" + }, + new ClientEncryptionIncludedPath + { + Path = "/diagnosis", + ClientEncryptionKeyId = "patient-cek", + EncryptionType = EncryptionType.Randomized, // stronger, never filtered on + EncryptionAlgorithm = "AEAD_AES_256_CBC_HMAC_SHA256" + } +}; + +await database.CreateContainerAsync(new ContainerProperties("patients", "/hospitalId") +{ + ClientEncryptionPolicy = new ClientEncryptionPolicy(paths) +}); +``` + +The same setup in Java with the `com.azure:azure-cosmos-encryption` package: + +```java +TokenCredential credential = new DefaultAzureCredentialBuilder().build(); +KeyEncryptionKeyClientBuilder keyResolver = + new KeyEncryptionKeyClientBuilder().credential(credential); + +CosmosAsyncClient client = new CosmosClientBuilder() + .endpoint(endpoint) + .credential(credential) + .buildAsyncClient(); +CosmosEncryptionAsyncClient encryptionClient = new CosmosEncryptionClientBuilder() + .cosmosAsyncClient(client) + .keyEncryptionKeyResolver(keyResolver) + .keyEncryptionKeyResolverName(CosmosEncryptionClientBuilder.KEY_RESOLVER_NAME_AZURE_KEY_VAULT) + .buildAsyncClient(); + +// One-time setup: a data encryption key, wrapped by a key you own in Key Vault +CosmosEncryptionAsyncDatabase database = + encryptionClient.getCosmosEncryptionAsyncDatabase("hospital"); +EncryptionKeyWrapMetadata metadata = new EncryptionKeyWrapMetadata( + encryptionClient.getKeyEncryptionKeyResolverName(), + "cosmos-kek", + "https://myvault.vault.azure.net/keys/cosmos-kek", + EncryptionAlgorithm.RSA_OAEP.toString()); +database.createClientEncryptionKey( + "patient-cek", + CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName(), + metadata).block(); + +// One-time setup: declare which paths are encrypted, and how +ClientEncryptionIncludedPath ssn = new ClientEncryptionIncludedPath(); +ssn.setPath("/ssn"); +ssn.setClientEncryptionKeyId("patient-cek"); +ssn.setEncryptionType(CosmosEncryptionType.DETERMINISTIC.getName()); // allows equality filters +ssn.setEncryptionAlgorithm(CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName()); + +ClientEncryptionIncludedPath diagnosis = new ClientEncryptionIncludedPath(); +diagnosis.setPath("/diagnosis"); +diagnosis.setClientEncryptionKeyId("patient-cek"); +diagnosis.setEncryptionType(CosmosEncryptionType.RANDOMIZED.getName()); // stronger, never filtered on +diagnosis.setEncryptionAlgorithm(CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName()); + +CosmosContainerProperties containerProperties = + new CosmosContainerProperties("patients", "/hospitalId"); +containerProperties.setClientEncryptionPolicy( + new ClientEncryptionPolicy(Arrays.asList(ssn, diagnosis))); +database.createEncryptionContainerAsync(containerProperties).block(); +``` + +Reads and writes stay the same; the SDK encrypts and decrypts the declared paths transparently: + +```csharp +// Stored as ciphertext, returned decrypted to callers holding key access +await container.CreateItemAsync(patient, new PartitionKey(patient.HospitalId)); +``` + +Querying a deterministically encrypted field requires the parameter to be encrypted too: + +```csharp +var query = new QueryDefinition("SELECT * FROM c WHERE c.ssn = @ssn"); +await query.AddParameterAsync("@ssn", "123-45-6789", "/ssn"); +``` + +**Choosing an encryption type per field:** + +- Deterministic: the same plaintext always produces the same ciphertext, so equality lookups work. Use for fields you must search by, like an SSN. +- Randomized: stronger protection, no querying at all. Use for fields you only read back, like a diagnosis or card number. +- No encryption: anything you index, sort, aggregate, or use as a partition key. + +**How this differs from encryption at rest:** + +- Encryption at rest is always on, managed by the service, and protects disks and backups. It does nothing against a leaked key or an over-permissioned reader, because data is decrypted for every authorized request. +- Always Encrypted is opt-in per field, the keys live in your Key Vault, and ciphertext is all the service ever holds. Revoking key access makes the fields unreadable even for the account owner. + +Reference: [Use client-side encryption with Always Encrypted for Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/how-to-always-encrypted) + +### 13.2 Enable Continuous Backup for Point-in-Time Restore + +**Impact: MEDIUM** (enables recovery from accidental data loss) + +## Enable Continuous Backup for Point-in-Time Restore + +**Impact: MEDIUM (enables recovery from accidental data loss)** + +Data loss is more often caused by mistakes than by attackers. Enable continuous backup (7 or 30 days) to allow point-in-time restore. Enable it at account creation if possible — switching from periodic to continuous is supported but is a one-way change. + +**Incorrect (relying on default periodic backup):** + +```bash +# Default periodic backup: +# - 4 hour intervals between backups +# - Only 2 copies retained +# - Recovery requires a support ticket +# - Cannot restore to a specific point in time +# - Data written between backups can be lost permanently + +az cosmosdb create \ + --name myaccount \ + --resource-group myrg + # Default periodic backup — limited recovery options +``` + +**Correct (continuous backup enabled):** + +```bash +# Enable at account creation (preferred) +az cosmosdb create \ + --name myaccount \ + --resource-group myrg \ + --backup-policy-type Continuous \ + --continuous-tier Continuous7Days + +# Or upgrade an existing account (one-way change) +az cosmosdb update \ + --name myaccount \ + --resource-group myrg \ + --backup-policy-type Continuous \ + --continuous-tier Continuous7Days + +# Tiers available: +# Continuous7Days — 7-day retention, lower cost +# Continuous30Days — 30-day retention, for compliance-sensitive workloads +``` + +```bash +# Restore to a specific point in time (self-service, no support ticket) +az cosmosdb restore \ + --account-name myaccount \ + --resource-group myrg \ + --target-database-account-name myaccount-restored \ + --restore-timestamp "2026-05-29T10:00:00Z" \ + --location "East US" +``` + +Continuous backup protects against: +- Accidental deletion of containers or databases +- Buggy deployments that corrupt data +- Unintended bulk updates or deletes +- Ransomware or malicious data modification (when combined with audit logs to identify the point of compromise) + +Reference: [Continuous backup with point-in-time restore in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/continuous-backup-restore-introduction) + +### 13.3 Disable Local Authentication (Keys) + +**Impact: CRITICAL** (eliminates credential leakage risk) + +## Disable Local Authentication (Keys) + +**Impact: CRITICAL (eliminates credential leakage risk)** + +Disable local authentication (shared keys and connection strings) on your Cosmos DB account. Keys are bearer tokens — anyone who has one can read, modify, or delete all data. If a key leaks, the only option is to regenerate it and update every dependent system. Disabling keys forces all access through Entra ID, eliminating this entire class of risk. + +**Incorrect (using connection string with keys):** + +```csharp +// WRONG: Connection string contains a master key +// If this leaks via source control, logs, or config, all data is exposed +var connectionString = "AccountEndpoint=https://myaccount.documents.azure.com:443/;AccountKey=abc123...==;"; +var client = new CosmosClient(connectionString); + +// Risks: +// - Key in source control (even in .env files that get committed) +// - Key in CI/CD logs or screenshots +// - Key shared across teams with no audit trail +// - No way to attribute access to a specific identity +// - Rotation requires updating every system simultaneously +``` + +**Correct (disable keys, use Entra ID exclusively):** + +```bash +# Disable local authentication on the account +az cosmosdb update \ + --name \ + --resource-group \ + --disable-local-auth true +``` + +```csharp +// Connect using Entra ID — no keys or connection strings needed +using Azure.Identity; +using Microsoft.Azure.Cosmos; + +var client = new CosmosClient( + accountEndpoint: "https://myaccount.documents.azure.com:443/", + tokenCredential: new DefaultAzureCredential() +); + +// Benefits: +// - No secrets to leak +// - Access is auditable per identity +// - Revocation is instant and targeted +// - Works in dev (az login), Azure (managed identity), and CI/CD (service principal) +``` + +If you cannot disable keys immediately, at minimum: never store connection strings in source control, use Azure Key Vault for secret storage, and enable secret scanning in your repository. + +Reference: [Disable local authentication in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/how-to-setup-rbac#disable-local-auth) + +### 13.4 Use Managed Identity with DefaultAzureCredential + +**Impact: CRITICAL** (zero-secret authentication for all environments) + +## Use Managed Identity with DefaultAzureCredential + +**Impact: CRITICAL (zero-secret authentication for all environments)** + +Authenticate to Cosmos DB using managed identity and `DefaultAzureCredential`. This provides a single code path that works in local development (via `az login`), Azure compute (via system-assigned managed identity), and CI/CD (via service principal or federated identity) — with no secrets in code or configuration. + +**Incorrect (hard-coded keys or environment-specific auth):** + +```csharp +// WRONG: Key stored in configuration +var client = new CosmosClient( + "https://myaccount.documents.azure.com:443/", + "abc123masterkey==" +); + +// WRONG: Connection string in environment variable still contains a secret +var connectionString = Environment.GetEnvironmentVariable("COSMOS_CONNECTION_STRING"); +var client = new CosmosClient(connectionString); + +// WRONG: Different auth code per environment +if (isDevelopment) + client = new CosmosClient(connectionString); // key-based +else + client = new CosmosClient(endpoint, new ManagedIdentityCredential()); // identity +``` + +**Correct (DefaultAzureCredential everywhere):** + +```csharp +using Azure.Identity; +using Microsoft.Azure.Cosmos; + +// Same code works in all environments: +// - Local dev: uses az login / Visual Studio / VS Code credentials +// - Azure (App Service, Functions, Container Apps, AKS): uses managed identity +// - CI/CD: uses service principal or workload identity federation +var client = new CosmosClient( + accountEndpoint: "https://myaccount.documents.azure.com:443/", + tokenCredential: new DefaultAzureCredential() +); +``` + +```python +from azure.identity import DefaultAzureCredential +from azure.cosmos import CosmosClient + +credential = DefaultAzureCredential() +client = CosmosClient("https://myaccount.documents.azure.com:443/", credential) +``` + +```javascript +const { DefaultAzureCredential } = require("@azure/identity"); +const { CosmosClient } = require("@azure/cosmos"); + +const credential = new DefaultAzureCredential(); +const client = new CosmosClient({ + endpoint: "https://myaccount.documents.azure.com:443/", + aadCredentials: credential +}); +``` + +```java +import com.azure.identity.DefaultAzureCredentialBuilder; +import com.azure.cosmos.CosmosClientBuilder; + +CosmosClient client = new CosmosClientBuilder() + .endpoint("https://myaccount.documents.azure.com:443/") + .credential(new DefaultAzureCredentialBuilder().build()) + .buildClient(); +``` + +For Azure compute, assign a system-assigned managed identity: + +```bash +# App Service +az webapp identity assign --name --resource-group + +# Azure Functions +az functionapp identity assign --name --resource-group + +# Container Apps +az containerapp identity assign --name --resource-group --system-assigned +``` + +Starting with `DefaultAzureCredential` from day one avoids a painful migration later — moving from keys to managed identity means touching every deployment, every environment, and potentially every SDK call. + +Reference: [DefaultAzureCredential Class](https://learn.microsoft.com/dotnet/api/azure.identity.defaultazurecredential) + +### 13.5 Restrict Network Access + +**Impact: HIGH** (reduces attack surface from public internet) + +## Restrict Network Access + +**Impact: HIGH (reduces attack surface from public internet)** + +By default, a Cosmos DB endpoint is publicly reachable from anywhere on the internet. If a credential leaks, nothing stands between an attacker and your data. Restrict access to known IP ranges as a baseline, and plan to move to private endpoints for production workloads. + +**Incorrect (unrestricted public access):** + +```bash +# WRONG: Default configuration — account is accessible from any IP address worldwide +# No --ip-range-filter means open to the internet + +az cosmosdb create \ + --name myaccount \ + --resource-group myrg + # No network restrictions = reachable from anywhere +``` + +**Correct (restrict to known IPs as baseline):** + +```bash +# Restrict access to known IP addresses (office, CI/CD egress, developer IPs) +az cosmosdb update \ + --name myaccount \ + --resource-group myrg \ + --ip-range-filter "203.0.113.10,198.51.100.0/24" + +# For production: use private endpoints (no public internet exposure) +az cosmosdb update \ + --name myaccount \ + --resource-group myrg \ + --public-network-access DISABLED + +# Create a private endpoint in your VNet +az network private-endpoint create \ + --name myaccount-pe \ + --resource-group myrg \ + --vnet-name myvnet \ + --subnet default \ + --private-connection-resource-id \ + --group-id Sql \ + --connection-name myaccount-connection +``` + +Network restriction tiers (from minimum to most secure): +1. **IP allowlisting** (day one minimum): restrict to office, CI/CD, and developer IPs +2. **Service endpoints**: allow access from specific Azure VNet subnets +3. **Private endpoints** (production goal): no public exposure, traffic stays on Microsoft backbone + +Even with Entra ID authentication, network restrictions add defense-in-depth — a compromised token is useless if the attacker cannot reach the endpoint. + +Reference: [Configure IP firewall in Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/how-to-configure-firewall) + +### 13.6 Assign Minimum RBAC Roles with Narrow Scope + +**Impact: HIGH** (limits blast radius of compromised identities) + +## Assign Minimum RBAC Roles with Narrow Scope + +**Impact: HIGH (limits blast radius of compromised identities)** + +Grant each identity only the Cosmos DB data plane role it needs, scoped to the narrowest resource level possible. Avoid account-wide contributor access when an app only reads from a single container. Separate data plane access (read/write data) from control plane access (manage account settings). + +**Incorrect (over-privileged access):** + +```bash +# WRONG: Granting full Contributor at account scope to an app that only reads data +az cosmosdb sql role assignment create \ + --account-name myaccount \ + --resource-group myrg \ + --role-definition-id "00000000-0000-0000-0000-000000000002" \ + --principal-id \ + --scope "/" + +# WRONG: Giving the app control plane access (can delete containers, change settings) +az role assignment create \ + --role "Contributor" \ + --assignee \ + --scope "/subscriptions//resourceGroups//providers/Microsoft.DocumentDB/databaseAccounts/myaccount" + +# WRONG: Sharing one identity across multiple services +# If one service is compromised, attacker gets access to everything +``` + +**Correct (least privilege, narrowly scoped):** + +```bash +# Built-in data plane roles: +# Cosmos DB Built-in Data Reader: 00000000-0000-0000-0000-000000000001 +# Cosmos DB Built-in Data Contributor: 00000000-0000-0000-0000-000000000002 + +# Read-only app: grant Reader scoped to specific container +az cosmosdb sql role assignment create \ + --account-name myaccount \ + --resource-group myrg \ + --role-definition-id "00000000-0000-0000-0000-000000000001" \ + --principal-id \ + --scope "/dbs/mydb/colls/products" + +# Read-write app: grant Contributor scoped to specific database +az cosmosdb sql role assignment create \ + --account-name myaccount \ + --resource-group myrg \ + --role-definition-id "00000000-0000-0000-0000-000000000002" \ + --principal-id \ + --scope "/dbs/mydb" + +# CI/CD pipeline: only data plane write for schema migrations +az cosmosdb sql role assignment create \ + --account-name myaccount \ + --resource-group myrg \ + --role-definition-id "00000000-0000-0000-0000-000000000002" \ + --principal-id \ + --scope "/dbs/mydb" +``` + +Guidelines for role assignment: +- **Application**: Data plane only, minimum role (Reader vs Contributor), scoped to its database or container +- **Developers**: Data plane access on dev accounts, scoped narrowly, using their own Entra ID identity +- **CI/CD pipeline**: Only permissions required to deploy — often just data plane write, sometimes control plane for container management +- **Each identity gets its own access** — never share a single credential across users, environments, or systems + +Reference: [Use data plane role-based access control with Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/security/how-to-grant-data-plane-role-based-access) + +--- + ## References - [Azure Cosmos DB documentation](https://learn.microsoft.com/azure/cosmos-db/) diff --git a/skills/cosmosdb-best-practices/rules/_sections.md b/skills/cosmosdb-best-practices/rules/_sections.md index 5fe9197..2cd5ed8 100644 --- a/skills/cosmosdb-best-practices/rules/_sections.md +++ b/skills/cosmosdb-best-practices/rules/_sections.md @@ -1,3 +1,59 @@ +--- +sections: + - prefix: model- + name: Data Modeling + number: 1 + impact: CRITICAL + - prefix: partition- + name: Partition Key Design + number: 2 + impact: CRITICAL + - prefix: query- + name: Query Optimization + number: 3 + impact: HIGH + - prefix: sdk- + name: SDK Best Practices + number: 4 + impact: HIGH + - prefix: index- + name: Indexing Strategies + number: 5 + impact: MEDIUM-HIGH + - prefix: throughput- + name: Throughput & Scaling + number: 6 + impact: MEDIUM + - prefix: global- + name: Global Distribution + number: 7 + impact: MEDIUM + - prefix: monitoring- + name: Monitoring & Diagnostics + number: 8 + impact: LOW-MEDIUM + - prefix: pattern- + name: Design Patterns + number: 9 + impact: HIGH + - prefix: tooling- + name: Developer Tooling + number: 10 + impact: MEDIUM + - prefix: vector- + name: Vector Search + number: 11 + impact: HIGH + - prefix: fts- + name: Full-Text Search + number: 12 + impact: HIGH + - prefix: security- + name: Security + number: 13 + impact: HIGH +--- + # 1. Data Modeling (model) **Impact:** CRITICAL diff --git a/skills/cosmosdb-best-practices/rules/security-client-encryption.md b/skills/cosmosdb-best-practices/rules/security-client-encryption.md new file mode 100644 index 0000000..fd0e021 --- /dev/null +++ b/skills/cosmosdb-best-practices/rules/security-client-encryption.md @@ -0,0 +1,159 @@ +--- +title: Encrypt sensitive fields with Always Encrypted client-side encryption +impact: MEDIUM +impactDescription: sensitive fields stay unreadable to anyone without key access, including operators with full account permissions +tags: security, always-encrypted, client-side-encryption, key-vault, compliance +--- + +## Encrypt Sensitive Fields with Always Encrypted Client-Side Encryption + +**Impact: MEDIUM (sensitive fields stay unreadable to anyone without key access, including operators with full account permissions)** + +Cosmos DB encrypts all data at rest by default, but that only protects the storage media. Anyone with account keys, RBAC read access, or portal access still sees documents in plaintext. For regulated data such as PII, health records, or payment details, use Always Encrypted: marked fields are encrypted inside the client SDK before the document leaves your process, and the service only ever stores ciphertext. Decryption requires access to the key in Azure Key Vault, which you control separately from database access. + +**When to use it:** + +- Compliance requirements like GDPR, HIPAA, or PCI DSS +- Fields such as national IDs, card numbers, salaries, or diagnoses +- Separating duties: database operators can administer data they cannot read + +**Limitations to plan around:** + +- Encrypted paths are not indexed; randomized encryption cannot be filtered on at all, deterministic encryption supports equality filters only +- The `id` and partition key paths cannot be encrypted +- The encryption policy is set at container creation and cannot be changed later, so decide the paths up front +- Key wrap and unwrap calls add latency on first use per key (cached afterwards) + +**Incorrect (plaintext PII, relying on encryption at rest):** + +```csharp +// Encryption at rest is always on, but it is transparent: anyone who can +// read the container sees the SSN in plaintext, portal included +var patient = new Patient { Id = "p1", HospitalId = "h1", Ssn = "123-45-6789" }; +await container.CreateItemAsync(patient, new PartitionKey(patient.HospitalId)); +``` + +**Correct (client-side encryption backed by Azure Key Vault):** + +```csharp +// Package: Microsoft.Azure.Cosmos.Encryption +using Azure.Identity; +using Azure.Security.KeyVault.Keys.Cryptography; +using Microsoft.Azure.Cosmos.Encryption; + +var credential = new DefaultAzureCredential(); +var client = new CosmosClient(endpoint, credential) + .WithEncryption(new KeyResolver(credential), KeyEncryptionKeyResolverName.AzureKeyVault); + +// One-time setup: a data encryption key, wrapped by a key you own in Key Vault +var database = client.GetDatabase("hospital"); +await database.CreateClientEncryptionKeyAsync( + "patient-cek", + DataEncryptionAlgorithm.AeadAes256CbcHmacSha256, + new EncryptionKeyWrapMetadata( + KeyEncryptionKeyResolverName.AzureKeyVault, + "cosmos-kek", + "https://myvault.vault.azure.net/keys/cosmos-kek", + EncryptionAlgorithm.RsaOaep.ToString())); + +// One-time setup: declare which paths are encrypted, and how +var paths = new List +{ + new ClientEncryptionIncludedPath + { + Path = "/ssn", + ClientEncryptionKeyId = "patient-cek", + EncryptionType = EncryptionType.Deterministic, // allows equality filters + EncryptionAlgorithm = "AEAD_AES_256_CBC_HMAC_SHA256" + }, + new ClientEncryptionIncludedPath + { + Path = "/diagnosis", + ClientEncryptionKeyId = "patient-cek", + EncryptionType = EncryptionType.Randomized, // stronger, never filtered on + EncryptionAlgorithm = "AEAD_AES_256_CBC_HMAC_SHA256" + } +}; + +await database.CreateContainerAsync(new ContainerProperties("patients", "/hospitalId") +{ + ClientEncryptionPolicy = new ClientEncryptionPolicy(paths) +}); +``` + +The same setup in Java with the `com.azure:azure-cosmos-encryption` package: + +```java +TokenCredential credential = new DefaultAzureCredentialBuilder().build(); +KeyEncryptionKeyClientBuilder keyResolver = + new KeyEncryptionKeyClientBuilder().credential(credential); + +CosmosAsyncClient client = new CosmosClientBuilder() + .endpoint(endpoint) + .credential(credential) + .buildAsyncClient(); +CosmosEncryptionAsyncClient encryptionClient = new CosmosEncryptionClientBuilder() + .cosmosAsyncClient(client) + .keyEncryptionKeyResolver(keyResolver) + .keyEncryptionKeyResolverName(CosmosEncryptionClientBuilder.KEY_RESOLVER_NAME_AZURE_KEY_VAULT) + .buildAsyncClient(); + +// One-time setup: a data encryption key, wrapped by a key you own in Key Vault +CosmosEncryptionAsyncDatabase database = + encryptionClient.getCosmosEncryptionAsyncDatabase("hospital"); +EncryptionKeyWrapMetadata metadata = new EncryptionKeyWrapMetadata( + encryptionClient.getKeyEncryptionKeyResolverName(), + "cosmos-kek", + "https://myvault.vault.azure.net/keys/cosmos-kek", + EncryptionAlgorithm.RSA_OAEP.toString()); +database.createClientEncryptionKey( + "patient-cek", + CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName(), + metadata).block(); + +// One-time setup: declare which paths are encrypted, and how +ClientEncryptionIncludedPath ssn = new ClientEncryptionIncludedPath(); +ssn.setPath("/ssn"); +ssn.setClientEncryptionKeyId("patient-cek"); +ssn.setEncryptionType(CosmosEncryptionType.DETERMINISTIC.getName()); // allows equality filters +ssn.setEncryptionAlgorithm(CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName()); + +ClientEncryptionIncludedPath diagnosis = new ClientEncryptionIncludedPath(); +diagnosis.setPath("/diagnosis"); +diagnosis.setClientEncryptionKeyId("patient-cek"); +diagnosis.setEncryptionType(CosmosEncryptionType.RANDOMIZED.getName()); // stronger, never filtered on +diagnosis.setEncryptionAlgorithm(CosmosEncryptionAlgorithm.AEAD_AES_256_CBC_HMAC_SHA256.getName()); + +CosmosContainerProperties containerProperties = + new CosmosContainerProperties("patients", "/hospitalId"); +containerProperties.setClientEncryptionPolicy( + new ClientEncryptionPolicy(Arrays.asList(ssn, diagnosis))); +database.createEncryptionContainerAsync(containerProperties).block(); +``` + +Reads and writes stay the same; the SDK encrypts and decrypts the declared paths transparently: + +```csharp +// Stored as ciphertext, returned decrypted to callers holding key access +await container.CreateItemAsync(patient, new PartitionKey(patient.HospitalId)); +``` + +Querying a deterministically encrypted field requires the parameter to be encrypted too: + +```csharp +var query = new QueryDefinition("SELECT * FROM c WHERE c.ssn = @ssn"); +await query.AddParameterAsync("@ssn", "123-45-6789", "/ssn"); +``` + +**Choosing an encryption type per field:** + +- Deterministic: the same plaintext always produces the same ciphertext, so equality lookups work. Use for fields you must search by, like an SSN. +- Randomized: stronger protection, no querying at all. Use for fields you only read back, like a diagnosis or card number. +- No encryption: anything you index, sort, aggregate, or use as a partition key. + +**How this differs from encryption at rest:** + +- Encryption at rest is always on, managed by the service, and protects disks and backups. It does nothing against a leaked key or an over-permissioned reader, because data is decrypted for every authorized request. +- Always Encrypted is opt-in per field, the keys live in your Key Vault, and ciphertext is all the service ever holds. Revoking key access makes the fields unreadable even for the account owner. + +Reference: [Use client-side encryption with Always Encrypted for Azure Cosmos DB](https://learn.microsoft.com/azure/cosmos-db/nosql/how-to-always-encrypted)