diff --git a/plugin/skills/azure-diagnostics/references/functions/README.md b/plugin/skills/azure-diagnostics/references/functions/README.md index 0dabbfe52..50874f7ad 100644 --- a/plugin/skills/azure-diagnostics/references/functions/README.md +++ b/plugin/skills/azure-diagnostics/references/functions/README.md @@ -78,3 +78,10 @@ az rest --method get \ Compare deployment timestamps against when errors started appearing in App Insights to identify if a deployment caused the issue. +--- + +## Additional References + +- [Trigger-Specific Troubleshooting](triggers.md) — HTTP, Timer, Queue, Blob, Service Bus, and Cosmos DB trigger diagnostics +- [Durable Functions Troubleshooting](durable.md) — Stuck orchestrations, non-determinism, task hub conflicts, and retry/timer patterns + diff --git a/plugin/skills/azure-diagnostics/references/functions/durable.md b/plugin/skills/azure-diagnostics/references/functions/durable.md new file mode 100644 index 000000000..7751937ea --- /dev/null +++ b/plugin/skills/azure-diagnostics/references/functions/durable.md @@ -0,0 +1,183 @@ +# Durable Functions Troubleshooting + +## Common Issues Matrix + +| Symptom | Likely Cause | Quick Fix | +|---------|--------------|-----------| +| Orchestration stuck in "Running" | Activity function failed or hung | Check task hub history; terminate and purge if needed | +| Non-deterministic orchestration error | Code uses `DateTime.Now`, `Guid.NewGuid()`, or random | Replace with deterministic context APIs (`IDurableOrchestrationContext` in-process / `TaskOrchestrationContext` isolated) | +| Task hub conflicts | Multiple apps sharing same hub | Set unique `hubName` per app in host.json | +| Fan-out never completes | One activity silently failed | Query instance status for sub-orchestrations; check `exceptions` table | +| Replay causes side effects | I/O in orchestrator function | Move all I/O into activity functions | + +--- + +## Stuck Orchestrations + +**Diagnose:** +```bash +# Query orchestration instances by status +func durable get-instances --connection-string-setting AzureWebJobsStorage \ + --task-hub-name TASKHUB --runtime-status Running --top 10 + +# Check specific instance history +func durable get-runtime-status --id INSTANCE_ID \ + --connection-string-setting AzureWebJobsStorage \ + --task-hub-name TASKHUB --show-history +``` + +**KQL — Stuck orchestrations:** +```kql +traces +| where timestamp > ago(24h) +| where message contains "orchestration" and + (message contains "stuck" or message contains "timeout" or message contains "failed") +| project timestamp, operation_Name, message +| order by timestamp desc +``` + +**Fix — Terminate and purge:** +```bash +# Terminate a stuck instance +func durable terminate --id INSTANCE_ID --reason "Manual termination - stuck" \ + --connection-string-setting AzureWebJobsStorage --task-hub-name TASKHUB + +# Purge completed/terminated instances older than 7 days +# Compute cutoff timestamp: try GNU date first, then BSD/macOS date +if CUTOFF=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then + : +else + CUTOFF=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ) +fi + +func durable purge-history --connection-string-setting AzureWebJobsStorage \ + --task-hub-name TASKHUB --created-before "$CUTOFF" +``` + +> ⚠️ **Warning:** Termination is immediate and non-recoverable. Activity functions already running will complete, but their results are discarded. + +--- + +## Non-Deterministic Code Detection + +Orchestrator functions replay from history. Any non-deterministic call breaks replay. + +| Anti-Pattern | Why It Breaks | Correct Alternative | +|-------------|---------------|---------------------| +| `DateTime.Now` / `DateTime.UtcNow` | Different value on replay | `context.CurrentUtcDateTime` | +| `Guid.NewGuid()` | Different GUID on replay | `context.NewGuid()` | +| `Thread.Sleep` / `Task.Delay` | Not replay-safe; uses non-durable timers so waits aren't persisted or deterministic | `context.CreateTimer()` | +| Direct HTTP calls | Different response on replay | Use activity function or `context.CallHttpAsync()` | +| Environment variables | May change between replays | Pass config as orchestrator input | +| Random number generation | Non-deterministic | Generate in activity, pass to orchestrator | + +**KQL — Detect non-deterministic errors:** +```kql +exceptions +| where timestamp > ago(24h) +| where type contains "NonDeterministic" or + outerMessage contains "non-deterministic" or + (outerMessage contains "orchestrator function" and + (outerMessage contains "completed with a different result" or + outerMessage contains "completed with a different task count")) +| project timestamp, operation_Name, type, outerMessage +| order by timestamp desc +``` + +**Diagnose:** +```bash +# Check for orchestration replay errors in App Insights +az monitor app-insights query --apps APPINSIGHTS -g RG \ + --analytics-query "exceptions | where type contains 'NonDeterministic' | take 10" +``` + +--- + +## Task Hub Conflicts + +Multiple function apps sharing the same task hub causes cross-contamination of orchestration state. + +**Diagnose:** +```bash +# Check if a task hub name override is set via app settings +az functionapp config appsettings list -n APP -g RG \ + --query "[?name=='AzureFunctionsJobHost__extensions__durableTask__hubName']" -o table + +# If no override is set, the hub name comes from host.json: +# - Inspect host.json in your source repo, or +# - Use Kudu (Advanced Tools) / zip-deployed content to view host.json in the deployed app +``` + +**Fix — Set unique hub names in host.json:** +```json +{ + "version": "2.0", + "extensions": { + "durableTask": { + "hubName": "MyAppTaskHubProd" + } + } +} +``` + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Orchestrations appear in wrong app | Shared `hubName` and storage account | Set unique `hubName` per app | +| Phantom instances after redeploy | Old hub data persists | Purge history or use new hub name | +| `PartitionNotFoundException` | Hub tables corrupted or deleted | Delete and recreate hub tables in storage | + +**Storage tables for a task hub (Azure Storage provider):** +```bash +# List task hub tables +az storage table list --account-name STORAGE \ + --query "[?starts_with(name, 'TASKHUB')]" --output table +``` + +> 💡 **Tip:** For the Azure Storage provider, task hub names must satisfy three constraints: letters and numbers only (no hyphens, underscores, or special characters), the name must start with a letter, and it must stay within the documented length limit for your runtime/provider. The hub name becomes a prefix for multiple tables and queues (`History`, `Instances`, and related control/work-item queues). See the official [Durable Functions storage provider documentation](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-storage-providers#azure-storage) for the current authoritative naming rules. + +--- + +## Timer / Retry Pattern Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Timer fires immediately on replay | Using `Task.Delay` instead of durable timer | Replace with `context.CreateTimer(fireAt, cancellationToken)` | +| Retry storms | `maxNumberOfAttempts` too high with short intervals | Use exponential backoff: `CallActivityWithRetryAsync` with `RetryOptions` | +| Sub-orchestration timeout | No timeout set on `CallSubOrchestratorAsync` | Wrap in `Task.WhenAny` with `context.CreateTimer` as deadline | +| Eternal orchestration memory growth | History never purged | Use `ContinueAsNew()` to reset history periodically | + +**Retry configuration example:** +```csharp +var retryOptions = new RetryOptions( + firstRetryInterval: TimeSpan.FromSeconds(5), + maxNumberOfAttempts: 3) +{ + BackoffCoefficient = 2.0, + MaxRetryInterval = TimeSpan.FromMinutes(1) +}; + +await context.CallActivityWithRetryAsync("ProcessItem", retryOptions, item); +``` + +**KQL — Retry and timer anomalies:** +```kql +traces +| where timestamp > ago(6h) +| where message contains "retry" or message contains "timer" or message contains "ContinueAsNew" +| summarize count() by bin(timestamp, 5m), operation_Name +| order by timestamp desc +``` + +**Diagnose excessive history growth:** +```bash +# Verify history table has entries (existence check — not a full count) +az storage entity query --table-name TASKHUBHistory --account-name STORAGE \ + --num-results 1 --select PartitionKey --query "items[0]" + +# To estimate history size for a specific orchestration instance, avoid relying on +# a single az storage entity query page. Use Storage Explorer, Azure Data Explorer, +# or a Storage SDK script that follows continuation tokens and accumulates all +# entities for PartitionKey eq 'INSTANCE_ID'. +``` + +> ⚠️ **Warning:** An orchestration with 10,000+ history events will experience significant replay latency. Use `ContinueAsNew()` in long-running orchestrations to keep history size manageable. diff --git a/plugin/skills/azure-diagnostics/references/functions/triggers.md b/plugin/skills/azure-diagnostics/references/functions/triggers.md new file mode 100644 index 000000000..f06f69d83 --- /dev/null +++ b/plugin/skills/azure-diagnostics/references/functions/triggers.md @@ -0,0 +1,174 @@ +# Function App Trigger-Specific Troubleshooting + +## HTTP Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| 5xx on every request | Function runtime crash | Check `host.json` for misconfig; review App Insights `exceptions` table | +| 401 Unauthorized | Auth level mismatch | Verify `authLevel` in function.json matches request (function key vs anonymous) | +| 404 Not Found | Route prefix misconfigured | Check `host.json` `extensions.http.routePrefix` — default is `api` | +| CORS blocked | Missing allowed origins | `az functionapp cors add -n APP -g RG --allowed-origins "https://DOMAIN"` | +| 408 / timeout | Long-running execution or HTTP client/gateway idle timeout | Check `functionTimeout` in host.json (execution timeout) and client/front-end idle limits. For long-running work use async patterns (202 + status endpoint or Durable HTTP APIs), or move to Premium for longer sync executions. | + +**KQL — HTTP errors by status code:** +```kql +requests +| where timestamp > ago(1h) +| where toint(resultCode) >= 400 +| summarize count() by resultCode, operation_Name +| order by count_ desc +``` + +**Diagnose:** +```bash +# Check function host status +az functionapp show -n APP -g RG --query "state" + +# Test function endpoint directly +curl -s -w "\n%{http_code}" "https://APP.azurewebsites.net/api/FUNCTION?code=FUNCTION_KEY" +``` + +--- + +## Timer Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Timer never fires | Invalid NCRONTAB expression | Verify the 6-field NCRONTAB format: `{second} {minute} {hour} {day} {month} {day-of-week}` | +| Timer fires twice | Multiple instances running | Configure singleton behavior via `host.json` (singleton settings) or language-specific singleton/lock attributes so only one instance runs the timer | +| Missed timer execution | App was stopped / scaled to zero | Enable Always On (`az functionapp config set -n APP -g RG --always-on true`) — requires App Service plan | +| Timer drift after deploy | Missed schedule catch-up | Ensure `"useMonitor": true` in the timer trigger binding in `function.json` (default) — runtime tracks missed executions in storage | + +**Diagnose:** +```bash +# Verify timer trigger config +az functionapp config appsettings list -n APP -g RG \ + --query "[?name=='AzureWebJobsStorage']" -o table + +# Check singleton lock status in storage +az storage blob list --account-name STORAGE --container-name azure-webjobs-hosts \ + --prefix "locks/" --output table +``` + +> ⚠️ **Warning:** `AzureWebJobsStorage` must be set and valid for timer triggers — the runtime uses it to store schedule status and singleton locks. + +--- + +## Queue Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Messages stuck in queue | Function failing silently | Check `exceptions` table in App Insights | +| Poison messages | Max dequeue exceeded | Default `maxDequeueCount` is 5; messages go to `QUEUE-poison`. Inspect and reprocess | +| Queue not processing | Connection string wrong | Verify `AzureWebJobsStorage` or custom connection setting | +| Duplicate processing | Visibility timeout too short | Increase `visibilityTimeout` in host.json `extensions.queues` | + +**KQL — Poison message tracking:** +```kql +traces +| where timestamp > ago(24h) +| where message contains "poison" or message contains "MaxDequeueCountExceeded" +| project timestamp, operation_Name, message +| order by timestamp desc +``` + +**Diagnose:** +```bash +# Check queue length and poison queue +az storage message peek --queue-name QUEUE --account-name STORAGE --num-messages 5 +az storage message peek --queue-name QUEUE-poison --account-name STORAGE --num-messages 5 + +# Check host.json queue config +az functionapp config appsettings list -n APP -g RG --query "[?name=='AzureWebJobsStorage']" +``` + +--- + +## Blob Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Trigger delayed (minutes) | Blob trigger may be polling-based and incur scan latency | Consider an Event Grid-based blob trigger for lower-latency processing, where supported and configured | +| Container not found | Wrong connection or name | Verify `connection` and `path` in function.json | +| Blobs processed multiple times | Blob receipt tracking failure | Check `azure-webjobs-hosts/blobreceipts/` in storage | +| Large blobs timeout | Consumption plan limits | Stream blobs or use Premium plan for larger payloads | + +**Diagnose:** +```bash +# Verify storage connection +az functionapp config appsettings list -n APP -g RG \ + --query "[?name=='AzureWebJobsStorage' || contains(name, 'BlobStorage')]" + +# Check blob receipts container +az storage container show --name azure-webjobs-hosts --account-name STORAGE +``` + +> 💡 **Tip:** For production workloads that need low latency, evaluate Event Grid-based blob triggers (`BlobTrigger` with `source: "EventGrid"` in host.json) and confirm your runtime, extension version, and storage account configuration support them. + +--- + +## Service Bus Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| `MessageLockLostException` | Processing exceeds lock duration | Increase lock duration on queue/subscription, or break into smaller work units | +| Messages go to dead-letter | Max delivery count exceeded | Check DLQ: `az servicebus queue show -n QUEUE --namespace-name NS -g RG --query "countDetails"` | +| Session handler errors | `isSessionsEnabled` mismatch | Ensure function.json `isSessionsEnabled` matches the queue/subscription setting | +| Connection failures | Firewall or managed identity | Verify connection string or identity role (`Azure Service Bus Data Receiver`) | + +**KQL — Service Bus processing errors:** +```kql +exceptions +| where timestamp > ago(1h) +| where type contains "ServiceBus" or outerMessage contains "lock" +| project timestamp, type, outerMessage, operation_Name +| order by timestamp desc +``` + +**Diagnose:** +```bash +# Check dead-letter queue depth +az servicebus queue show -n QUEUE --namespace-name NS -g RG \ + --query "{dlqCount:countDetails.deadLetterMessageCount, activeCount:countDetails.activeMessageCount}" + +# View dead-letter message contents +# If your tooling cannot inspect DLQ message bodies directly, use one of the following: +# - Service Bus Explorer in the Azure portal (Service Bus namespace -> Queues -> -> Dead-letter) +# - An Azure Service Bus SDK (for example, .NET, Java, Python, or JavaScript) with a receiver scoped to: +# "/$DeadLetterQueue" +# and using a "peek" or "receive" operation to inspect messages. +``` + +--- + +## Cosmos DB Trigger Issues + +| Symptom | Cause | Fix | +|---------|-------|-----| +| Trigger not firing | Lease container missing | Create lease container (default: `leases`) in same database | +| Change feed lag | Insufficient RU/s on lease container | Increase RU/s on leases container or enable autoscale | +| Duplicate processing | Multiple apps share lease container | Configure a unique `leaseContainerPrefix` on each Cosmos DB trigger binding per function app, or use separate lease containers | +| Partial document data | Projection or TTL conflict | Ensure `StartFromBeginning` config and verify no TTL on lease container | + +**Diagnose:** +```bash +# Verify lease container exists +az cosmosdb sql container show --account-name COSMOS -g RG \ + --database-name DB --name leases + +# Check function app Cosmos DB connection +az functionapp config appsettings list -n APP -g RG \ + --query "[?contains(name, 'CosmosDB') || contains(name, 'cosmos')]" +``` + +**KQL — Cosmos DB trigger lag:** +```kql +traces +| where timestamp > ago(1h) +| where message contains "lease" or message contains "ChangeFeed" +| project timestamp, message, operation_Name +| order by timestamp desc +| take 30 +``` + +> ⚠️ **Warning:** If the monitored container has high throughput, set `MaxItemsPerInvocation` in host.json to limit batch size and prevent function timeouts.