Gap-7: Functions Diagnose (B → A) — Trigger + Durable troubleshooting

plugin/skills/azure-diagnostics/references/functions/README.md

@@ -78,3 +78,10 @@ az rest --method get \
 
 Compare deployment timestamps against when errors started appearing in App Insights to identify if a deployment caused the issue.
 
+---
+
+## Additional References
+
+- [Trigger-Specific Troubleshooting](triggers.md) — HTTP, Timer, Queue, Blob, Service Bus, and Cosmos DB trigger diagnostics
+- [Durable Functions Troubleshooting](durable.md) — Stuck orchestrations, non-determinism, task hub conflicts, and retry/timer patterns
+

plugin/skills/azure-diagnostics/references/functions/durable.md

@@ -0,0 +1,183 @@
+# Durable Functions Troubleshooting
+
+## Common Issues Matrix
+
+| Symptom | Likely Cause | Quick Fix |
+|---------|--------------|-----------|
+| Orchestration stuck in "Running" | Activity function failed or hung | Check task hub history; terminate and purge if needed |
+| Non-deterministic orchestration error | Code uses `DateTime.Now`, `Guid.NewGuid()`, or random | Replace with deterministic context APIs (`IDurableOrchestrationContext` in-process / `TaskOrchestrationContext` isolated) |
+| Task hub conflicts | Multiple apps sharing same hub | Set unique `hubName` per app in host.json |
+| Fan-out never completes | One activity silently failed | Query instance status for sub-orchestrations; check `exceptions` table |
+| Replay causes side effects | I/O in orchestrator function | Move all I/O into activity functions |
+
+---
+
+## Stuck Orchestrations
+
+**Diagnose:**
+```bash
+# Query orchestration instances by status
+func durable get-instances --connection-string-setting AzureWebJobsStorage \
+  --task-hub-name TASKHUB --runtime-status Running --top 10
+
+# Check specific instance history
+func durable get-runtime-status --id INSTANCE_ID \
+  --connection-string-setting AzureWebJobsStorage \
+  --task-hub-name TASKHUB --show-history
+```
+
+**KQL — Stuck orchestrations:**
+```kql
+traces
+| where timestamp > ago(24h)
+| where message contains "orchestration" and
+  (message contains "stuck" or message contains "timeout" or message contains "failed")
+| project timestamp, operation_Name, message
+| order by timestamp desc
+```
+
+**Fix — Terminate and purge:**
+```bash
+# Terminate a stuck instance
+func durable terminate --id INSTANCE_ID --reason "Manual termination - stuck" \
+  --connection-string-setting AzureWebJobsStorage --task-hub-name TASKHUB
+
+# Purge completed/terminated instances older than 7 days
+# Compute cutoff timestamp: try GNU date first, then BSD/macOS date
+if CUTOFF=$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ 2>/dev/null); then
+  :
+else
+  CUTOFF=$(date -u -v-7d +%Y-%m-%dT%H:%M:%SZ)
+fi
+
+func durable purge-history --connection-string-setting AzureWebJobsStorage \
+  --task-hub-name TASKHUB --created-before "$CUTOFF"
+```
+
+> ⚠️ **Warning:** Termination is immediate and non-recoverable. Activity functions already running will complete, but their results are discarded.
+
+---
+
+## Non-Deterministic Code Detection
+
+Orchestrator functions replay from history. Any non-deterministic call breaks replay.
+
+| Anti-Pattern | Why It Breaks | Correct Alternative |
+|-------------|---------------|---------------------|
+| `DateTime.Now` / `DateTime.UtcNow` | Different value on replay | `context.CurrentUtcDateTime` |
+| `Guid.NewGuid()` | Different GUID on replay | `context.NewGuid()` |
+| `Thread.Sleep` / `Task.Delay` | Not replay-safe; uses non-durable timers so waits aren't persisted or deterministic | `context.CreateTimer()` |
+| Direct HTTP calls | Different response on replay | Use activity function or `context.CallHttpAsync()` |
+| Environment variables | May change between replays | Pass config as orchestrator input |
+| Random number generation | Non-deterministic | Generate in activity, pass to orchestrator |
+
+**KQL — Detect non-deterministic errors:**
+```kql
+exceptions
+| where timestamp > ago(24h)
+| where type contains "NonDeterministic" or
+  outerMessage contains "non-deterministic" or
+  (outerMessage contains "orchestrator function" and
+    (outerMessage contains "completed with a different result" or
+      outerMessage contains "completed with a different task count"))
+| project timestamp, operation_Name, type, outerMessage
+| order by timestamp desc
+```
+
+**Diagnose:**
+```bash
+# Check for orchestration replay errors in App Insights
+az monitor app-insights query --apps APPINSIGHTS -g RG \
+  --analytics-query "exceptions | where type contains 'NonDeterministic' | take 10"
+```
+
+---
+
+## Task Hub Conflicts
+
+Multiple function apps sharing the same task hub causes cross-contamination of orchestration state.
+
+**Diagnose:**
+```bash
+# Check if a task hub name override is set via app settings
+az functionapp config appsettings list -n APP -g RG \
+  --query "[?name=='AzureFunctionsJobHost__extensions__durableTask__hubName']" -o table
+
+# If no override is set, the hub name comes from host.json:
+#   - Inspect host.json in your source repo, or
+#   - Use Kudu (Advanced Tools) / zip-deployed content to view host.json in the deployed app
+```
+
+**Fix — Set unique hub names in host.json:**
+```json
+{
+  "version": "2.0",
+  "extensions": {
+    "durableTask": {
+      "hubName": "MyAppTaskHubProd"
+    }
+  }
+}
+```
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Orchestrations appear in wrong app | Shared `hubName` and storage account | Set unique `hubName` per app |
+| Phantom instances after redeploy | Old hub data persists | Purge history or use new hub name |
+| `PartitionNotFoundException` | Hub tables corrupted or deleted | Delete and recreate hub tables in storage |
+
+**Storage tables for a task hub (Azure Storage provider):**
+```bash
+# List task hub tables
+az storage table list --account-name STORAGE \
+  --query "[?starts_with(name, 'TASKHUB')]" --output table
+```
+
+> 💡 **Tip:** For the Azure Storage provider, task hub names must satisfy three constraints: letters and numbers only (no hyphens, underscores, or special characters), the name must start with a letter, and it must stay within the documented length limit for your runtime/provider. The hub name becomes a prefix for multiple tables and queues (`<HubName>History`, `<HubName>Instances`, and related control/work-item queues). See the official [Durable Functions storage provider documentation](https://learn.microsoft.com/azure/azure-functions/durable/durable-functions-storage-providers#azure-storage) for the current authoritative naming rules.
+
+---
+
+## Timer / Retry Pattern Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Timer fires immediately on replay | Using `Task.Delay` instead of durable timer | Replace with `context.CreateTimer(fireAt, cancellationToken)` |
+| Retry storms | `maxNumberOfAttempts` too high with short intervals | Use exponential backoff: `CallActivityWithRetryAsync` with `RetryOptions` |
+| Sub-orchestration timeout | No timeout set on `CallSubOrchestratorAsync` | Wrap in `Task.WhenAny` with `context.CreateTimer` as deadline |
+| Eternal orchestration memory growth | History never purged | Use `ContinueAsNew()` to reset history periodically |
+
+**Retry configuration example:**
+```csharp
+var retryOptions = new RetryOptions(
+    firstRetryInterval: TimeSpan.FromSeconds(5),
+    maxNumberOfAttempts: 3)
+{
+    BackoffCoefficient = 2.0,
+    MaxRetryInterval = TimeSpan.FromMinutes(1)
+};
+
+await context.CallActivityWithRetryAsync("ProcessItem", retryOptions, item);
+```
+
+**KQL — Retry and timer anomalies:**
+```kql
+traces
+| where timestamp > ago(6h)
+| where message contains "retry" or message contains "timer" or message contains "ContinueAsNew"
+| summarize count() by bin(timestamp, 5m), operation_Name
+| order by timestamp desc
+```
+
+**Diagnose excessive history growth:**
+```bash
+# Verify history table has entries (existence check — not a full count)
+az storage entity query --table-name TASKHUBHistory --account-name STORAGE \
+  --num-results 1 --select PartitionKey --query "items[0]"
+
+# To estimate history size for a specific orchestration instance, avoid relying on
+# a single az storage entity query page. Use Storage Explorer, Azure Data Explorer,
+# or a Storage SDK script that follows continuation tokens and accumulates all
+# entities for PartitionKey eq 'INSTANCE_ID'.
+```
+
+> ⚠️ **Warning:** An orchestration with 10,000+ history events will experience significant replay latency. Use `ContinueAsNew()` in long-running orchestrations to keep history size manageable.

plugin/skills/azure-diagnostics/references/functions/triggers.md

@@ -0,0 +1,174 @@
+# Function App Trigger-Specific Troubleshooting
+
+## HTTP Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| 5xx on every request | Function runtime crash | Check `host.json` for misconfig; review App Insights `exceptions` table |
+| 401 Unauthorized | Auth level mismatch | Verify `authLevel` in function.json matches request (function key vs anonymous) |
+| 404 Not Found | Route prefix misconfigured | Check `host.json` `extensions.http.routePrefix` — default is `api` |
+| CORS blocked | Missing allowed origins | `az functionapp cors add -n APP -g RG --allowed-origins "https://DOMAIN"` |
+| 408 / timeout | Long-running execution or HTTP client/gateway idle timeout | Check `functionTimeout` in host.json (execution timeout) and client/front-end idle limits. For long-running work use async patterns (202 + status endpoint or Durable HTTP APIs), or move to Premium for longer sync executions. |
+
+**KQL — HTTP errors by status code:**
+```kql
+requests
+| where timestamp > ago(1h)
+| where toint(resultCode) >= 400
+| summarize count() by resultCode, operation_Name
+| order by count_ desc
+```
+
+**Diagnose:**
+```bash
+# Check function host status
+az functionapp show -n APP -g RG --query "state"
+
+# Test function endpoint directly
+curl -s -w "\n%{http_code}" "https://APP.azurewebsites.net/api/FUNCTION?code=FUNCTION_KEY"
+```
+
+---
+
+## Timer Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Timer never fires | Invalid NCRONTAB expression | Verify the 6-field NCRONTAB format: `{second} {minute} {hour} {day} {month} {day-of-week}` |
+| Timer fires twice | Multiple instances running | Configure singleton behavior via `host.json` (singleton settings) or language-specific singleton/lock attributes so only one instance runs the timer |
+| Missed timer execution | App was stopped / scaled to zero | Enable Always On (`az functionapp config set -n APP -g RG --always-on true`) — requires App Service plan |
+| Timer drift after deploy | Missed schedule catch-up | Ensure `"useMonitor": true` in the timer trigger binding in `function.json` (default) — runtime tracks missed executions in storage |
+
+**Diagnose:**
+```bash
+# Verify timer trigger config
+az functionapp config appsettings list -n APP -g RG \
+  --query "[?name=='AzureWebJobsStorage']" -o table
+
+# Check singleton lock status in storage
+az storage blob list --account-name STORAGE --container-name azure-webjobs-hosts \
+  --prefix "locks/" --output table
+```
+
+> ⚠️ **Warning:** `AzureWebJobsStorage` must be set and valid for timer triggers — the runtime uses it to store schedule status and singleton locks.
+
+---
+
+## Queue Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Messages stuck in queue | Function failing silently | Check `exceptions` table in App Insights |
+| Poison messages | Max dequeue exceeded | Default `maxDequeueCount` is 5; messages go to `QUEUE-poison`. Inspect and reprocess |
+| Queue not processing | Connection string wrong | Verify `AzureWebJobsStorage` or custom connection setting |
+| Duplicate processing | Visibility timeout too short | Increase `visibilityTimeout` in host.json `extensions.queues` |
+
+**KQL — Poison message tracking:**
+```kql
+traces
+| where timestamp > ago(24h)
+| where message contains "poison" or message contains "MaxDequeueCountExceeded"
+| project timestamp, operation_Name, message
+| order by timestamp desc
+```
+
+**Diagnose:**
+```bash
+# Check queue length and poison queue
+az storage message peek --queue-name QUEUE --account-name STORAGE --num-messages 5
+az storage message peek --queue-name QUEUE-poison --account-name STORAGE --num-messages 5
+
+# Check host.json queue config
+az functionapp config appsettings list -n APP -g RG --query "[?name=='AzureWebJobsStorage']"
+```
+
+---
+
+## Blob Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Trigger delayed (minutes) | Blob trigger may be polling-based and incur scan latency | Consider an Event Grid-based blob trigger for lower-latency processing, where supported and configured |
+| Container not found | Wrong connection or name | Verify `connection` and `path` in function.json |
+| Blobs processed multiple times | Blob receipt tracking failure | Check `azure-webjobs-hosts/blobreceipts/` in storage |
+| Large blobs timeout | Consumption plan limits | Stream blobs or use Premium plan for larger payloads |
+
+**Diagnose:**
+```bash
+# Verify storage connection
+az functionapp config appsettings list -n APP -g RG \
+  --query "[?name=='AzureWebJobsStorage' || contains(name, 'BlobStorage')]"
+
+# Check blob receipts container
+az storage container show --name azure-webjobs-hosts --account-name STORAGE
+```
+
+> 💡 **Tip:** For production workloads that need low latency, evaluate Event Grid-based blob triggers (`BlobTrigger` with `source: "EventGrid"` in host.json) and confirm your runtime, extension version, and storage account configuration support them.
+
+---
+
+## Service Bus Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| `MessageLockLostException` | Processing exceeds lock duration | Increase lock duration on queue/subscription, or break into smaller work units |
+| Messages go to dead-letter | Max delivery count exceeded | Check DLQ: `az servicebus queue show -n QUEUE --namespace-name NS -g RG --query "countDetails"` |
+| Session handler errors | `isSessionsEnabled` mismatch | Ensure function.json `isSessionsEnabled` matches the queue/subscription setting |
+| Connection failures | Firewall or managed identity | Verify connection string or identity role (`Azure Service Bus Data Receiver`) |
+
+**KQL — Service Bus processing errors:**
+```kql
+exceptions
+| where timestamp > ago(1h)
+| where type contains "ServiceBus" or outerMessage contains "lock"
+| project timestamp, type, outerMessage, operation_Name
+| order by timestamp desc
+```
+
+**Diagnose:**
+```bash
+# Check dead-letter queue depth
+az servicebus queue show -n QUEUE --namespace-name NS -g RG \
+  --query "{dlqCount:countDetails.deadLetterMessageCount, activeCount:countDetails.activeMessageCount}"
+
+# View dead-letter message contents
+# If your tooling cannot inspect DLQ message bodies directly, use one of the following:
+#   - Service Bus Explorer in the Azure portal (Service Bus namespace -> Queues -> <QUEUE> -> Dead-letter)
+#   - An Azure Service Bus SDK (for example, .NET, Java, Python, or JavaScript) with a receiver scoped to:
+#       "<QUEUE>/$DeadLetterQueue"
+#     and using a "peek" or "receive" operation to inspect messages.
+```
+
+---
+
+## Cosmos DB Trigger Issues
+
+| Symptom | Cause | Fix |
+|---------|-------|-----|
+| Trigger not firing | Lease container missing | Create lease container (default: `leases`) in same database |
+| Change feed lag | Insufficient RU/s on lease container | Increase RU/s on leases container or enable autoscale |
+| Duplicate processing | Multiple apps share lease container | Configure a unique `leaseContainerPrefix` on each Cosmos DB trigger binding per function app, or use separate lease containers |
+| Partial document data | Projection or TTL conflict | Ensure `StartFromBeginning` config and verify no TTL on lease container |
+
+**Diagnose:**
+```bash
+# Verify lease container exists
+az cosmosdb sql container show --account-name COSMOS -g RG \
+  --database-name DB --name leases
+
+# Check function app Cosmos DB connection
+az functionapp config appsettings list -n APP -g RG \
+  --query "[?contains(name, 'CosmosDB') || contains(name, 'cosmos')]"
+```
+
+**KQL — Cosmos DB trigger lag:**
+```kql
+traces
+| where timestamp > ago(1h)
+| where message contains "lease" or message contains "ChangeFeed"
+| project timestamp, message, operation_Name
+| order by timestamp desc
+| take 30
+```
+
+> ⚠️ **Warning:** If the monitored container has high throughput, set `MaxItemsPerInvocation` in host.json to limit batch size and prevent function timeouts.