Add SDK Worker alerting guidance and triage pages

[austonli]

Temporal SDKs emit metrics that expose failure modes the Temporal Service can't surface on your behalf — Workflow code failing on replay, Workers that stop polling, Task slots that never free up, Local Activities outliving the Workflow Task heartbeat window. Docs currently document the metrics (SDK metrics reference) and some symptoms (Performance bottlenecks), but there's no recommended alert set, no thresholds tied to alerts, and no per-failure-mode triage.

This adds that.

What's here

Page	Contents
best-practices/sdk-worker-alerting	Recommended alert set — 22 rows with tag filters, thresholds, for durations, and default severities — plus a five-alert starting subset and a table of conditions that fire during normal operation. Entry point for the other three.
troubleshooting/sdk-request-failures	NOT_FOUND on respond operations and Activity heartbeat, RESOURCE_EXHAUSTED on user-facing / respond / poll operations, UNIMPLEMENTED and INTERNAL, request latency
troubleshooting/sdk-worker-capacity	Task slots exhausted, all pollers disconnected, Task completions at zero, schedule-to-start latency, Sticky Execution cache empty under load
troubleshooting/sdk-execution-failures	Non-determinism error, gRPC message too large, Workflow Task failures and latency, Activity failures, unregistered Activity invocation, Local Activity latency

Each failure mode is an anchored heading so alert notifications can deep-link straight to the relevant triage.

Three existing files are updated:

• references/sdk-metrics — documents status_code on the request-failure metrics, GrpcMessageTooLarge as a failure_reason, and two previously undocumented Go metrics (below). Also corrects the poller_type value sticky_workflow_task → workflow_sticky_task, which disagreed with both SDKs.
• troubleshooting/blob-size-limit-error — the Workflow Task case said the Workflow gets stuck retrying. Current server behavior is termination: respondworkflowtaskfailed/api.go calls TerminateWorkflow on WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE. The Activity case, which does retry, is unchanged.
• bin/check-metrics-reference.js and bin/metrics-baseline.json — allowlist entries for the new tags, and removal of two baseline entries now that the metrics are documented.

Source and attribution

Built on Tiho's SDK alerting work — the alert set, thresholds, severities, and triage sequences are his. Duncan is working with him on contributing the underlying rulesets directly.

Coverage against his essential set: 23 of 24 alerts. Two pairs are merged into single rows where they share a failure mode and triage path (NOT_FOUND on Workflow Task and Activity respond; Workflow Task and Activity completions at zero). One is dropped — see below. His alert #2c, RESOURCE_EXHAUSTED on poll operations, is promoted out of the full 36-alert index because three triage paths depend on it.

Two follow-ups once the rulesets land:

1. A reference page with all 36 alerts and per-SDK PromQL, generated from the YAMLs via snipsync rather than hand-maintained.
2. Alert-import sections on Prometheus Grafana setup and Monitor Temporal Platform metrics, which currently dead-end at Grafana's own alerting docs.

PromQL is deliberately omitted here for that reason — it belongs in the generated reference, not duplicated in prose that will drift. The alert tables give the metric plus tag filters, which is what a reader needs to write the query for their own reporter.

Corrections against SDK and server source

Several claims in the source material don't survive verification. Corrected here, and worth fixing in that repo before the YAMLs are contributed:

Claim	Verified behavior	Source
status_code="NotFound" (PascalCase, Go)	UPPER_SNAKE_CASE in every SDK	Go canonicalString returns "NOT_FOUND"; Java uses code.name()
worker_type=WorkflowWorker on num_pollers, in 11 runbooks	tag is poller_type, values workflow_task / workflow_sticky_task / activity_task / nexus_task	sdk-go constants.go, sdk-java PollerTypeMetricsTag
Go cache option maxWorkflowCacheSize	worker.SetStickyWorkflowCacheSize(int), default 10,000	sdk-go worker/worker.go
Alert on local_activity_total_execution_latency	Not defined in Go, Java, Python, or TypeScript	absent from all four
request_failure rate "per task queue"	No task_queue tag on that metric	sdk-go grpc.go
Sticky cache alert on size == 0	Gauge is current occupancy, not configured capacity — zero is normal on an idle Worker	sdk-go internal_task_handlers.go
Workflow Task failure threshold	README table says 10/s, runbook body says 20/s	internal contradiction

temporal_local_activity_total_execution_latency has no alert here. Its intent — a retry chain accumulating past the heartbeat timeout — is covered as a triage step using local_activity_execution_failed instead.

A better throttling signal, now documented

The Go SDK emits request_resource_exhausted and long_request_resource_exhausted with a cause tag carrying the ResourceExhaustedCause enum. Neither was documented. Both sat in metrics-baseline.json with an empty note, which that file defines as "not yet reviewed; fill it in or document the metric and remove the entry."

This matters because all three RESOURCE_EXHAUSTED sections open triage with "identify the throttle cause," and without this metric a Cloud reader had no way to do it — the only concrete instruction was in a self-hosted admonition pointing at a server dashboard. The cause distinguishes a rate limit you can ask to have raised from an overloaded Service where a higher limit makes things worse.

On usability as an on-call artifact

• Alert tables carry tag filters, for durations, and grouping guidance, so a reader can build each alert without opening another page.
• A five-alert starting subset, since 22 is too many to adopt at once.
• A table of conditions that fire during normal operation — deploys, rolling restarts, scale-down, idle Task Queues — because most of these are binary conditions where the for duration does all the work.
• Two alerts reframed so they don't fire on healthy Workers: the sticky cache gauge (occupancy, not capacity) and the Task-completion rate (a zero rate is consistent with an idle Task Queue). Both now require corroborating evidence of load.
• Severity is a default routing suggestion rather than a property of the metric, with the two cases where it doesn't fit spelled out.

Notes for review

• Thresholds want a second opinion from the SDK teams. They're starting points and the pages say so, but published values read as recommendations. The consequential ones: p99 above 5s for Workflow Task schedule-to-start, 10s for Workflow Task execution latency, rate above 10/s for execution failures, 30m for Local Activity latency.
• Overlapping thresholds are cross-referenced, not reconciled. Performance bottlenecks gives P95 above 1s for the schedule-to-start metrics; this PR gives p99 above 5s. Both sections now explain the difference — investigation threshold versus paging threshold — and link Worker health, which already supplies Prometheus queries for the same two metrics. Aligning on single values is a follow-up that needs the same SDK-team input as the thresholds.
• Deployment-neutral. SDK metrics are identical on Cloud and self-hosted, so triage applies to both; steps that route to server dashboards or dynamic config are in self-hosted-only admonitions.
• Headings are symptom noun phrases rather than the infinitive form STYLE.md prescribes. Deliberate — the IA guide says troubleshooting pages should lead with the symptom, and performance-bottlenecks uses the same pattern — but flagging it as a known deviation.
• performance-bottlenecks still overlaps these pages structurally. Left alone to keep the diff reviewable; worth consolidating rather than maintaining two symptom lists.

Verified: yarn build exits 0 with onBrokenLinks and onBrokenAnchors both set to throw, so link and anchor integrity is enforced by the build. check:metrics, check:metrics:sdks, and check:orphans all pass.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
temporal-documentation	Ready	Preview	Aug 5, 2026 6:40pm

[github-actions]

📖 Docs PR preview links

• Best Practices
  • Worker alerting
• References
  • SDK metrics
• Troubleshooting
  • Execution failures
  • Message size limit errors
  • Request failures
  • Troubleshooting
  • Worker capacity

[Duncanma]

Hey @austonli, this is awesome, appreciate you creating this from @tsurdilo's content. One thoughts, do we need to call the pages "SDK Worker..." ... so could "How to alert on Temporal SDK Worker metrics" just be "How to alert on Worker metrics". And "How to troubleshoot SDK request failures" could be "Request failures"