From be13cae64b906dc6ab270968b12ac98552e3d4a7 Mon Sep 17 00:00:00 2001 From: Auston Li Date: Tue, 4 Aug 2026 14:21:23 -0700 Subject: [PATCH 1/5] Add SDK Worker alerting guidance and triage pages Adds a recommended alert set for Temporal SDK Worker metrics and three troubleshooting pages covering the failure modes it alerts on. - best-practices/sdk-worker-alerting: recommended alert set with starting thresholds, grouped by request failures, Worker capacity, and execution failures - troubleshooting/sdk-request-failures: gRPC status codes and request latency on calls to the Temporal Service - troubleshooting/sdk-worker-capacity: Task slots, pollers, Task completions, schedule-to-start latency, Sticky Execution cache - troubleshooting/sdk-execution-failures: non-determinism errors, oversized Workflow Task responses, Workflow and Activity code failures, Local Activity latency Pages are deployment-neutral, with self-hosted-only triage steps in admonitions. PromQL is intentionally omitted pending a published ruleset. Co-Authored-By: Claude Opus 5 --- docs/best-practices/sdk-worker-alerting.mdx | 92 +++++++++ docs/troubleshooting/index.mdx | 3 + .../sdk-execution-failures.mdx | 163 +++++++++++++++ docs/troubleshooting/sdk-request-failures.mdx | 181 +++++++++++++++++ docs/troubleshooting/sdk-worker-capacity.mdx | 189 ++++++++++++++++++ sidebars.js | 4 + 6 files changed, 632 insertions(+) create mode 100644 docs/best-practices/sdk-worker-alerting.mdx create mode 100644 docs/troubleshooting/sdk-execution-failures.mdx create mode 100644 docs/troubleshooting/sdk-request-failures.mdx create mode 100644 docs/troubleshooting/sdk-worker-capacity.mdx diff --git a/docs/best-practices/sdk-worker-alerting.mdx b/docs/best-practices/sdk-worker-alerting.mdx new file mode 100644 index 0000000000..8f7d8d388a --- /dev/null +++ b/docs/best-practices/sdk-worker-alerting.mdx @@ -0,0 +1,92 @@ +--- +id: sdk-worker-alerting +title: How to alert on Temporal SDK Worker metrics +sidebar_label: SDK Worker alerting +description: A recommended alert set for Temporal SDK Workers, with thresholds and links to triage guidance +toc_max_heading_level: 4 +keywords: + - sdk metrics + - worker metrics + - alerting + - monitoring temporal metrics + - observability +tags: + - Metrics + - Observability + - Workers +--- + +Temporal SDKs emit metrics from your Worker processes that expose failure modes the Temporal Service cannot see on your behalf: Workflow code that fails on replay, Workers that stop polling, Task slots that never free up, and Local Activities that outlive their heartbeat window. + +This page recommends a set of alerts covering those failure modes, with a starting threshold for each. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For metric definitions and label sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). For Worker configuration guidance, see [Worker deployment and performance](/best-practices/worker). + +## Choose your thresholds {/* #choose-your-thresholds */} + +Every threshold on this page is a starting point, not a service level objective. + +A high-throughput Task Queue, a latency-sensitive Namespace, and a bursty batch workload each want different values. Two things to tune per alert: + +- **The threshold** determines what counts as unhealthy. Set it against observed p99 during a known-good period, not against the value listed here. +- **The `for` duration** determines how long the condition must hold before the alert fires. Shorter durations detect problems faster and fire more often on transient spikes. Longer durations suppress noise and delay detection. + +Alerts on binary conditions — any occurrence of a gRPC status code, a gauge reaching zero — need no threshold tuning. Only the `for` duration matters, and only to filter out deploys and rolling restarts. + +## Recommended alert set {/* #recommended-alert-set */} + +### Request failures {/* #request-failures */} + +These fire on gRPC responses from the Temporal Service to your Worker or Client. Triage guidance: [SDK request failures](/troubleshooting/sdk-request-failures). + +| Failure mode | Metric | Threshold | Severity | +| --- | --- | --- | --- | +| [NOT_FOUND on respond operations](/troubleshooting/sdk-request-failures#not_found-on-respond-operations) | `temporal_request_failure` | Any occurrence | Critical | +| [NOT_FOUND on Activity heartbeat](/troubleshooting/sdk-request-failures#not_found-on-activity-heartbeat) | `temporal_request_failure` | Any occurrence | Warning | +| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) | `temporal_request_failure` | Any occurrence | Critical | +| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) | `temporal_request_failure` | Any occurrence | Critical | +| [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `temporal_request_failure` | Any occurrence | Critical | +| [Request latency high on user-facing operations](/troubleshooting/sdk-request-failures#request-latency-high-on-user-facing-operations) | `temporal_request_latency` | p99 above 2s | Critical | + +### Worker capacity {/* #worker-capacity */} + +These fire when Workers stop keeping up with the Task Queue. Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). + +| Failure mode | Metric | Threshold | Severity | +| --- | --- | --- | --- | +| [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `temporal_worker_task_slots_available` | Reaches 0 | Critical | +| [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `temporal_num_pollers` | Reaches 0 | Critical | +| [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `temporal_request` | Rate reaches 0 per Task Queue | Critical | +| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `temporal_workflow_task_schedule_to_start_latency` | p99 above 5s, and again above 30m | Critical | +| [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `temporal_activity_schedule_to_start_latency` | p99 above 30m | Critical | +| [Sticky cache disabled](/troubleshooting/sdk-worker-capacity#sticky-cache-disabled) | `temporal_sticky_cache_size` | Reaches 0 | Warning | + +### Execution failures {/* #execution-failures */} + +These fire on Workflow and Activity code failing on the Worker. Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failures). + +| Failure mode | Metric | Threshold | Severity | +| --- | --- | --- | --- | +| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `temporal_workflow_task_execution_failed` | Any occurrence | Critical | +| [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `temporal_workflow_task_execution_failed` | Any occurrence | Critical | +| [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `temporal_workflow_task_execution_failed` | Rate above 10/s | Warning | +| [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `temporal_workflow_task_execution_latency` | p99 above 10s | Critical | +| [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `temporal_activity_execution_failed` | Rate above 10/s | Warning | +| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `temporal_local_activity_execution_latency`, `temporal_local_activity_total_execution_latency` | p99 above 30m | Critical | + +## Set up the alerts {/* #set-up-the-alerts */} + +Before you can alert on these metrics, your Workers must be emitting them and your monitoring system must be scraping them: + +- **Temporal Cloud** — [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). +- **Self-hosted** — [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). + +Metric names vary by SDK and metrics reporter. Counters carry a `_total` suffix and histograms a `_seconds_bucket` suffix when scraped through Prometheus. The `status_code` label is `PascalCase` in the Go SDK (`NotFound`) and `UPPER_SNAKE_CASE` in the Java and Core SDKs (`NOT_FOUND`). Confirm the exact names in your own metrics endpoint before writing queries. + +## Route alerts by severity {/* #route-alerts-by-severity */} + +Alerts marked Critical indicate that Workflow Executions are not progressing, or are progressing with data loss or duplicate side effects. Route these to a paging channel. + +Alerts marked Warning indicate degradation that has not yet stopped progress: elevated failure churn, a misconfigured cache, an Activity heartbeat that is timing out and retrying. Route these to a review channel and address them before they escalate. + +Several of these alerts have causal relationships — Task slots exhausted drives pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. When multiple fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/troubleshooting/index.mdx b/docs/troubleshooting/index.mdx index 6a3bcad470..0308c5f68d 100644 --- a/docs/troubleshooting/index.mdx +++ b/docs/troubleshooting/index.mdx @@ -24,4 +24,7 @@ Our troubleshooting guides are designed to help you quickly identify and resolve - [Troubleshoot the Failed Reaching Server Error](/troubleshooting/last-connection-error): The message "Failed reaching server: last connection error" often happens due to an expired TLS certificate or during the Server startup process when Client requests reach the Server before roles are fully initialized. - [Troubleshoot missed Schedule Actions](/troubleshooting/schedule-missed-actions): When a Schedule does not fire at its expected time, alert on the missed catchup window metric, then narrow down to the affected Schedule with `ListSchedules` and `DescribeSchedule`. - [Troubleshoot Serverless Workers](/troubleshooting/serverless-workers): Diagnose issues with Serverless Workers by tracing the flow from Task Queue to Worker execution, on [AWS Lambda](/troubleshooting/serverless-workers/aws-lambda) or [GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run). +- [Troubleshoot SDK request failures](/troubleshooting/sdk-request-failures): Diagnose gRPC failures and elevated latency on requests from Workers and Clients to the Temporal Service, including NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses. +- [Troubleshoot SDK Worker capacity](/troubleshooting/sdk-worker-capacity): Diagnose exhausted Task slots, disconnected pollers, Task completions dropping to zero, elevated schedule-to-start latency, and a disabled Sticky Execution cache. +- [Troubleshoot SDK execution failures](/troubleshooting/sdk-execution-failures): Diagnose non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout. - [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows): Recover pinned Workflows that have failed or are stuck retrying tasks after rolling out a faulty Worker Deployment Version. diff --git a/docs/troubleshooting/sdk-execution-failures.mdx b/docs/troubleshooting/sdk-execution-failures.mdx new file mode 100644 index 0000000000..8b0193ae16 --- /dev/null +++ b/docs/troubleshooting/sdk-execution-failures.mdx @@ -0,0 +1,163 @@ +--- +id: sdk-execution-failures +title: How to troubleshoot SDK execution failures +sidebar_label: SDK execution failures +description: Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency +toc_max_heading_level: 4 +keywords: + - non determinism error + - workflow task failed + - activity execution failed + - local activity + - grpc message too large +tags: + - Metrics + - Observability + - Troubleshooting +--- + +This guide covers failures that occur while your Workflow and Activity code is executing on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that run past the Workflow Task heartbeat timeout. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). + +`temporal_workflow_task_execution_failed` carries a `failure_reason` label. The reason determines what the Temporal Service does next, and the difference matters: some reasons cause indefinite retries, one causes immediate termination. + +## Non-determinism error {/* #non-determinism-error */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError` + +Replay produced a different command sequence than the one recorded in Event History. The Worker detected that the Workflow code it is running does not match the commands the Workflow Execution has already produced. + +### Why it matters {/* #why-nde-matters */} + +Affected Executions are not progressing. The Temporal Service retries the Workflow Task continuously, adding pressure to your Workflow Workers, and by default the Executions stay in Running status — prolonging their end-to-end time indefinitely. A non-determinism error does not resolve on its own. + +### Triage {/* #triage-nde */} + +1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures. +1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause. +1. **Determine whether this is a code change or a deploy artifact.** Common causes: + - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code. + - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. + - Changed Activity or Timer parameters in existing Workflow code without versioning. +1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Workflow versioning](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). +1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue. + +## gRPC message too large {/* #grpc-message-too-large */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge` + +The Workflow Task response payload exceeded the gRPC message size limit. The Worker attempted `RespondWorkflowTaskCompleted` and the response was rejected — by the gRPC library on the SDK side, by a proxy or load balancer in the path, or by the gRPC library on the Temporal Service side on receive. + +Because the Temporal Service never saw the original request, the SDK sends a follow-up `RespondWorkflowTaskFailed` with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. The Service then terminates the Workflow Execution. + +### Why it matters {/* #why-message-too-large-matters */} + +Affected Executions are terminated immediately and permanently, with `TERMINATED` status and no retry. Any in-progress work in those Executions is lost, and they must be restarted manually. + +This is the one `failure_reason` on this page that ends Executions rather than retrying them. + +### Triage {/* #triage-message-too-large */} + +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History. +1. **Find what is oversized.** The fix depends entirely on which part of the response is too large: + - **Oversized Activity inputs or outputs.** Move large payloads out of band — store them in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern. + - **Accumulated Signals or Updates.** A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals. + - **Too many commands in one response.** A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks. +1. **Fix and deploy before restarting anything.** Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI. + +:::note Self-hosted Temporal Service + +Check the Workflow terminate rate on your server dashboard — a spike alongside this metric confirms Executions are being terminated at volume. + +::: + +## Workflow Task execution failures elevated {/* #workflow-task-execution-failures-elevated */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError` + +Sustained Workflow Task failures from unhandled exceptions and panics in Workflow code that the SDK catches and reports. This covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. + +### Why it matters {/* #why-workflow-error-matters */} + +The Temporal Service retries the Workflow Task. If the error is deterministic and reproduces on every replay, the Execution is stuck retrying indefinitely, consuming Worker capacity and staying in a permanently unhealthy state. + +Unlike a non-determinism error, there is no ceiling on how bad this gets: at high rates the retry pressure saturates Workflow Worker slots and affects healthy Executions on the same Task Queue. Unlike `GrpcMessageTooLarge`, the Service does not terminate the Execution, so the impact compounds until you resolve it. + +### Triage {/* #triage-workflow-error */} + +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. +1. **Determine which failure mode this is.** `WorkflowError` covers several: + - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool, caused by `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` being too low for the number of concurrent Executions. New Workflow Tasks are rejected before they can execute. Raise the thread count, and check whether the Worker pool needs to scale out as well. + - **Unhandled exception in Workflow code.** A bug or unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event identifies the error. + - **Data Converter error.** A failure serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and Payload Codec configuration. +1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion often occur together under load, and a CPU-starved Worker completes Workflow Tasks more slowly, accelerating both. +1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running. + +## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */} + +**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) + +Workflow Tasks are taking too long to execute on the Worker. The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks. + +### Why it matters {/* #why-wft-latency-matters */} + +The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue. Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay. + +If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats. Non-idempotent Local Activities produce duplicate side effects with real business impact. + +At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts. + +### Triage {/* #triage-wft-latency */} + +1. **Check replay latency.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-executing Event History rather than running new commands — usually caused by large histories, slow Data Converter execution during replay, or a high cache eviction rate forcing cold replays. +1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache disabled](/troubleshooting/sdk-worker-capacity#sticky-cache-disabled) for the case where the cache is off entirely. +1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker. +1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — the SDK holds the slot until the respond call succeeds, inflating this metric even when the Workflow code finished quickly. + +## Activity execution failures elevated {/* #activity-execution-failures-elevated */} + +**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed) + +Activities are explicitly failing — returning failures rather than timing out — at a sustained rate. + +`ApplicationFailure` instances marked with category `BENIGN` are excluded and do not increment this counter, so this metric tracks unexpected failures only to the extent your application uses benign failures correctly. + +### Why it matters {/* #why-activity-failures-matter */} + +A high failure rate drives a burst of retry Tasks. If Workers cannot keep up with the retry volume, the Activity Task backlog grows — see [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). At scale, sustained retry bursts put significant pressure on Task matching and the underlying database. + +### Triage {/* #triage-activity-failures */} + +1. **Identify which Activity is failing.** The metric carries an `activity_type` label. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. +1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not. +1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying. +1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. +1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`. All SDKs support this and suppress this metric for them, which lets this alert track unexpected failures without per-`activity_type` threshold tuning. + +## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */} + +**Metrics:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency) for a single attempt, and `temporal_local_activity_total_execution_latency` for the full retry chain + +A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes. The SDK sends Workflow Task heartbeats to keep the Task alive while the Local Activity runs, but once the timeout is exceeded the Temporal Service times out the heartbeating Workflow Task. + +Watch both metrics. The single-attempt metric catches one long-running attempt. The total metric catches a retry chain that accumulates past the timeout even when every individual attempt is short — usually a high failure rate paired with an aggressive Retry Policy. + +### Why it matters {/* #why-la-latency-matters */} + +When the Temporal Service times out the heartbeating Workflow Task, it reschedules the Task on the normal Task Queue and the Local Activity re-executes from scratch. Local Activities cannot heartbeat, and their progress is not checkpointed between Workflow Task heartbeats. A non-idempotent Local Activity produces duplicate side effects with real business impact. + +Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly. + +The Local Activity also occupies an executor slot for its entire duration. Several in this state at once can occupy every available slot, blocking new Local Activities from starting. See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#triage-slots-localactivityworker) for `worker_type=LocalActivityWorker`. + +Local Activities are designed for short, fast operations. A single attempt running for 30 minutes is a design problem, not a tuning problem. + +### Triage {/* #triage-la-latency */} + +1. **Identify the affected Local Activity.** The metric carries an `activity_type` label. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids. +1. **Find what it is blocked on.** A Local Activity running this long is almost always blocked on a downstream call — a slow service, a slow query, or a network call with a very long timeout. Fix the dependency, or shorten the timeout on the call so the Local Activity fails fast instead of hanging. +1. **Check the failure rate driving retries.** If the total-latency metric is elevated but single attempts are short, a high failure rate with aggressive retries is accumulating the chain. Fix the underlying failure first. +1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have already timed out heartbeating Workflow Tasks. Check Worker logs for timeout errors and Event History for `WorkflowTaskTimedOut` events. If they are present, Local Activities have already re-executed — verify idempotency and address any duplicate side effects. +1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed. diff --git a/docs/troubleshooting/sdk-request-failures.mdx b/docs/troubleshooting/sdk-request-failures.mdx new file mode 100644 index 0000000000..da69b3c0a8 --- /dev/null +++ b/docs/troubleshooting/sdk-request-failures.mdx @@ -0,0 +1,181 @@ +--- +id: sdk-request-failures +title: How to troubleshoot SDK request failures +sidebar_label: SDK request failures +description: Diagnose gRPC failures and elevated latency on requests from Temporal SDK Workers and Clients +toc_max_heading_level: 4 +keywords: + - sdk metrics + - request failure + - resource exhausted + - grpc errors + - monitoring temporal metrics +tags: + - Metrics + - Observability + - Troubleshooting +--- + +This guide covers gRPC failures and elevated latency on requests that Temporal SDK Workers and Clients make to the Temporal Service. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +Each section names the metric to watch, explains the impact, and walks through triage. For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). + +`temporal_request_failure` increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. `temporal_long_request_failure` covers poll operations and long-poll `GetWorkflowExecutionHistory`. Both carry `namespace`, `operation`, `status_code`, and `task_queue` labels. Severity depends on which status code appears on which operation. + +:::note + +`UpdateWithStartWorkflowExecution` appears in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. + +::: + +## NOT_FOUND on respond operations {/* #not_found-on-respond-operations */} + +**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` + +A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. There are three causes: + +- The task timed out. The Worker ran past the Workflow Task timeout, or past the Activity `startToClose` or `scheduleToClose` timeout, and the Service discarded the in-flight task. +- The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished. +- The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up. + +### Why it matters {/* #why-not_found-matters */} + +The result the Worker just produced was discarded. For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a Sticky Execution cache eviction and a cold replay on the retry. + +Repeated NOT_FOUND on respond operations means Workers are consistently finishing too late. Every discarded result is Worker capacity spent on work that was thrown away, and every rescheduled task adds to Workflow end-to-end latency. + +If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried task. Local Activity results are not written to Event History between Workflow Task heartbeats, so re-execution means running them again. If they are not idempotent, this produces duplicate side effects with real business impact. + +### Triage {/* #triage-not_found */} + +1. **Check task execution latency.** For Workflow Tasks, check [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency). For Activities, check [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause. +1. **Check replay latency.** If Workflow Task execution latency is elevated, check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. High replay latency means the Worker is spending its time re-executing Event History rather than running new commands — check for large histories and slow [Data Converter](/dataconversion) execution during replay. +1. **Check Worker resources.** High CPU on the Worker slows task execution directly. Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays. +1. **Confirm the Workflow Execution is still running.** Check the Execution status in the Temporal UI or with `temporal workflow describe`. If it completed, was terminated, or hit its Run Timeout, the NOT_FOUND is expected and the Execution lifecycle is what to investigate. +1. **Check for Worker restarts.** Frequent restarts lose in-flight Task Tokens. Check pod restart counts in your infrastructure observability stack. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations) below. Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands. + +:::note Self-hosted Temporal Service + +If SDK-side metrics look normal and the Execution was not terminated or timed out, check server-side latency: Frontend Service latency filtered to the affected respond operation, and persistence latency filtered to `UpdateWorkflowExecution`. + +::: + +## NOT_FOUND on Activity heartbeat {/* #not_found-on-activity-heartbeat */} + +**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat` + +A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running. + +Normal Workflow-side cancellation is not a cause. Cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure. + +### Why it matters {/* #why-heartbeat-not_found-matters */} + +If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. The Activity re-executes from scratch on the next attempt, so a non-idempotent Activity produces duplicate side effects. + +Repeated NOT_FOUND on heartbeat calls means the Worker is consistently failing to heartbeat within the configured interval. The Activity will keep timing out on every attempt until the cause is fixed, holding Task slots and generating retry tasks the whole time. + +### Triage {/* #triage-heartbeat-not_found */} + +1. **Compare the heartbeat interval against `heartbeatTimeout`.** The Worker must call heartbeat more frequently than the timeout. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat. +1. **Check Worker CPU.** A CPU-starved Worker slows down between heartbeat calls even when the Activity is making progress. If utilization is consistently high, reduce per-Worker concurrency or scale out horizontally. +1. **Check for throttling on heartbeat calls.** If the Temporal Service is throttling `RecordActivityTaskHeartbeat`, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time. +1. **Check the `startToClose` timeout.** If the Activity has run longer than `startToClose`, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND. Compare [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` against the configured timeout. +1. **Check heartbeat payload size.** The last heartbeat details payload is held in memory for the life of the Activity attempt. Large payloads on high-throughput Activity Workers contribute to memory pressure on the Temporal Service. Store only the minimum progress state needed to resume on retry. + +## RESOURCE_EXHAUSTED on user-facing operations {/* #resource_exhausted-on-user-facing-operations */} + +**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation` + +The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates. The SDK retries these automatically for up to 60 seconds. Beyond that, the call fails and the error propagates to your caller. + +### Why it matters {/* #why-resource_exhausted-matters */} + +These operations are on your application's critical path. Within the retry window, callers experience elevated latency. Past it, calls fail outright and your application must handle the error. + +If it does not, starts and Signals are silently dropped. A dropped start means the Workflow never runs. A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. Log these failures in your application code so you can backfill starts and Signals afterward. + +The Temporal Service throttles these operations last. Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread. + +### Triage {/* #triage-resource_exhausted */} + +1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, or an open circuit breaker are different problems. +1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare current throughput against your Namespace's [service limits](/cloud/limits) and open a support request if you need them raised. +1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure. + +:::note Self-hosted Temporal Service + +Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution` — slow persistence is the most common root cause of cascading throttling. If the cause is a rate limit, `frontend.namespaceRPS` may be set too low for your traffic, but only raise it after confirming persistence is healthy. If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself and needs capacity, not a higher limit. + +::: + +## RESOURCE_EXHAUSTED on respond operations {/* #resource_exhausted-on-respond-operations */} + +**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` + +The Temporal Service is throttling Workers reporting task results. The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands. + +### Why it matters {/* #why-respond-throttling-matters */} + +This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations). If throttling persists long enough, the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task is rescheduled — with the cache eviction, cold replay, and Local Activity re-execution consequences described in that section. + +Meanwhile every in-flight task holds its slot, reducing the concurrency available for new work. Sustained throttling here escalates into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). + +### Triage {/* #triage-respond-throttling */} + +1. **Identify the throttle cause,** as in the section above. +1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. +1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) — slots are not released until the respond call succeeds, so throttling here drains the slot pool. + +:::note Self-hosted Temporal Service + +Check persistence latency filtered to `UpdateWorkflowExecution` — slow persistence on this operation is the most common root cause of throttling on respond operations. + +::: + +## UNIMPLEMENTED or INTERNAL from the Temporal Service {/* #unimplemented-or-internal-from-the-temporal-service */} + +**Metric:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation + +These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK. + +**UNIMPLEMENTED** means the Service does not recognize an operation the Worker called. By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down. + +**INTERNAL** means the Service encountered an error it could not attribute to the request. Short bursts during Service restarts and rolling deploys are expected — set the `for` duration long enough that your own deploy process does not page you. The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency. + +### Triage {/* #triage-unimplemented-internal */} + +1. **Check SDK and Temporal Service version compatibility.** For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version. +1. **Check whether recent deploys correlate.** Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate. +1. **Check whether the errors are Namespace-scoped or cluster-wide.** Errors isolated to one Namespace point at Namespace configuration. Cluster-wide errors point at infrastructure. +1. **Check downstream effects.** Sustained errors on poll operations cause Workers to back off. Cross-check [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) and [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero). + +:::note Self-hosted Temporal Service + +Check service panics first — any panic is critical and is almost always the root cause of sustained INTERNAL errors. Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. For UNIMPLEMENTED, verify every Frontend, History, and Matching pod is running the intended binary — a wrong or corrupted binary on a subset of pods produces UNIMPLEMENTED on valid operations, usually alongside panics. + +::: + +## Request latency high on user-facing operations {/* #request-latency-high-on-user-facing-operations */} + +**Metric:** `temporal_request_latency` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation` + +p99 latency on the operations your application calls synchronously has risen above your threshold. + +### Why it matters {/* #why-request-latency-matters */} + +These calls block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing. + +The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes. If throttling is the cause and retries exhaust the 60-second budget, the call fails outright. + +### Triage {/* #triage-request-latency */} + +1. **Check for throttling on the same operations.** See [RESOURCE_EXHAUSTED on user-facing operations](#resource_exhausted-on-user-facing-operations). If both are firing, throttling is the cause of the latency and retries are what you are measuring. +1. **Check payload sizes.** This metric includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive [Payload Codec](/payload-codec), raise it without any Service-side slowdown. +1. **Check network path and region.** Clients in a different region from the Temporal Service pay that round trip on every call. + +:::note Self-hosted Temporal Service + +Check Frontend Service latency filtered to the affected operations — server-side latency is the more precise signal, since the SDK metric includes serialization and network time. Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, the usual driver of elevated Frontend latency on starts and Signals. If persistence is healthy and there is no throttling, check Frontend pod CPU. + +::: diff --git a/docs/troubleshooting/sdk-worker-capacity.mdx b/docs/troubleshooting/sdk-worker-capacity.mdx new file mode 100644 index 0000000000..310350c02d --- /dev/null +++ b/docs/troubleshooting/sdk-worker-capacity.mdx @@ -0,0 +1,189 @@ +--- +id: sdk-worker-capacity +title: How to troubleshoot SDK Worker capacity problems +sidebar_label: SDK Worker capacity +description: Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems +toc_max_heading_level: 4 +keywords: + - worker capacity + - task slots + - pollers + - schedule to start latency + - sticky cache + - task backlog +tags: + - Metrics + - Observability + - Troubleshooting + - Workers +--- + +This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a Sticky Execution cache that is not doing its job. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For Worker sizing and configuration guidance, see [Worker deployment and performance](/best-practices/worker) and [Worker performance](/develop/worker-performance). + +These failure modes are causally linked. Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions. When several fire at once, work backward to the earliest link in the chain rather than treating each as independent. + +## Worker Task slots exhausted {/* #worker-task-slots-exhausted */} + +**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0 + +Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up. The SDK blocks before issuing its next poll until a slot is released. + +### Why it matters {/* #why-slots-matter */} + +Slots stay occupied because existing Tasks are not completing. Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, `temporal_num_pollers` drops to zero as a secondary effect because those SDKs block on slot acquisition before incrementing the poller gauge. + +Impact and remediation differ by `worker_type`, so check that label on the firing alert. + +### Triage: WorkflowWorker {/* #triage-slots-workflowworker */} + +1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). +1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer. +1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — slots are not released until the respond call succeeds. + +To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options. + +### Triage: ActivityWorker {/* #triage-slots-activityworker */} + +1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected. +1. **Check Worker CPU.** High CPU slows Activity execution directly. +1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). + +To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size — but only after confirming a downstream dependency is not the constraint. + +### Triage: LocalActivityWorker {/* #triage-slots-localactivityworker */} + +[Local Activities](/local-activity) run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task. The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive. If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch. + +1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling. +1. **Check Worker CPU.** +1. **Check the Local Activity latency alerts.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun. + +## All pollers disconnected {/* #all-pollers-disconnected */} + +**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0 + +No active pollers remain for this `worker_type` and Task Queue. Workers have stopped polling entirely and Tasks are accumulating with nothing to process them. + +### Why it matters {/* #why-pollers-matter */} + +Workflow and Activity Tasks accumulate on the Temporal Service. At scale this grows into a large backlog that puts pressure on Task matching and persistence. Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed. + +### Triage {/* #triage-pollers */} + +1. **Check whether Worker processes are running.** Check pod status, restart counts, and logs. Workers may have crashed, been evicted, or been OOM killed — this is the most common cause. +1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same `worker_type` and Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. +1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry. +1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) — sustained INTERNAL errors cause Workers to back off and stop polling. + +:::note Self-hosted Temporal Service + +Cross-check the server-side concurrent poller count for this Task Queue. If it has also dropped, Workers have fully disconnected from the Service's perspective. For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential. + +::: + +## Task completions dropped to zero {/* #task-completions-dropped-to-zero */} + +**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`, per Task Queue + +Workers have stopped completing Tasks. The SDK only increments this counter after a successful response, so a zero rate means either no Tasks are being finished or no responses are landing. + +### Why it matters {/* #why-completions-matter */} + +No Workflow Task completions means no Workflow Execution on this Task Queue is making progress. Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them. + +No Activity Task completions means every Workflow waiting on an Activity result is stalled. Activities begin timing out against their `scheduleToClose` timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too. + +### Triage {/* #triage-completions */} + +1. **Check whether polling also stopped.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the same `worker_type`. If pollers are also at zero, Workers are down entirely — go to [All pollers disconnected](#all-pollers-disconnected). +1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check `temporal_workflow_task_execution_failed`. Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). +1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — sustained throttling suppresses this counter directly. +1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers. +1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers. + +:::note Self-hosted Temporal Service + +Check overall cluster health: service errors by Namespace, persistence availability, and resource-exhausted counts by cause. + +::: + +## Workflow Task schedule-to-start latency elevated {/* #workflow-task-schedule-to-start-latency-elevated */} + +**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency) + +Workflow Tasks are sitting in the queue before a Worker picks them up. Alert at two levels: a threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. + +### Why it matters {/* #why-wft-s2s-matters */} + +Schedule-to-start latency adds directly to Workflow end-to-end latency. Every second a Task waits is a second added to how long your Workflows take. + +Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all. At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database. A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue. + +### Triage {/* #triage-wft-s2s */} + +1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else. +1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `worker_type=WorkflowWorker`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) — with every slot occupied, the SDK blocks before its next poll and the effective poll rate falls. +1. **Check for throttling on poll operations.** If the Temporal Service is throttling `PollWorkflowTaskQueue`, Workers back off and poll less frequently. +1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, the Worker pool is undersized for current throughput. Scale horizontally. Expect a delay before latency recovers — the accumulated backlog has to drain first. + +:::note Self-hosted Temporal Service + +Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up. Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk. If poll operations are being throttled at scale, raising `frontend.namespaceCount` or `frontend.globalNamespaceCount` may be needed alongside scaling Workers. + +::: + +## Activity schedule-to-start latency elevated {/* #activity-schedule-to-start-latency-elevated */} + +**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency) + +Activity Tasks are sitting in the queue before a Worker picks them up. + +### Why it matters {/* #why-activity-s2s-matters */} + +Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database. + +### Triage {/* #triage-activity-s2s */} + +1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting. +1. **Check poller counts** for `worker_type=ActivityWorker`. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted). +1. **Check for throttling on poll operations.** If `PollActivityTaskQueue` is being throttled, Workers pick up Tasks less frequently. +1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) — high failure churn generates retry Tasks that grow the backlog and keep this latency elevated even after Worker capacity is restored. + +:::note Self-hosted Temporal Service + +Check the approximate Task backlog and the server-side concurrent poller count for this Task Queue, as in the Workflow Task section above. + +::: + +## Sticky cache disabled {/* #sticky-cache-disabled */} + +**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 + +The Worker's [Sticky Execution](/sticky-execution) cache is holding nothing. No Workflow Executions are cached in memory between Workflow Tasks, so every Workflow Task requires a full cold replay from the beginning of Event History. + +### Why it matters {/* #why-sticky-cache-matters */} + +Every Workflow Task for every Execution on this Worker fetches all History pages from the Temporal Service and re-executes every command from scratch. Every Workflow Task becomes the equivalent of a cache miss. + +At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). + +### Triage {/* #triage-sticky-cache */} + +1. **Check your Worker cache configuration.** This is almost always a misconfiguration: + - **Go SDK** — `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` disables the cache entirely. Restore a non-zero value. This must be called before any Worker starts; the cache is shared across all Workers in the process. + - **Java SDK** — `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. If this fires on a Java Worker, check `setMaxWorkflowThreadCount(int)` instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively. +1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits. +1. **Check the replay cost while the cache is down.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) and the `GetWorkflowExecutionHistory` long-poll rate to gauge how much load the repeated History reads are generating. + +:::note Self-hosted Temporal Service + +Check persistence latency filtered to `GetWorkflowExecution` — sustained high latency there confirms the Temporal Service is under pressure from repeated History reads. + +::: diff --git a/sidebars.js b/sidebars.js index 745e8c1517..09f81c7ddd 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1703,6 +1703,9 @@ module.exports = { 'troubleshooting/last-connection-error', 'troubleshooting/performance-bottlenecks', 'troubleshooting/schedule-missed-actions', + 'troubleshooting/sdk-request-failures', + 'troubleshooting/sdk-worker-capacity', + 'troubleshooting/sdk-execution-failures', { type: 'category', label: 'Serverless Workers', @@ -1728,6 +1731,7 @@ module.exports = { }, items: [ 'best-practices/worker', + 'best-practices/sdk-worker-alerting', 'best-practices/pre-production-testing', 'production-deployment/multi-tenant-patterns', 'best-practices/managing-namespace', From fd5ba22becb456a6dd27526cbb460257660763de Mon Sep 17 00:00:00 2001 From: Auston Li Date: Tue, 4 Aug 2026 14:39:24 -0700 Subject: [PATCH 2/5] Correct metric semantics and make the alert set implementable Review pass on the SDK Worker alerting pages. Metric corrections, verified against SDK and server source: - num_pollers is tagged poller_type (workflow_task, workflow_sticky_task, activity_task, nexus_task), not worker_type - request_failure carries no task_queue tag, so the Task-completions alert cannot be scoped per Task Queue - Removed local_activity_total_execution_latency, which does not exist in the Go, Java, Python, or TypeScript SDKs; the retry-chain case now uses local_activity_execution_failed - Gauges take no _total suffix, unlike counters and histograms Reframed two alerts that would fire on healthy Workers: - sticky_cache_size reports current occupancy, not configured capacity, so zero is normal on an idle Worker; the alert now requires a non-zero Workflow Task rate alongside it - A zero Task-completion rate is equally consistent with an idle Task Queue, so the section now requires evidence of demand Made the alert set implementable: added the tag filters that distinguish each alert, the for durations, grouping guidance, a five-alert starting subset, and a table of conditions that fire during normal operation. Severity is now stated as a default routing suggestion, with the cases where it does not fit. Also: noted that worker_task_slots_available works only with fixed-size slot suppliers, added a RESOURCE_EXHAUSTED on poll operations section so triage steps no longer reference uncovered conditions, explained Workflow Task heartbeating where Local Activity alerts depend on it, and replaced repeated "Why it matters"/"Triage" subheadings with run-in labels to keep the page table of contents usable. Reference and cross-page consistency: - sdk-metrics: documented the status_code tag on request_failure and long_request_failure, and GrpcMessageTooLarge as a failure_reason - blob-size-limit-error: corrected the Workflow Task case, which the Temporal Service terminates rather than retrying indefinitely (service/history/api/respondworkflowtaskfailed/api.go) Co-Authored-By: Claude Opus 5 --- docs/best-practices/sdk-worker-alerting.mdx | 146 +++++++++++---- docs/references/sdk-metrics.mdx | 13 +- .../troubleshooting/blob-size-limit-error.mdx | 12 +- .../sdk-execution-failures.mdx | 129 ++++++++----- docs/troubleshooting/sdk-request-failures.mdx | 159 +++++++++++----- docs/troubleshooting/sdk-worker-capacity.mdx | 173 ++++++++++++------ 6 files changed, 432 insertions(+), 200 deletions(-) diff --git a/docs/best-practices/sdk-worker-alerting.mdx b/docs/best-practices/sdk-worker-alerting.mdx index 8f7d8d388a..127aaf0545 100644 --- a/docs/best-practices/sdk-worker-alerting.mdx +++ b/docs/best-practices/sdk-worker-alerting.mdx @@ -1,8 +1,8 @@ --- id: sdk-worker-alerting -title: How to alert on Temporal SDK Worker metrics +title: Alerting on Temporal SDK Worker metrics sidebar_label: SDK Worker alerting -description: A recommended alert set for Temporal SDK Workers, with thresholds and links to triage guidance +description: A recommended alert set for Temporal SDK Workers, with label filters, thresholds, and links to triage guidance toc_max_heading_level: 4 keywords: - sdk metrics @@ -16,63 +16,110 @@ tags: - Workers --- -Temporal SDKs emit metrics from your Worker processes that expose failure modes the Temporal Service cannot see on your behalf: Workflow code that fails on replay, Workers that stop polling, Task slots that never free up, and Local Activities that outlive their heartbeat window. +Temporal SDKs emit metrics from your Worker processes that expose failure modes the Temporal Service cannot see on your behalf: Workflow code that fails on replay, Workers that stop polling, Task slots that never free up, and Local Activities that outlive the Workflow Task heartbeat window. -This page recommends a set of alerts covering those failure modes, with a starting threshold for each. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. +This page recommends a set of alerts covering those failure modes. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For metric definitions and label sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). For Worker configuration guidance, see [Worker deployment and performance](/best-practices/worker). +For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). +For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). + +## Start with these five {/* #start-with-these-five */} + +If you are adding Worker alerts for the first time, start here. +These five catch the failure modes that stop Workflow Executions outright, and they produce the fewest false positives. + +1. [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) — Workers have stopped polling. +1. [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) — Workflow code no longer matches recorded history. +1. [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) — Executions are being terminated with work lost. +1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) — Tasks are backing up. +1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) — starts, Signals, and Updates are being throttled. + +Add the rest once these are tuned and quiet. ## Choose your thresholds {/* #choose-your-thresholds */} Every threshold on this page is a starting point, not a service level objective. -A high-throughput Task Queue, a latency-sensitive Namespace, and a bursty batch workload each want different values. Two things to tune per alert: +A high-throughput Task Queue, a latency-sensitive Namespace, and a bursty batch workload each want different values. +Two things to tune per alert: - **The threshold** determines what counts as unhealthy. Set it against observed p99 during a known-good period, not against the value listed here. - **The `for` duration** determines how long the condition must hold before the alert fires. Shorter durations detect problems faster and fire more often on transient spikes. Longer durations suppress noise and delay detection. -Alerts on binary conditions — any occurrence of a gRPC status code, a gauge reaching zero — need no threshold tuning. Only the `for` duration matters, and only to filter out deploys and rolling restarts. +For alerts on binary conditions — any occurrence of a gRPC status code, a gauge reaching zero — the `for` duration is doing all the work. +Those conditions occur briefly during normal operation, so a `for` duration that is too short will page you on routine events. + +### Alerts that fire during normal operation {/* #expected-firing */} + +Several of these conditions are expected in a healthy deployment. +Set the `for` duration long enough to ride them out, and confirm your value against your own deploy cadence: + +| Event | Alerts it can trigger | +| --- | --- | +| Worker deploy or rolling restart | NOT_FOUND on respond operations, all pollers disconnected, Task completions dropped to zero, non-determinism error | +| Temporal Service upgrade | INTERNAL from the Temporal Service | +| Normal Workflow completion or termination while a Task is in flight | NOT_FOUND on respond operations | +| Idle or low-volume Task Queue | Task completions dropped to zero, sticky cache holding zero entries | +| Scale-down or Worker autoscaling | All pollers disconnected, Task slots exhausted | ## Recommended alert set {/* #recommended-alert-set */} +The condition column gives the tag filters that distinguish each alert. +Group each alert by `namespace` and by the tags in its condition, so a firing alert tells you which Namespace, operation, or Task Queue is affected. + ### Request failures {/* #request-failures */} -These fire on gRPC responses from the Temporal Service to your Worker or Client. Triage guidance: [SDK request failures](/troubleshooting/sdk-request-failures). +These fire on gRPC responses from the Temporal Service to your Worker or Client. +Triage guidance: [SDK request failures](/troubleshooting/sdk-request-failures). -| Failure mode | Metric | Threshold | Severity | -| --- | --- | --- | --- | -| [NOT_FOUND on respond operations](/troubleshooting/sdk-request-failures#not_found-on-respond-operations) | `temporal_request_failure` | Any occurrence | Critical | -| [NOT_FOUND on Activity heartbeat](/troubleshooting/sdk-request-failures#not_found-on-activity-heartbeat) | `temporal_request_failure` | Any occurrence | Warning | -| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) | `temporal_request_failure` | Any occurrence | Critical | -| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) | `temporal_request_failure` | Any occurrence | Critical | -| [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `temporal_request_failure` | Any occurrence | Critical | -| [Request latency high on user-facing operations](/troubleshooting/sdk-request-failures#request-latency-high-on-user-facing-operations) | `temporal_request_latency` | p99 above 2s | Critical | +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [NOT_FOUND on respond operations](/troubleshooting/sdk-request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical | +| [NOT_FOUND on Activity heartbeat](/troubleshooting/sdk-request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning | +| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical | +| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical | +| [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning | +| [UNIMPLEMENTED from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical | +| [INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical | +| [Request latency high on user-facing operations](/troubleshooting/sdk-request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical | ### Worker capacity {/* #worker-capacity */} -These fire when Workers stop keeping up with the Task Queue. Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). +These fire when Workers stop keeping up with the Task Queue. +Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | +| [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | +| [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | +| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Warning | +| [Workflow Task schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Activity schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | -| Failure mode | Metric | Threshold | Severity | -| --- | --- | --- | --- | -| [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `temporal_worker_task_slots_available` | Reaches 0 | Critical | -| [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `temporal_num_pollers` | Reaches 0 | Critical | -| [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `temporal_request` | Rate reaches 0 per Task Queue | Critical | -| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `temporal_workflow_task_schedule_to_start_latency` | p99 above 5s, and again above 30m | Critical | -| [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `temporal_activity_schedule_to_start_latency` | p99 above 30m | Critical | -| [Sticky cache disabled](/troubleshooting/sdk-worker-capacity#sticky-cache-disabled) | `temporal_sticky_cache_size` | Reaches 0 | Warning | +:::note + +`worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. +It cannot be used with resource-based slot suppliers — see [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, alert on schedule-to-start latency instead and treat it as the signal that Worker capacity is short. + +::: ### Execution failures {/* #execution-failures */} -These fire on Workflow and Activity code failing on the Worker. Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failures). +These fire on Workflow and Activity code failing on the Worker. +Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failures). -| Failure mode | Metric | Threshold | Severity | -| --- | --- | --- | --- | -| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `temporal_workflow_task_execution_failed` | Any occurrence | Critical | -| [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `temporal_workflow_task_execution_failed` | Any occurrence | Critical | -| [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `temporal_workflow_task_execution_failed` | Rate above 10/s | Warning | -| [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `temporal_workflow_task_execution_latency` | p99 above 10s | Critical | -| [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `temporal_activity_execution_failed` | Rate above 10/s | Warning | -| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `temporal_local_activity_execution_latency`, `temporal_local_activity_total_execution_latency` | p99 above 30m | Critical | +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 5m | Critical | +| [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | +| [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | +| [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | +| [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | +| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | ## Set up the alerts {/* #set-up-the-alerts */} @@ -81,12 +128,35 @@ Before you can alert on these metrics, your Workers must be emitting them and yo - **Temporal Cloud** — [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). - **Self-hosted** — [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). -Metric names vary by SDK and metrics reporter. Counters carry a `_total` suffix and histograms a `_seconds_bucket` suffix when scraped through Prometheus. The `status_code` label is `PascalCase` in the Go SDK (`NotFound`) and `UPPER_SNAKE_CASE` in the Java and Core SDKs (`NOT_FOUND`). Confirm the exact names in your own metrics endpoint before writing queries. +### Resolve the metric names for your setup {/* #resolve-metric-names */} + +The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). +The names you query depend on your SDK and metrics reporter: + +- All metrics carry a `temporal_` prefix. +- **Counters** take a `_total` suffix when scraped through Prometheus: `temporal_request_failure_total`. +- **Histograms** take a `_seconds_bucket` suffix on the underlying bucket series: `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. +- **Gauges take no suffix.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Querying them with `_total` returns no data. + +Tag values also vary. +The `status_code` tag is `PascalCase` in the Go SDK (`NotFound`) and `UPPER_SNAKE_CASE` in the Java and Core SDKs (`NOT_FOUND`). +Some tags are not emitted by every SDK — `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs and `task_queue` only in TypeScript. + +Confirm the exact names and tags in your own metrics endpoint before writing queries. ## Route alerts by severity {/* #route-alerts-by-severity */} -Alerts marked Critical indicate that Workflow Executions are not progressing, or are progressing with data loss or duplicate side effects. Route these to a paging channel. +The severity in each table is a default routing suggestion, not a property of the metric. +Tune it to your workload. + +As a starting point, page on alerts whose condition means Workflow Executions have stopped progressing or are progressing with data loss or duplicate side effects. +Send the rest to a review channel. + +Two reasons the defaults will not fit every deployment: -Alerts marked Warning indicate degradation that has not yet stopped progress: elevated failure churn, a misconfigured cache, an Activity heartbeat that is timing out and retrying. Route these to a review channel and address them before they escalate. +- Some Critical rows are latency thresholds or leading indicators rather than confirmed stoppage. Elevated Workflow Task execution latency matters a great deal on a latency-sensitive Namespace and very little on a batch workload where Tasks routinely run long. +- Some Warning rows can still cause duplicate side effects. A NOT_FOUND on Activity heartbeat means the Activity attempt was already timed out and will re-execute from scratch, which is a real problem if that Activity is not idempotent. -Several of these alerts have causal relationships — Task slots exhausted drives pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. When multiple fire together, the triage pages identify which is the root cause and which are symptoms. +Several of these alerts have causal relationships. +Task slots exhausted drives pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. +When several fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/references/sdk-metrics.mdx b/docs/references/sdk-metrics.mdx index ccf4939ec9..68f835371a 100644 --- a/docs/references/sdk-metrics.mdx +++ b/docs/references/sdk-metrics.mdx @@ -253,7 +253,11 @@ This number is included into the total `long_request` counter for long poll RPC - Type: Counter - Available in: Core, Go, Java -- Tags: `namespace`, `operation` +- Tags: `namespace`, `operation`, `status_code` + +The `status_code` tag carries the gRPC status code returned by the Temporal Service. +Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`. +The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `long_request_latency` @@ -352,7 +356,11 @@ This number is included into the total `request` counter for RPC requests. - Type: Counter - Available in: Core, Go, Java -- Tags: `namespace`, `operation` +- Tags: `namespace`, `operation`, `status_code` + +The `status_code` tag carries the gRPC status code returned by the Temporal Service. +Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`. +The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `request_latency` @@ -506,6 +514,7 @@ A Workflow Task Execution failed. Valid values for the `failure_reason` tag: - `NonDeterminismError`: The Workflow Task failed due to a non-determinism error. +- `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and could not be delivered. The Temporal Service terminates the Workflow Execution in response. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). - `WorkflowError`: The Workflow Task failed for any other reason. ### `workflow_task_execution_latency` diff --git a/docs/troubleshooting/blob-size-limit-error.mdx b/docs/troubleshooting/blob-size-limit-error.mdx index a9c67f527b..62c375f2b9 100644 --- a/docs/troubleshooting/blob-size-limit-error.mdx +++ b/docs/troubleshooting/blob-size-limit-error.mdx @@ -103,11 +103,13 @@ The error message depends on which operation carried the oversized gRPC message The behavior depends on where the oversized message originates: -- **Workflow Tasks:** The Workflow gets stuck in a retry loop that isn't visible in the Event History. This happens - because when the Worker completes a Workflow Task, it sends all the commands the Workflow produced (such as Activity - schedules and their inputs) back to the Temporal Service. If the combined size exceeds 4 MB, the SDK catches the gRPC - error and sends a failed Workflow Task response with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Replay - produces the same oversized request every time, so the Workflow never makes progress. +- **Workflow Tasks:** The Temporal Service terminates the Workflow Execution. When the Worker completes a Workflow Task, + it sends all the commands the Workflow produced (such as Activity schedules and their inputs) back to the Temporal + Service. If the combined size exceeds 4 MB, the SDK catches the gRPC error and sends a failed Workflow Task response + with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Because replay would produce the same oversized request + on every attempt, the Temporal Service ends the Execution with `TERMINATED` status rather than retrying it. Terminated + Executions do not retry automatically, so fix the oversized payload and deploy the corrected Worker before restarting + them. See [SDK execution failures](/troubleshooting/sdk-execution-failures#grpc-message-too-large) for triage steps. - **Activity Tasks:** The Activity gets stuck in a retry loop or exits with a `ScheduleToCloseTimeout`. The Activity executes successfully, but the Worker can't deliver the oversized result over gRPC. The server never receives the diff --git a/docs/troubleshooting/sdk-execution-failures.mdx b/docs/troubleshooting/sdk-execution-failures.mdx index 8b0193ae16..86fdb2ddb7 100644 --- a/docs/troubleshooting/sdk-execution-failures.mdx +++ b/docs/troubleshooting/sdk-execution-failures.mdx @@ -1,9 +1,9 @@ --- id: sdk-execution-failures -title: How to troubleshoot SDK execution failures +title: Troubleshoot SDK execution failures sidebar_label: SDK execution failures description: Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency -toc_max_heading_level: 4 +toc_max_heading_level: 3 keywords: - non determinism error - workflow task failed @@ -16,48 +16,58 @@ tags: - Troubleshooting --- -This guide covers failures that occur while your Workflow and Activity code is executing on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that run past the Workflow Task heartbeat timeout. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. +This guide covers failures that occur while your Workflow and Activity code is executing on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that run past the Workflow Task heartbeat timeout. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). +For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). -`temporal_workflow_task_execution_failed` carries a `failure_reason` label. The reason determines what the Temporal Service does next, and the difference matters: some reasons cause indefinite retries, one causes immediate termination. +[`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`. +The reason determines what the Temporal Service does next, and the difference matters: two of the three cause indefinite retries, one causes immediate termination. +Alert on each `failure_reason` separately. ## Non-determinism error {/* #non-determinism-error */} **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError` -Replay produced a different command sequence than the one recorded in Event History. The Worker detected that the Workflow code it is running does not match the commands the Workflow Execution has already produced. +Replay produced a different command sequence than the one recorded in Event History. +The Worker detected that the Workflow code it is running does not match the commands the Workflow Execution has already produced. -### Why it matters {/* #why-nde-matters */} +**Why it matters.** +Affected Executions are not progressing. +The Temporal Service retries the Workflow Task continuously, adding pressure to your Workflow Workers, and by default the Executions stay in Running status — prolonging their end-to-end time indefinitely. +A non-determinism error does not resolve on its own. -Affected Executions are not progressing. The Temporal Service retries the Workflow Task continuously, adding pressure to your Workflow Workers, and by default the Executions stay in Running status — prolonging their end-to-end time indefinitely. A non-determinism error does not resolve on its own. - -### Triage {/* #triage-nde */} +**Triage.** 1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures. 1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause. 1. **Determine whether this is a code change or a deploy artifact.** Common causes: - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code. - - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. + - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes, which is why this alert needs a `for` duration longer than your deploy takes. - Changed Activity or Timer parameters in existing Workflow code without versioning. -1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Workflow versioning](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). +1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). 1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue. ## gRPC message too large {/* #grpc-message-too-large */} **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge` -The Workflow Task response payload exceeded the gRPC message size limit. The Worker attempted `RespondWorkflowTaskCompleted` and the response was rejected — by the gRPC library on the SDK side, by a proxy or load balancer in the path, or by the gRPC library on the Temporal Service side on receive. +The Workflow Task response payload exceeded the gRPC message size limit. +The Worker attempted `RespondWorkflowTaskCompleted` and the response was rejected — by the gRPC library on the SDK side, by a proxy or load balancer in the path, or by the gRPC library on the Temporal Service side on receive. -Because the Temporal Service never saw the original request, the SDK sends a follow-up `RespondWorkflowTaskFailed` with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. The Service then terminates the Workflow Execution. +Because the Temporal Service never saw the original request, the SDK sends a follow-up `RespondWorkflowTaskFailed` with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. +Since replay would produce the same oversized response on every attempt, the Temporal Service terminates the Workflow Execution rather than retrying it. -### Why it matters {/* #why-message-too-large-matters */} +For the payload size limits themselves and the equivalent Activity Task behavior — which retries rather than terminating — see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). -Affected Executions are terminated immediately and permanently, with `TERMINATED` status and no retry. Any in-progress work in those Executions is lost, and they must be restarted manually. +**Why it matters.** +Affected Executions are terminated immediately and permanently, with `TERMINATED` status and no retry. +Any in-progress work in those Executions is lost, and they must be restarted manually. -This is the one `failure_reason` on this page that ends Executions rather than retrying them. +This is the one `failure_reason` on this page that ends Executions rather than retrying them, which is why it warrants a short `for` duration. -### Triage {/* #triage-message-too-large */} +**Triage.** 1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History. 1. **Find what is oversized.** The fix depends entirely on which part of the response is too large: @@ -76,61 +86,69 @@ Check the Workflow terminate rate on your server dashboard — a spike alongside **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError` -Sustained Workflow Task failures from unhandled exceptions and panics in Workflow code that the SDK catches and reports. This covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. - -### Why it matters {/* #why-workflow-error-matters */} +Sustained Workflow Task failures from unhandled exceptions and panics in Workflow code that the SDK catches and reports. +`WorkflowError` is the catch-all reason: it covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. -The Temporal Service retries the Workflow Task. If the error is deterministic and reproduces on every replay, the Execution is stuck retrying indefinitely, consuming Worker capacity and staying in a permanently unhealthy state. +**Why it matters.** +The Temporal Service retries the Workflow Task. +If the error is deterministic and reproduces on every replay, the Execution is stuck retrying indefinitely, consuming Worker capacity and staying in a permanently unhealthy state. -Unlike a non-determinism error, there is no ceiling on how bad this gets: at high rates the retry pressure saturates Workflow Worker slots and affects healthy Executions on the same Task Queue. Unlike `GrpcMessageTooLarge`, the Service does not terminate the Execution, so the impact compounds until you resolve it. +At high rates the retry pressure saturates Workflow Worker slots and affects healthy Executions on the same Task Queue. +Unlike `GrpcMessageTooLarge`, the Temporal Service does not terminate the Execution, so the impact compounds until you resolve it. -### Triage {/* #triage-workflow-error */} +**Triage.** -1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. The `workflow_type` tag on the metric narrows which Workflow is failing. 1. **Determine which failure mode this is.** `WorkflowError` covers several: - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool, caused by `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` being too low for the number of concurrent Executions. New Workflow Tasks are rejected before they can execute. Raise the thread count, and check whether the Worker pool needs to scale out as well. - **Unhandled exception in Workflow code.** A bug or unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event identifies the error. - - **Data Converter error.** A failure serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and Payload Codec configuration. + - **Data Converter error.** A failure serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration. 1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion often occur together under load, and a CPU-starved Worker completes Workflow Tasks more slowly, accelerating both. 1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running. ## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */} -**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) +**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type` -Workflow Tasks are taking too long to execute on the Worker. The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks. +Workflow Tasks are taking too long to execute on the Worker. +The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks. -### Why it matters {/* #why-wft-latency-matters */} +A batch workload where Workflow Tasks routinely run long may sit above this threshold without any problem. +Set the threshold against your own observed p99, and treat the default as a signal only if your Workflows are latency-sensitive. -The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue. Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay. +**Why it matters.** +The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue. +Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay. -If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats. Non-idempotent Local Activities produce duplicate side effects with real business impact. +If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats. +Non-idempotent Local Activities produce duplicate side effects with real business impact. At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts. -### Triage {/* #triage-wft-latency */} +**Triage.** 1. **Check replay latency.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-executing Event History rather than running new commands — usually caused by large histories, slow Data Converter execution during replay, or a high cache eviction rate forcing cold replays. -1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache disabled](/troubleshooting/sdk-worker-capacity#sticky-cache-disabled) for the case where the cache is off entirely. +1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load). 1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker. 1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. 1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — the SDK holds the slot until the respond call succeeds, inflating this metric even when the Workflow code finished quickly. ## Activity execution failures elevated {/* #activity-execution-failures-elevated */} -**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed) +**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type` Activities are explicitly failing — returning failures rather than timing out — at a sustained rate. `ApplicationFailure` instances marked with category `BENIGN` are excluded and do not increment this counter, so this metric tracks unexpected failures only to the extent your application uses benign failures correctly. -### Why it matters {/* #why-activity-failures-matter */} - -A high failure rate drives a burst of retry Tasks. If Workers cannot keep up with the retry volume, the Activity Task backlog grows — see [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). At scale, sustained retry bursts put significant pressure on Task matching and the underlying database. +**Why it matters.** +A high failure rate drives a burst of retry Tasks. +If Workers cannot keep up with the retry volume, the Activity Task backlog grows — see [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). +At scale, sustained retry bursts put significant pressure on Task matching and the underlying database. -### Triage {/* #triage-activity-failures */} +**Triage.** -1. **Identify which Activity is failing.** The metric carries an `activity_type` label. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. +1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. 1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not. 1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying. 1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. @@ -138,26 +156,39 @@ A high failure rate drives a burst of retry Tasks. If Workers cannot keep up wit ## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */} -**Metrics:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency) for a single attempt, and `temporal_local_activity_total_execution_latency` for the full retry chain +**Metric:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type` + +A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes. + +### How Workflow Task heartbeating works {/* #how-workflow-task-heartbeating-works */} -A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes. The SDK sends Workflow Task heartbeats to keep the Task alive while the Local Activity runs, but once the timeout is exceeded the Temporal Service times out the heartbeating Workflow Task. +A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task. +That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout. -Watch both metrics. The single-attempt metric catches one long-running attempt. The total metric catches a retry chain that accumulates past the timeout even when every individual attempt is short — usually a high failure rate paired with an aggressive Retry Policy. +To keep the Task alive, the SDK sends Workflow Task heartbeats — repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still in progress and request more time. +The Service allows this up to the Workflow Task heartbeat timeout. +Past that, it times out the Task and reschedules it on the normal Task Queue. -### Why it matters {/* #why-la-latency-matters */} +Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats. +So when the Task is rescheduled, every Local Activity in it runs again from the beginning. -When the Temporal Service times out the heartbeating Workflow Task, it reschedules the Task on the normal Task Queue and the Local Activity re-executes from scratch. Local Activities cannot heartbeat, and their progress is not checkpointed between Workflow Task heartbeats. A non-idempotent Local Activity produces duplicate side effects with real business impact. +**Why it matters.** +When the Temporal Service times out the heartbeating Workflow Task, the Local Activity re-executes from scratch. +A non-idempotent Local Activity produces duplicate side effects with real business impact. Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly. -The Local Activity also occupies an executor slot for its entire duration. Several in this state at once can occupy every available slot, blocking new Local Activities from starting. See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#triage-slots-localactivityworker) for `worker_type=LocalActivityWorker`. +The Local Activity also occupies an executor slot for its entire duration. +Several in this state at once can occupy every available slot, blocking new Local Activities from starting. +See [LocalActivityWorker slots](/troubleshooting/sdk-worker-capacity#localactivityworker-slots). -Local Activities are designed for short, fast operations. A single attempt running for 30 minutes is a design problem, not a tuning problem. +Local Activities are designed for short, fast operations. +A single attempt running for 30 minutes is a design problem, not a tuning problem. -### Triage {/* #triage-la-latency */} +**Triage.** -1. **Identify the affected Local Activity.** The metric carries an `activity_type` label. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids. +1. **Identify the affected Local Activity.** The `activity_type` tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids. 1. **Find what it is blocked on.** A Local Activity running this long is almost always blocked on a downstream call — a slow service, a slow query, or a network call with a very long timeout. Fix the dependency, or shorten the timeout on the call so the Local Activity fails fast instead of hanging. -1. **Check the failure rate driving retries.** If the total-latency metric is elevated but single attempts are short, a high failure rate with aggressive retries is accumulating the chain. Fix the underlying failure first. +1. **Check whether a retry chain is accumulating.** A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check [`temporal_local_activity_execution_failed`](/references/sdk-metrics#local_activity_execution_failed) for the same `activity_type`, and fix the underlying failure first. 1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have already timed out heartbeating Workflow Tasks. Check Worker logs for timeout errors and Event History for `WorkflowTaskTimedOut` events. If they are present, Local Activities have already re-executed — verify idempotency and address any duplicate side effects. 1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed. diff --git a/docs/troubleshooting/sdk-request-failures.mdx b/docs/troubleshooting/sdk-request-failures.mdx index da69b3c0a8..c748386f8d 100644 --- a/docs/troubleshooting/sdk-request-failures.mdx +++ b/docs/troubleshooting/sdk-request-failures.mdx @@ -1,9 +1,9 @@ --- id: sdk-request-failures -title: How to troubleshoot SDK request failures +title: Troubleshoot SDK request failures sidebar_label: SDK request failures description: Diagnose gRPC failures and elevated latency on requests from Temporal SDK Workers and Clients -toc_max_heading_level: 4 +toc_max_heading_level: 3 keywords: - sdk metrics - request failure @@ -16,15 +16,22 @@ tags: - Troubleshooting --- -This guide covers gRPC failures and elevated latency on requests that Temporal SDK Workers and Clients make to the Temporal Service. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. +This guide covers gRPC failures and elevated latency on requests that Temporal SDK Workers and Clients make to the Temporal Service. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -Each section names the metric to watch, explains the impact, and walks through triage. For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). +For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). -`temporal_request_failure` increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. `temporal_long_request_failure` covers poll operations and long-poll `GetWorkflowExecutionHistory`. Both carry `namespace`, `operation`, `status_code`, and `task_queue` labels. Severity depends on which status code appears on which operation. +[`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. +[`temporal_long_request_failure`](/references/sdk-metrics#long_request_failure) covers poll operations and long-poll `GetWorkflowExecutionHistory`. +Both carry `namespace`, `operation`, and `status_code` tags. +Which status code appears on which operation determines how urgent the failure is. :::note +Two naming details will affect your queries. `UpdateWithStartWorkflowExecution` appears in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. +And the `status_code` tag can be suppressed through Client options — confirm it is present in your metrics endpoint before filtering on it. ::: @@ -32,28 +39,35 @@ Each section names the metric to watch, explains the impact, and walks through t **Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` -A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. There are three causes: +A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. +There are three causes: - The task timed out. The Worker ran past the Workflow Task timeout, or past the Activity `startToClose` or `scheduleToClose` timeout, and the Service discarded the in-flight task. - The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished. - The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up. -### Why it matters {/* #why-not_found-matters */} +The second and third causes occur during normal operation, so isolated occurrences are not a problem. +What matters is a sustained rate. -The result the Worker just produced was discarded. For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a Sticky Execution cache eviction and a cold replay on the retry. +**Why it matters.** +The result the Worker just produced was discarded. +For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. +For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a [Sticky Execution](/sticky-execution) cache eviction and a cold replay on the retry. -Repeated NOT_FOUND on respond operations means Workers are consistently finishing too late. Every discarded result is Worker capacity spent on work that was thrown away, and every rescheduled task adds to Workflow end-to-end latency. +A sustained rate means Workers are consistently finishing too late. +Every discarded result is Worker capacity spent on work that was thrown away, and every rescheduled task adds to Workflow end-to-end latency. -If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried task. Local Activity results are not written to Event History between Workflow Task heartbeats, so re-execution means running them again. If they are not idempotent, this produces duplicate side effects with real business impact. +If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried task. +Local Activity results are not written to Event History between Workflow Task heartbeats, so re-execution means running them again. +If they are not idempotent, this produces duplicate side effects with real business impact. -### Triage {/* #triage-not_found */} +**Triage.** +1. **Rule out the expected causes first.** Check the status of a few affected Executions in the Temporal UI or with `temporal workflow describe`. If they completed, were terminated, or hit their Run Timeout, the NOT_FOUND is expected. Check Worker restart counts in your infrastructure observability stack for the same reason. If either explains the volume, stop here. 1. **Check task execution latency.** For Workflow Tasks, check [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency). For Activities, check [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause. 1. **Check replay latency.** If Workflow Task execution latency is elevated, check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. High replay latency means the Worker is spending its time re-executing Event History rather than running new commands — check for large histories and slow [Data Converter](/dataconversion) execution during replay. 1. **Check Worker resources.** High CPU on the Worker slows task execution directly. Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays. -1. **Confirm the Workflow Execution is still running.** Check the Execution status in the Temporal UI or with `temporal workflow describe`. If it completed, was terminated, or hit its Run Timeout, the NOT_FOUND is expected and the Execution lifecycle is what to investigate. -1. **Check for Worker restarts.** Frequent restarts lose in-flight Task Tokens. Check pod restart counts in your infrastructure observability stack. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations) below. Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations). Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands. :::note Self-hosted Temporal Service @@ -65,47 +79,59 @@ If SDK-side metrics look normal and the Execution was not terminated or timed ou **Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat` -A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running. +A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. +The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running. -Normal Workflow-side cancellation is not a cause. Cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure. +Normal Workflow-side cancellation is not a cause. +Cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure. -### Why it matters {/* #why-heartbeat-not_found-matters */} +**Why it matters.** +If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. +The Activity re-executes from scratch on the next attempt, so a non-idempotent Activity produces duplicate side effects — which is why this is worth treating more seriously than its default severity suggests. -If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. The Activity re-executes from scratch on the next attempt, so a non-idempotent Activity produces duplicate side effects. +A sustained rate means the Worker is consistently failing to heartbeat within the configured interval. +The Activity will keep timing out on every attempt until the cause is fixed, holding Task slots and generating retry tasks the whole time. -Repeated NOT_FOUND on heartbeat calls means the Worker is consistently failing to heartbeat within the configured interval. The Activity will keep timing out on every attempt until the cause is fixed, holding Task slots and generating retry tasks the whole time. - -### Triage {/* #triage-heartbeat-not_found */} +**Triage.** 1. **Compare the heartbeat interval against `heartbeatTimeout`.** The Worker must call heartbeat more frequently than the timeout. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat. 1. **Check Worker CPU.** A CPU-starved Worker slows down between heartbeat calls even when the Activity is making progress. If utilization is consistently high, reduce per-Worker concurrency or scale out horizontally. -1. **Check for throttling on heartbeat calls.** If the Temporal Service is throttling `RecordActivityTaskHeartbeat`, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time. 1. **Check the `startToClose` timeout.** If the Activity has run longer than `startToClose`, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND. Compare [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` against the configured timeout. +1. **Check for throttling on heartbeat calls.** Query `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` and `operation=RecordActivityTaskHeartbeat`. If the Temporal Service is throttling these calls, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time. See [RESOURCE_EXHAUSTED on poll operations](#resource_exhausted-on-poll-operations) for how to work through a throttling cause. 1. **Check heartbeat payload size.** The last heartbeat details payload is held in memory for the life of the Activity attempt. Large payloads on high-throughput Activity Workers contribute to memory pressure on the Temporal Service. Store only the minimum progress state needed to resume on retry. ## RESOURCE_EXHAUSTED on user-facing operations {/* #resource_exhausted-on-user-facing-operations */} **Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation` -The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates. The SDK retries these automatically for up to 60 seconds. Beyond that, the call fails and the error propagates to your caller. - -### Why it matters {/* #why-resource_exhausted-matters */} +The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates. +The SDK retries these automatically for up to 60 seconds. +Beyond that, the call fails and the error propagates to your caller. -These operations are on your application's critical path. Within the retry window, callers experience elevated latency. Past it, calls fail outright and your application must handle the error. +**Why it matters.** +These operations are on your application's critical path. +Within the retry window, callers experience elevated latency. +Past it, calls fail outright and your application must handle the error. -If it does not, starts and Signals are silently dropped. A dropped start means the Workflow never runs. A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. Log these failures in your application code so you can backfill starts and Signals afterward. +If it does not, starts and Signals are silently dropped. +A dropped start means the Workflow never runs. +A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. +Log these failures in your application code so you can backfill starts and Signals afterward. -The Temporal Service throttles these operations last. Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread. +The Temporal Service throttles these operations last. +Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread. -### Triage {/* #triage-resource_exhausted */} +**Triage.** -1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, or an open circuit breaker are different problems. +1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, or an open circuit breaker are different problems with different remedies. 1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare current throughput against your Namespace's [service limits](/cloud/limits) and open a support request if you need them raised. 1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure. :::note Self-hosted Temporal Service -Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution` — slow persistence is the most common root cause of cascading throttling. If the cause is a rate limit, `frontend.namespaceRPS` may be set too low for your traffic, but only raise it after confirming persistence is healthy. If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself and needs capacity, not a higher limit. +Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution` — slow persistence is the most common root cause of cascading throttling. +If the cause is a rate limit, `frontend.namespaceRPS` may be set too low for your traffic, but only raise it after confirming persistence is healthy. +If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself and needs capacity, not a higher limit. ::: @@ -113,15 +139,17 @@ Check the resource-exhausted cause on your server dashboard, then check persiste **Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` -The Temporal Service is throttling Workers reporting task results. The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands. +The Temporal Service is throttling Workers reporting task results. +The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands. -### Why it matters {/* #why-respond-throttling-matters */} +**Why it matters.** +This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations). +If throttling persists long enough, the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task is rescheduled — with the cache eviction, cold replay, and Local Activity re-execution consequences described in that section. -This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations). If throttling persists long enough, the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task is rescheduled — with the cache eviction, cold replay, and Local Activity re-execution consequences described in that section. +Meanwhile every in-flight task holds its slot, reducing the concurrency available for new work. +Sustained throttling here escalates into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). -Meanwhile every in-flight task holds its slot, reducing the concurrency available for new work. Sustained throttling here escalates into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). - -### Triage {/* #triage-respond-throttling */} +**Triage.** 1. **Identify the throttle cause,** as in the section above. 1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. @@ -133,17 +161,50 @@ Check persistence latency filtered to `UpdateWorkflowExecution` — slow persist ::: +## RESOURCE_EXHAUSTED on poll operations {/* #resource_exhausted-on-poll-operations */} + +**Metric:** `temporal_long_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `PollWorkflowTaskQueue` or `PollActivityTaskQueue` + +The Temporal Service is throttling Worker poll calls. +Poll operations are long-poll requests, so they increment `temporal_long_request_failure` rather than `temporal_request_failure`. + +**Why it matters.** +Throttled Workers back off and poll less frequently, which lowers the effective poll rate for the Task Queue even when every Worker is healthy. +That shows up as rising schedule-to-start latency and, if it persists, as a growing Task backlog. + +This is often a symptom rather than a cause. +The Temporal Service throttles poll operations before it throttles respond or user-facing operations, so throttling here can be the first visible sign of pressure that has nothing to do with your Workers. + +**Triage.** + +1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. +1. **Check whether you are over-polling.** A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against [Worker performance](/develop/worker-performance) guidance before assuming the limit is too low. +1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet. +1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare against your Namespace's [service limits](/cloud/limits). + +:::note Self-hosted Temporal Service + +If poll operations are being throttled at scale, the Namespace concurrent poller limit may need raising through `frontend.namespaceCount` or `frontend.globalNamespaceCount` — but scale Worker capacity first if schedule-to-start latency is the actual problem. + +::: + ## UNIMPLEMENTED or INTERNAL from the Temporal Service {/* #unimplemented-or-internal-from-the-temporal-service */} **Metric:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK. +Alert on them separately. -**UNIMPLEMENTED** means the Service does not recognize an operation the Worker called. By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down. +**UNIMPLEMENTED** means the Service does not recognize an operation the Worker called. +By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. +Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down. -**INTERNAL** means the Service encountered an error it could not attribute to the request. Short bursts during Service restarts and rolling deploys are expected — set the `for` duration long enough that your own deploy process does not page you. The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency. +**INTERNAL** means the Service encountered an error it could not attribute to the request. +Short bursts during Service restarts and rolling deploys are expected — set the `for` duration long enough that your own deploy process does not page you. +The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. +Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency. -### Triage {/* #triage-unimplemented-internal */} +**Triage.** 1. **Check SDK and Temporal Service version compatibility.** For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version. 1. **Check whether recent deploys correlate.** Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate. @@ -152,23 +213,25 @@ These two status codes point at the Temporal Service rather than at your applica :::note Self-hosted Temporal Service -Check service panics first — any panic is critical and is almost always the root cause of sustained INTERNAL errors. Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. For UNIMPLEMENTED, verify every Frontend, History, and Matching pod is running the intended binary — a wrong or corrupted binary on a subset of pods produces UNIMPLEMENTED on valid operations, usually alongside panics. +Check service panics first — any panic is critical and is almost always the root cause of sustained INTERNAL errors. +Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. +For UNIMPLEMENTED, verify every Frontend, History, and Matching pod is running the intended binary — a wrong or corrupted binary on a subset of pods produces UNIMPLEMENTED on valid operations, usually alongside panics. ::: ## Request latency high on user-facing operations {/* #request-latency-high-on-user-facing-operations */} -**Metric:** `temporal_request_latency` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation` +**Metric:** [`temporal_request_latency`](/references/sdk-metrics#request_latency) on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation` p99 latency on the operations your application calls synchronously has risen above your threshold. -### Why it matters {/* #why-request-latency-matters */} - +**Why it matters.** These calls block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing. -The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes. If throttling is the cause and retries exhaust the 60-second budget, the call fails outright. +The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes. +If throttling is the cause and retries exhaust the 60-second budget, the call fails outright. -### Triage {/* #triage-request-latency */} +**Triage.** 1. **Check for throttling on the same operations.** See [RESOURCE_EXHAUSTED on user-facing operations](#resource_exhausted-on-user-facing-operations). If both are firing, throttling is the cause of the latency and retries are what you are measuring. 1. **Check payload sizes.** This metric includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive [Payload Codec](/payload-codec), raise it without any Service-side slowdown. @@ -176,6 +239,8 @@ The SDK retries transient errors but does not hide the latency cost: every retry :::note Self-hosted Temporal Service -Check Frontend Service latency filtered to the affected operations — server-side latency is the more precise signal, since the SDK metric includes serialization and network time. Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, the usual driver of elevated Frontend latency on starts and Signals. If persistence is healthy and there is no throttling, check Frontend pod CPU. +Check Frontend Service latency filtered to the affected operations — server-side latency is the more precise signal, since the SDK metric includes serialization and network time. +Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, the usual driver of elevated Frontend latency on starts and Signals. +If persistence is healthy and there is no throttling, check Frontend pod CPU. ::: diff --git a/docs/troubleshooting/sdk-worker-capacity.mdx b/docs/troubleshooting/sdk-worker-capacity.mdx index 310350c02d..32a7f2695c 100644 --- a/docs/troubleshooting/sdk-worker-capacity.mdx +++ b/docs/troubleshooting/sdk-worker-capacity.mdx @@ -1,9 +1,9 @@ --- id: sdk-worker-capacity -title: How to troubleshoot SDK Worker capacity problems +title: Troubleshoot SDK Worker capacity problems sidebar_label: SDK Worker capacity description: Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems -toc_max_heading_level: 4 +toc_max_heading_level: 3 keywords: - worker capacity - task slots @@ -18,25 +18,39 @@ tags: - Workers --- -This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a Sticky Execution cache that is not doing its job. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. +This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a [Sticky Execution](/sticky-execution) cache that is not doing its job. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For recommended alert thresholds, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For Worker sizing and configuration guidance, see [Worker deployment and performance](/best-practices/worker) and [Worker performance](/develop/worker-performance). +For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). -These failure modes are causally linked. Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions. When several fire at once, work backward to the earliest link in the chain rather than treating each as independent. +These failure modes are causally linked. +Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions. +When several fire at once, work backward to the earliest link in the chain rather than treating each as independent. ## Worker Task slots exhausted {/* #worker-task-slots-exhausted */} -**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0 +**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0, tagged `worker_type` -Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up. The SDK blocks before issuing its next poll until a slot is released. +Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up. +The SDK blocks before issuing its next poll until a slot is released. -### Why it matters {/* #why-slots-matter */} +:::note -Slots stay occupied because existing Tasks are not completing. Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, `temporal_num_pollers` drops to zero as a secondary effect because those SDKs block on slot acquisition before incrementing the poller gauge. +This metric reports meaningful values only with fixed-size slot suppliers. +It cannot be used with resource-based slot suppliers — see [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this section and use [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) as your signal that Worker capacity is short. -Impact and remediation differ by `worker_type`, so check that label on the firing alert. +::: + +**Why it matters.** +Slots stay occupied because existing Tasks are not completing. +Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) drops to zero as a secondary effect, because those SDKs block on slot acquisition before incrementing the poller gauge. + +Impact and remediation differ by `worker_type`, so check that tag on the firing alert. +Values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`. -### Triage: WorkflowWorker {/* #triage-slots-workflowworker */} +### WorkflowWorker slots {/* #workflowworker-slots */} 1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). 1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer. @@ -45,7 +59,7 @@ Impact and remediation differ by `worker_type`, so check that label on the firin To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options. -### Triage: ActivityWorker {/* #triage-slots-activityworker */} +### ActivityWorker slots {/* #activityworker-slots */} 1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected. 1. **Check Worker CPU.** High CPU slows Activity execution directly. @@ -54,55 +68,79 @@ To recover immediately, scale out Workflow Worker pods or raise the maximum conc To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size — but only after confirming a downstream dependency is not the constraint. -### Triage: LocalActivityWorker {/* #triage-slots-localactivityworker */} +### LocalActivityWorker slots {/* #localactivityworker-slots */} -[Local Activities](/local-activity) run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task. The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive. If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch. +[Local Activities](/local-activity) run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task. +The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive. +If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch. 1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling. 1. **Check Worker CPU.** -1. **Check the Local Activity latency alerts.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun. +1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun. ## All pollers disconnected {/* #all-pollers-disconnected */} -**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0 +**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0, tagged `poller_type` and `task_queue` -No active pollers remain for this `worker_type` and Task Queue. Workers have stopped polling entirely and Tasks are accumulating with nothing to process them. +No active pollers remain for this poller type and Task Queue. +Workers have stopped polling entirely and Tasks are accumulating with nothing to process them. -### Why it matters {/* #why-pollers-matter */} +:::note + +This metric is tagged `poller_type`, not `worker_type`. +The values are `workflow_task`, `workflow_sticky_task`, `activity_task`, and `nexus_task`. +Alert on each poller type your Workers use, and note that a Worker with an active Sticky Execution cache reports `workflow_sticky_task` alongside `workflow_task`. + +::: -Workflow and Activity Tasks accumulate on the Temporal Service. At scale this grows into a large backlog that puts pressure on Task matching and persistence. Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed. +**Why it matters.** +Workflow and Activity Tasks accumulate on the Temporal Service. +At scale this grows into a large backlog that puts pressure on Task matching and persistence. +Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed. -### Triage {/* #triage-pollers */} +**Triage.** 1. **Check whether Worker processes are running.** Check pod status, restart counts, and logs. Workers may have crashed, been evicted, or been OOM killed — this is the most common cause. -1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same `worker_type` and Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. +1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further. +1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. 1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry. 1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) — sustained INTERNAL errors cause Workers to back off and stop polling. :::note Self-hosted Temporal Service -Cross-check the server-side concurrent poller count for this Task Queue. If it has also dropped, Workers have fully disconnected from the Service's perspective. For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential. +Cross-check the server-side concurrent poller count for this Task Queue. +If it has also dropped, Workers have fully disconnected from the Service's perspective. +For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential. ::: ## Task completions dropped to zero {/* #task-completions-dropped-to-zero */} -**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted`, per Task Queue +**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted` -Workers have stopped completing Tasks. The SDK only increments this counter after a successful response, so a zero rate means either no Tasks are being finished or no responses are landing. +The SDK increments this counter only after a successful response, so a zero rate means no responses are landing. -### Why it matters {/* #why-completions-matter */} +**A zero rate on its own does not prove a problem.** +It is equally consistent with an idle Task Queue, a Namespace outside business hours, or a batch workload between runs. +This metric is tagged `namespace` and `operation`, not `task_queue`, so you cannot narrow it to a single Task Queue without correlating against another signal. -No Workflow Task completions means no Workflow Execution on this Task Queue is making progress. Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them. +Treat it as actionable only alongside evidence of demand: a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency that keeps climbing. +Without one of those, there is nothing for Workers to complete and the metric is reporting the truth. -No Activity Task completions means every Workflow waiting on an Activity result is stalled. Activities begin timing out against their `scheduleToClose` timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too. +**Why it matters.** +When there is demand, no Workflow Task completions means Workflow Executions are not making progress. +Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them. -### Triage {/* #triage-completions */} +No Activity Task completions means Workflows waiting on Activity results are stalled. +Activities begin timing out against their `scheduleToClose` timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too. -1. **Check whether polling also stopped.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the same `worker_type`. If pollers are also at zero, Workers are down entirely — go to [All pollers disconnected](#all-pollers-disconnected). -1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check `temporal_workflow_task_execution_failed`. Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). +**Triage.** + +1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed. +1. **Check whether polling also stopped.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are also at zero, Workers are down entirely — go to [All pollers disconnected](#all-pollers-disconnected). +1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). 1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — sustained throttling suppresses this counter directly. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — sustained throttling suppresses this counter directly, because the SDK only counts successful responses. 1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers. 1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers. @@ -114,47 +152,52 @@ Check overall cluster health: service errors by Namespace, persistence availabil ## Workflow Task schedule-to-start latency elevated {/* #workflow-task-schedule-to-start-latency-elevated */} -**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency) - -Workflow Tasks are sitting in the queue before a Worker picks them up. Alert at two levels: a threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. +**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue` -### Why it matters {/* #why-wft-s2s-matters */} +Workflow Tasks are sitting in the queue before a Worker picks them up. +Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. -Schedule-to-start latency adds directly to Workflow end-to-end latency. Every second a Task waits is a second added to how long your Workflows take. +**Why it matters.** +Schedule-to-start latency adds directly to Workflow end-to-end latency. +Every second a Task waits is a second added to how long your Workflows take. -Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all. At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database. A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue. +Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all. +At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database. +A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue. -### Triage {/* #triage-wft-s2s */} +**Triage.** 1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else. -1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `worker_type=WorkflowWorker`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). 1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) — with every slot occupied, the SDK blocks before its next poll and the effective poll rate falls. -1. **Check for throttling on poll operations.** If the Temporal Service is throttling `PollWorkflowTaskQueue`, Workers back off and poll less frequently. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations) — throttled Workers back off and poll less frequently. 1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, the Worker pool is undersized for current throughput. Scale horizontally. Expect a delay before latency recovers — the accumulated backlog has to drain first. :::note Self-hosted Temporal Service -Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up. Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk. If poll operations are being throttled at scale, raising `frontend.namespaceCount` or `frontend.globalNamespaceCount` may be needed alongside scaling Workers. +Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up. +Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk. ::: ## Activity schedule-to-start latency elevated {/* #activity-schedule-to-start-latency-elevated */} -**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency) +**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue` Activity Tasks are sitting in the queue before a Worker picks them up. -### Why it matters {/* #why-activity-s2s-matters */} - -Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database. +**Why it matters.** +Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. +At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database. -### Triage {/* #triage-activity-s2s */} +**Triage.** 1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting. -1. **Check poller counts** for `worker_type=ActivityWorker`. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected). 1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted). -1. **Check for throttling on poll operations.** If `PollActivityTaskQueue` is being throttled, Workers pick up Tasks less frequently. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations). 1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) — high failure churn generates retry Tasks that grow the backlog and keep this latency elevated even after Worker capacity is restored. +1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks). :::note Self-hosted Temporal Service @@ -162,25 +205,37 @@ Check the approximate Task backlog and the server-side concurrent poller count f ::: -## Sticky cache disabled {/* #sticky-cache-disabled */} +## Sticky cache holding zero entries under load {/* #sticky-cache-holding-zero-entries-under-load */} + +**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 while the Worker is processing Workflow Tasks + +This gauge reports how many Workflow Executions the Worker is **currently** holding in its [Sticky Execution](/sticky-execution) cache, not the configured cache capacity. -**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 +**A zero value is normal on an idle Worker.** +A Worker that is not processing Workflow Tasks has nothing to cache, and a low-volume Task Queue may sit at zero for long stretches. +Alerting on this gauge alone will page you on idle Workers. -The Worker's [Sticky Execution](/sticky-execution) cache is holding nothing. No Workflow Executions are cached in memory between Workflow Tasks, so every Workflow Task requires a full cold replay from the beginning of Event History. +The condition worth alerting on is zero occupancy *while the Worker is doing work*. +Pair the gauge with a non-zero Workflow Task rate on the same Worker, and use a long `for` duration. +Corroborate with elevated [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate, either of which confirms Workflow Tasks are replaying from history rather than resuming from cache. -### Why it matters {/* #why-sticky-cache-matters */} +Tag coverage is uneven, which limits how precisely you can scope this alert: `namespace` is emitted only by the TypeScript and Java SDKs, and `task_queue` only by TypeScript. -Every Workflow Task for every Execution on this Worker fetches all History pages from the Temporal Service and re-executes every command from scratch. Every Workflow Task becomes the equivalent of a cache miss. +**Why it matters.** +When the cache holds nothing while Workflow Tasks are running, every Workflow Task fetches all History pages from the Temporal Service and re-executes every command from scratch. +Every Workflow Task becomes the equivalent of a cache miss. -At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). +At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker. +See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). -### Triage {/* #triage-sticky-cache */} +**Triage.** -1. **Check your Worker cache configuration.** This is almost always a misconfiguration: +1. **Confirm the Worker is actually processing Workflow Tasks.** If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix. +1. **Check whether the cache is disabled by configuration.** A cache size of zero in Worker options means no Execution is ever cached: - **Go SDK** — `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` disables the cache entirely. Restore a non-zero value. This must be called before any Worker starts; the cache is shared across all Workers in the process. - - **Java SDK** — `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. If this fires on a Java Worker, check `setMaxWorkflowThreadCount(int)` instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively. -1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits. -1. **Check the replay cost while the cache is down.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) and the `GetWorkflowExecutionHistory` long-poll rate to gauge how much load the repeated History reads are generating. + - **Java SDK** — `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively. +1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they are added. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction) — a high rate points at a cache too small for the number of concurrent Executions on the Worker, or at repeated Workflow Task timeouts forcing evictions. +1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero under load, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits. :::note Self-hosted Temporal Service From 310408a67ded5811f2ccc2ed015fb3695d6b0e38 Mon Sep 17 00:00:00 2001 From: Auston Li Date: Tue, 4 Aug 2026 17:40:56 -0700 Subject: [PATCH 3/5] Fix status_code casing, document the resource-exhausted metrics, restore alert 27 Audit pass. Verified every metric name, tag, tag value, SDK option, and described behavior against SDK and server source rather than against the source material, which was wrong on several of them. Blocking fixes: - status_code values are UPPER_SNAKE_CASE in every SDK. The claim that Go reports PascalCase (NotFound) was wrong and had reached the metrics reference: Go's canonicalString returns "NOT_FOUND" (sdk-go internal/common/metrics/tags.go) and Java uses code.name() (sdk-java GrpcMetricsInterceptor). A Go reader filtering status_code="NotFound" would have matched nothing. - yarn check:metrics failed on the previous commit. status_code was added to two metrics without being added to the checker's tag allowlist or to the shared tag list on the page. Both now updated, along with cause. Reference additions and corrections: - Documented request_resource_exhausted and long_request_resource_exhausted (Go), which carry a cause tag naming the reason the Temporal Service rejected the request. Both were in bin/metrics-baseline.json with an empty note, which that file defines as "not yet reviewed"; removed the entries per its instructions. The three throttling sections now point at this metric instead of asking the reader to identify the cause with no means to do so. - poller_type value is workflow_sticky_task, not sticky_workflow_task (sdk-go constants.go, sdk-java PollerTypeMetricsTag). Scope, checked against the upstream alert set: - Restored alert 27, Unregistered Activity invocation, which had been dropped entirely rather than deferred as the previous PR description claimed. Go SDK only, and labelled as such. - Restored Critical severity on Workflow Task schedule-to-start and a 1m for duration on non-determinism error, both of which had been changed without a factual basis. - Sticky cache keeps its 15m for duration, which is a deliberate deviation: the gauge is occupancy, so a shorter window fires on idle Workers. Cross-page consistency: - Both schedule-to-start sections now explain why performance-bottlenecks gives a lower threshold for the same metric (investigation, not paging) and link the ready-made queries in cloud/worker-health. - Softened the claim that every SDK supports the BENIGN failure category, which was verified in Go only, and noted that internal failures increment activity_execution_failed regardless of category. Verified: yarn build exits 0 with onBrokenLinks and onBrokenAnchors both set to throw; check:metrics, check:metrics:sdks, and check:orphans pass. Co-Authored-By: Claude Opus 5 --- bin/check-metrics-reference.js | 2 ++ bin/metrics-baseline.json | 10 ------ docs/best-practices/sdk-worker-alerting.mdx | 9 ++--- docs/references/sdk-metrics.mdx | 34 +++++++++++++++++-- .../sdk-execution-failures.mdx | 26 ++++++++++++-- docs/troubleshooting/sdk-request-failures.mdx | 11 +++--- docs/troubleshooting/sdk-worker-capacity.mdx | 7 ++++ 7 files changed, 75 insertions(+), 24 deletions(-) diff --git a/bin/check-metrics-reference.js b/bin/check-metrics-reference.js index 77208c38f3..bc121c18e9 100644 --- a/bin/check-metrics-reference.js +++ b/bin/check-metrics-reference.js @@ -27,12 +27,14 @@ const SDKS = ['Core', 'Go', 'Java']; // against this, so a hyphen/underscore slip in either place is a failure. const TAGS = [ 'activity_type', + 'cause', 'failure_reason', 'namespace', 'nexus_operation', 'nexus_service', 'operation', 'poller_type', + 'status_code', 'task_queue', 'worker_type', 'workflow_type', diff --git a/bin/metrics-baseline.json b/bin/metrics-baseline.json index 55a5ab3c8d..1ed0b2b7a8 100644 --- a/bin/metrics-baseline.json +++ b/bin/metrics-baseline.json @@ -26,16 +26,6 @@ "sdks": "Go, Java", "note": "Deprecated in both SDKs. Superseded by local_activity_execution_failed." }, - { - "name": "long_request_resource_exhausted", - "sdks": "Go", - "note": "" - }, - { - "name": "request_resource_exhausted", - "sdks": "Go", - "note": "" - }, { "name": "resource_slots_cpu_pid_output", "sdks": "Core, Java", diff --git a/docs/best-practices/sdk-worker-alerting.mdx b/docs/best-practices/sdk-worker-alerting.mdx index 127aaf0545..359b77c815 100644 --- a/docs/best-practices/sdk-worker-alerting.mdx +++ b/docs/best-practices/sdk-worker-alerting.mdx @@ -94,7 +94,7 @@ Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). | [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | | [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | | [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | -| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Warning | +| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical | | [Workflow Task schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | | [Activity schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | | [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | @@ -114,11 +114,12 @@ Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failure | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | -| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 5m | Critical | +| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | | [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | | [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | | [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | | [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | +| [Unregistered Activity invocation](/troubleshooting/sdk-execution-failures#unregistered-activity-invocation) — Go SDK only | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | | [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | ## Set up the alerts {/* #set-up-the-alerts */} @@ -138,8 +139,8 @@ The names you query depend on your SDK and metrics reporter: - **Histograms** take a `_seconds_bucket` suffix on the underlying bucket series: `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. - **Gauges take no suffix.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Querying them with `_total` returns no data. -Tag values also vary. -The `status_code` tag is `PascalCase` in the Go SDK (`NotFound`) and `UPPER_SNAKE_CASE` in the Java and Core SDKs (`NOT_FOUND`). +Tag coverage also varies. +`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and the tag can be suppressed through Client options. Some tags are not emitted by every SDK — `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs and `task_queue` only in TypeScript. Confirm the exact names and tags in your own metrics endpoint before writing queries. diff --git a/docs/references/sdk-metrics.mdx b/docs/references/sdk-metrics.mdx index 68f835371a..4726ecab4b 100644 --- a/docs/references/sdk-metrics.mdx +++ b/docs/references/sdk-metrics.mdx @@ -64,9 +64,9 @@ Each metric may have some combination of the following [tags](/glossary#tag) att - `namespace`: Namespace the Worker is bound to - `poller_type`: One of the following: - `workflow_task` + - `workflow_sticky_task` - `activity_task` - `nexus_task` - - `sticky_workflow_task` - `worker_type`: One of the following: - `ActivityWorker` - `WorkflowWorker` @@ -75,6 +75,8 @@ Each metric may have some combination of the following [tags](/glossary#tag) att - `activity_type`: The name of the Activity Function the metric is associated with - `workflow_type`: The name of the Workflow Function the metric is associated with - `operation`: RPC method name; available for metrics related to Temporal Client gRPC requests +- `status_code`: gRPC status code returned by the Temporal Service, in `UPPER_SNAKE_CASE` (for example, `NOT_FOUND`); available on Temporal Client gRPC request failure metrics +- `cause`: reason the Temporal Service rejected a request as resource exhausted (for example, `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`); available on the resource-exhausted metrics Some tags may not be available in every SDK, and Histogram metrics may have different buckets in each SDK. @@ -96,6 +98,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff | [temporal_long_request](#long_request) | Service Client | Counter | Core, Go, Java | | [temporal_long_request_failure](#long_request_failure) | Service Client | Counter | Core, Go, Java | | [temporal_long_request_latency](#long_request_latency) | Service Client | Histogram | Core, Go, Java | +| [temporal_long_request_resource_exhausted](#long_request_resource_exhausted) | Service Client | Counter | Go | | [temporal_nexus_poll_no_task](#nexus_poll_no_task) | Worker | Counter | Core, Go, Java | | [temporal_nexus_task_schedule_to_start_latency](#nexus_task_schedule_to_start_latency) | Worker | Histogram | Core, Go, Java | | [temporal_nexus_task_execution_failed](#nexus_task_execution_failed) | Worker | Counter | Core, Go, Java | @@ -106,6 +109,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff | [temporal_request](#request) | Service Client | Counter | Core, Go, Java | | [temporal_request_failure](#request_failure) | Service Client | Counter | Core, Go, Java | | [temporal_request_latency](#request_latency) | Service Client | Histogram | Core, Go, Java | +| [temporal_request_resource_exhausted](#request_resource_exhausted) | Service Client | Counter | Go | | [temporal_resource_slots_cpu_usage](#resource_slots_cpu_usage) | Worker | Gauge | Core, Java | | [temporal_resource_slots_mem_usage](#resource_slots_mem_usage) | Worker | Gauge | Core, Java | | [temporal_sticky_cache_hit](#sticky_cache_hit) | Worker | Counter | Core, Go, Java | @@ -256,7 +260,7 @@ This number is included into the total `long_request` counter for long poll RPC - Tags: `namespace`, `operation`, `status_code` The `status_code` tag carries the gRPC status code returned by the Temporal Service. -Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`. +Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on. The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `long_request_latency` @@ -267,6 +271,18 @@ Latency of a Temporal Client gRPC long poll request. - Available in: Core, Go, Java - Tags: `namespace`, `operation` +### `long_request_resource_exhausted` + +Temporal Client made an RPC long poll request that the Temporal Service rejected as resource exhausted. +These failures are also counted in `long_request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection. + +- Type: Counter +- Available in: Go +- Tags: `namespace`, `operation`, `cause` + +The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`. +The cause determines the remedy, so prefer this metric over `long_request_failure` when investigating throttling. + ### `nexus_poll_no_task` A Nexus Worker poll for a Nexus Task timed out, and no Nexus Task is available to pick from the Task Queue. @@ -359,7 +375,7 @@ This number is included into the total `request` counter for RPC requests. - Tags: `namespace`, `operation`, `status_code` The `status_code` tag carries the gRPC status code returned by the Temporal Service. -Its casing differs by SDK: the Go SDK reports `PascalCase` values such as `NotFound`, while the Java and Core SDKs report `UPPER_SNAKE_CASE` values such as `NOT_FOUND`. +Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on. The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `request_latency` @@ -370,6 +386,18 @@ Latency of a Temporal Client gRPC request. - Available in: Core, Go, Java - Tags: `namespace`, `operation` +### `request_resource_exhausted` + +Temporal Client made an RPC request that the Temporal Service rejected as resource exhausted. +These failures are also counted in `request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection. + +- Type: Counter +- Available in: Go +- Tags: `namespace`, `operation`, `cause` + +The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`. +The cause determines the remedy, so prefer this metric over `request_failure` when investigating throttling. + ### `resource_slots_cpu_usage` CPU usage as a value between 0 and 100. As perceived by the resource-based slots tuner, if diff --git a/docs/troubleshooting/sdk-execution-failures.mdx b/docs/troubleshooting/sdk-execution-failures.mdx index 86fdb2ddb7..dbdb077416 100644 --- a/docs/troubleshooting/sdk-execution-failures.mdx +++ b/docs/troubleshooting/sdk-execution-failures.mdx @@ -44,7 +44,7 @@ A non-determinism error does not resolve on its own. 1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause. 1. **Determine whether this is a code change or a deploy artifact.** Common causes: - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code. - - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes, which is why this alert needs a `for` duration longer than your deploy takes. + - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes. - Changed Activity or Timer parameters in existing Workflow code without versioning. 1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). 1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue. @@ -152,7 +152,29 @@ At scale, sustained retry bursts put significant pressure on Task matching and t 1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not. 1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying. 1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. -1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`. All SDKs support this and suppress this metric for them, which lets this alert track unexpected failures without per-`activity_type` threshold tuning. +1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`, which suppresses this metric for them and lets the alert track unexpected failures without per-`activity_type` threshold tuning. Confirm your SDK version supports the category before relying on it. + +Note that internal failures — a context propagation error or a context timeout, rather than an Activity returning a failure — increment this counter regardless of category. + +## Unregistered Activity invocation {/* #unregistered-activity-invocation */} + +**Metric:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type` + +A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for. + +This metric is emitted by the Go SDK only. + +**Why it matters.** +The Activity cannot execute. It will keep being retried against a Worker that cannot run it until the Activity's `scheduleToClose` timeout expires, or indefinitely if no such timeout is set — so the Workflow Execution waiting on it makes no progress. + +This is almost always a deployment error rather than a runtime condition: Workflow code that schedules an Activity the deployed Worker does not register. It does not resolve on its own. + +**Triage.** + +1. **Identify the Activity and the Task Queue.** The `activity_type` and `task_queue` tags name both. The `workflow_type` tag identifies which Workflow is scheduling it. +1. **Check whether the Activity is registered on the right Worker.** Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue. +1. **Check for a partial rollout.** If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout. +1. **Check for a renamed Activity.** Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard. ## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */} diff --git a/docs/troubleshooting/sdk-request-failures.mdx b/docs/troubleshooting/sdk-request-failures.mdx index c748386f8d..88fcdaec49 100644 --- a/docs/troubleshooting/sdk-request-failures.mdx +++ b/docs/troubleshooting/sdk-request-failures.mdx @@ -31,7 +31,7 @@ Which status code appears on which operation determines how urgent the failure i Two naming details will affect your queries. `UpdateWithStartWorkflowExecution` appears in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. -And the `status_code` tag can be suppressed through Client options — confirm it is present in your metrics endpoint before filtering on it. +And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names — the tag can also be suppressed through Client options, so confirm it is present in your metrics endpoint before filtering on it. ::: @@ -123,8 +123,9 @@ Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread **Triage.** -1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, or an open circuit breaker are different problems with different remedies. -1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare current throughput against your Namespace's [service limits](/cloud/limits) and open a support request if you need them raised. +1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, and an open circuit breaker are different problems with different remedies. On the Go SDK, [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) carries a `cause` tag with the reason — group by it to see which limit you are hitting. On other SDKs, use the guidance below to narrow it down. +1. **Check your traffic against your Namespace limits.** If the cause is a rate limit — `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT` — compare current throughput against your Namespace's [service limits](/cloud/limits) on Temporal Cloud, and open a support request if you need them raised. +1. **Treat overload and circuit-breaker causes as capacity problems.** `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED` and `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN` mean the Temporal Service is shedding load to protect itself. A higher limit will not help; the Service needs capacity, or your workload needs to slow down. 1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure. :::note Self-hosted Temporal Service @@ -151,7 +152,7 @@ Sustained throttling here escalates into [Worker Task slots exhausted](/troubles **Triage.** -1. **Identify the throttle cause,** as in the section above. +1. **Identify the throttle cause,** as in the section above — on the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`. 1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. 1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) — slots are not released until the respond call succeeds, so throttling here drains the slot pool. @@ -177,7 +178,7 @@ The Temporal Service throttles poll operations before it throttles respond or us **Triage.** -1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. +1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. On the Go SDK, [`temporal_long_request_resource_exhausted`](/references/sdk-metrics#long_request_resource_exhausted) carries the `cause` tag for poll operations. 1. **Check whether you are over-polling.** A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against [Worker performance](/develop/worker-performance) guidance before assuming the limit is too low. 1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet. 1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare against your Namespace's [service limits](/cloud/limits). diff --git a/docs/troubleshooting/sdk-worker-capacity.mdx b/docs/troubleshooting/sdk-worker-capacity.mdx index 32a7f2695c..15f73567c6 100644 --- a/docs/troubleshooting/sdk-worker-capacity.mdx +++ b/docs/troubleshooting/sdk-worker-capacity.mdx @@ -157,6 +157,11 @@ Check overall cluster health: service errors by Namespace, persistence availabil Workflow Tasks are sitting in the queue before a Worker picks them up. Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. +Two other pages give thresholds for this metric, for different purposes. +[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth investigating — use that when you are tuning Worker capacity, not paging on it. +[Worker health](/cloud/worker-health#schedule-to-start-latency) supplies ready-made Prometheus queries for both this metric and the Activity equivalent. +The thresholds on this page are set higher because they are meant to page someone. + **Why it matters.** Schedule-to-start latency adds directly to Workflow end-to-end latency. Every second a Task waits is a second added to how long your Workflows take. @@ -186,6 +191,8 @@ Check the server-side concurrent poller count for this Task Queue to confirm whe Activity Tasks are sitting in the queue before a Worker picks them up. +As with the Workflow Task equivalent above, [Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike) gives a lower investigation threshold for this metric, and [Worker health](/cloud/worker-health#schedule-to-start-latency) supplies the Prometheus query. + **Why it matters.** Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database. From 264e8a38148e88561e45846cc9f729e87f29a260 Mon Sep 17 00:00:00 2001 From: Auston Li Date: Tue, 4 Aug 2026 18:00:02 -0700 Subject: [PATCH 4/5] Rewrite the SDK alerting pages in the repo's voice The prose read as machine-generated and did not match the house style. Removed all 76 em dashes. The five closest existing pages (performance-bottlenecks, worker-health, best-practices/worker, worker-performance, blob-size-limit-error) use none across 2,226 lines, so the appositive dash was a style break, not just a tic. Replaced with sentence breaks, colons, and parentheses. Loosened the register throughout. Contractions where they read naturally, plain verbs over formal ones, and second person where the reader is the one taking action: "burning its time re-running Event History" over "spending its time re-executing", "your Workers are finishing too late, over and over" over "Workers are consistently finishing too late". Cut the constructions that made the hedged sections read like legal copy. "A zero rate on its own does not prove a problem" became "Check for demand before you act on this one", and the sticky cache section now says "Alert on this gauge by itself and you will get paged for Workers that are doing nothing wrong" instead of "Alerting on this gauge alone will page you on idle Workers". Also dropped rhetorical filler: "which is why this is worth treating", "the for duration is doing all the work", "Treat it as actionable only alongside", and the three-item parallel lists. No technical content, thresholds, metric names, tags, anchors, or links changed. Verified: build exits 0, check:metrics, check:metrics:sdks, and check:orphans all pass. Co-Authored-By: Claude Opus 5 --- docs/best-practices/sdk-worker-alerting.mdx | 94 +++++++-------- .../sdk-execution-failures.mdx | 100 ++++++++-------- docs/troubleshooting/sdk-request-failures.mdx | 75 ++++++------ docs/troubleshooting/sdk-worker-capacity.mdx | 107 +++++++++--------- 4 files changed, 193 insertions(+), 183 deletions(-) diff --git a/docs/best-practices/sdk-worker-alerting.mdx b/docs/best-practices/sdk-worker-alerting.mdx index 359b77c815..4bed58f579 100644 --- a/docs/best-practices/sdk-worker-alerting.mdx +++ b/docs/best-practices/sdk-worker-alerting.mdx @@ -16,9 +16,10 @@ tags: - Workers --- -Temporal SDKs emit metrics from your Worker processes that expose failure modes the Temporal Service cannot see on your behalf: Workflow code that fails on replay, Workers that stop polling, Task slots that never free up, and Local Activities that outlive the Workflow Task heartbeat window. +Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window. +Nobody catches those for you. -This page recommends a set of alerts covering those failure modes. +This page recommends a set of alerts that covers them. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). @@ -26,34 +27,34 @@ For Worker sizing and tuning, see [Worker performance](/develop/worker-performan ## Start with these five {/* #start-with-these-five */} -If you are adding Worker alerts for the first time, start here. -These five catch the failure modes that stop Workflow Executions outright, and they produce the fewest false positives. +If you are setting up Worker alerts for the first time, start here. +These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing. -1. [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) — Workers have stopped polling. -1. [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) — Workflow code no longer matches recorded history. -1. [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) — Executions are being terminated with work lost. -1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) — Tasks are backing up. -1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) — starts, Signals, and Updates are being throttled. +1. [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected). Your Workers have stopped polling. +1. [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error). Workflow code no longer matches recorded history. +1. [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). Executions are being terminated and losing work. +1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up. +1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled. Add the rest once these are tuned and quiet. ## Choose your thresholds {/* #choose-your-thresholds */} Every threshold on this page is a starting point, not a service level objective. +A high-throughput Task Queue needs different values than a bursty batch workload. -A high-throughput Task Queue, a latency-sensitive Namespace, and a bursty batch workload each want different values. -Two things to tune per alert: +There are two knobs per alert: -- **The threshold** determines what counts as unhealthy. Set it against observed p99 during a known-good period, not against the value listed here. -- **The `for` duration** determines how long the condition must hold before the alert fires. Shorter durations detect problems faster and fire more often on transient spikes. Longer durations suppress noise and delay detection. +- **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here. +- **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection. -For alerts on binary conditions — any occurrence of a gRPC status code, a gauge reaching zero — the `for` duration is doing all the work. -Those conditions occur briefly during normal operation, so a `for` duration that is too short will page you on routine events. +Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero. There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page. +All of those conditions show up briefly during normal operation, so set it generously. ### Alerts that fire during normal operation {/* #expected-firing */} -Several of these conditions are expected in a healthy deployment. -Set the `for` duration long enough to ride them out, and confirm your value against your own deploy cadence: +Several of these conditions show up in a perfectly healthy deployment. +Set the `for` duration long enough to ride them out, and check your value against how long your own deploys take: | Event | Alerts it can trigger | | --- | --- | @@ -65,13 +66,13 @@ Set the `for` duration long enough to ride them out, and confirm your value agai ## Recommended alert set {/* #recommended-alert-set */} -The condition column gives the tag filters that distinguish each alert. -Group each alert by `namespace` and by the tags in its condition, so a firing alert tells you which Namespace, operation, or Task Queue is affected. +The condition column gives the tag filters that tell each alert apart. +Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from. ### Request failures {/* #request-failures */} -These fire on gRPC responses from the Temporal Service to your Worker or Client. -Triage guidance: [SDK request failures](/troubleshooting/sdk-request-failures). +These fire on gRPC responses coming back from the Temporal Service to your Worker or Client. +For triage, see [SDK request failures](/troubleshooting/sdk-request-failures). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | @@ -86,8 +87,8 @@ Triage guidance: [SDK request failures](/troubleshooting/sdk-request-failures). ### Worker capacity {/* #worker-capacity */} -These fire when Workers stop keeping up with the Task Queue. -Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). +These fire when your Workers stop keeping up with the Task Queue. +For triage, see [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | @@ -102,15 +103,15 @@ Triage guidance: [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). :::note `worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. -It cannot be used with resource-based slot suppliers — see [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). -If your Workers use resource-based tuning, alert on schedule-to-start latency instead and treat it as the signal that Worker capacity is short. +It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short. ::: ### Execution failures {/* #execution-failures */} -These fire on Workflow and Activity code failing on the Worker. -Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failures). +These fire when your Workflow or Activity code fails on the Worker. +For triage, see [SDK execution failures](/troubleshooting/sdk-execution-failures). | Failure mode | Metric | Condition | Threshold | `for` | Default severity | | --- | --- | --- | --- | --- | --- | @@ -119,45 +120,48 @@ Triage guidance: [SDK execution failures](/troubleshooting/sdk-execution-failure | [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | | [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | | [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | -| [Unregistered Activity invocation](/troubleshooting/sdk-execution-failures#unregistered-activity-invocation) — Go SDK only | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | +| [Unregistered Activity invocation](/troubleshooting/sdk-execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | | [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | ## Set up the alerts {/* #set-up-the-alerts */} Before you can alert on these metrics, your Workers must be emitting them and your monitoring system must be scraping them: -- **Temporal Cloud** — [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). -- **Self-hosted** — [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). +- **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). +- **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). ### Resolve the metric names for your setup {/* #resolve-metric-names */} The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). The names you query depend on your SDK and metrics reporter: -- All metrics carry a `temporal_` prefix. -- **Counters** take a `_total` suffix when scraped through Prometheus: `temporal_request_failure_total`. -- **Histograms** take a `_seconds_bucket` suffix on the underlying bucket series: `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. -- **Gauges take no suffix.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Querying them with `_total` returns no data. +- Every metric carries a `temporal_` prefix. +- **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`. +- **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. +- **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why. -Tag coverage also varies. -`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and the tag can be suppressed through Client options. -Some tags are not emitted by every SDK — `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs and `task_queue` only in TypeScript. +Tag coverage varies too. +`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely. +Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript. Confirm the exact names and tags in your own metrics endpoint before writing queries. ## Route alerts by severity {/* #route-alerts-by-severity */} -The severity in each table is a default routing suggestion, not a property of the metric. +The severity in each table is a suggested routing default, not a property of the metric. Tune it to your workload. -As a starting point, page on alerts whose condition means Workflow Executions have stopped progressing or are progressing with data loss or duplicate side effects. -Send the rest to a review channel. +Start by paging on the alerts that mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects. +Send the rest to a channel someone reads during the day. -Two reasons the defaults will not fit every deployment: +The defaults won't fit every deployment, for two reasons. -- Some Critical rows are latency thresholds or leading indicators rather than confirmed stoppage. Elevated Workflow Task execution latency matters a great deal on a latency-sensitive Namespace and very little on a batch workload where Tasks routinely run long. -- Some Warning rows can still cause duplicate side effects. A NOT_FOUND on Activity heartbeat means the Activity attempt was already timed out and will re-execute from scratch, which is a real problem if that Activity is not idempotent. +Some Critical rows are latency thresholds or early warnings rather than confirmed stoppage. +High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. -Several of these alerts have causal relationships. -Task slots exhausted drives pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. +Some Warning rows can still lose you data. +A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start, which is a real problem if that Activity isn't idempotent. + +These alerts are also chained together. +Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. When several fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/troubleshooting/sdk-execution-failures.mdx b/docs/troubleshooting/sdk-execution-failures.mdx index dbdb077416..3e4d9210a9 100644 --- a/docs/troubleshooting/sdk-execution-failures.mdx +++ b/docs/troubleshooting/sdk-execution-failures.mdx @@ -16,69 +16,69 @@ tags: - Troubleshooting --- -This guide covers failures that occur while your Workflow and Activity code is executing on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that run past the Workflow Task heartbeat timeout. +This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`. -The reason determines what the Temporal Service does next, and the difference matters: two of the three cause indefinite retries, one causes immediate termination. +The reason decides what the Temporal Service does next, and the difference is worth knowing: two of the three retry forever, one terminates the Execution on the spot. Alert on each `failure_reason` separately. ## Non-determinism error {/* #non-determinism-error */} **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError` -Replay produced a different command sequence than the one recorded in Event History. -The Worker detected that the Workflow code it is running does not match the commands the Workflow Execution has already produced. +Replay produced a different sequence of commands than the one recorded in Event History. +The Worker noticed that the Workflow code it is running no longer matches what the Execution has already done. **Why it matters.** -Affected Executions are not progressing. -The Temporal Service retries the Workflow Task continuously, adding pressure to your Workflow Workers, and by default the Executions stay in Running status — prolonging their end-to-end time indefinitely. -A non-determinism error does not resolve on its own. +Affected Executions stop making progress. +The Temporal Service retries the Workflow Task over and over, loading up your Workflow Workers, and by default those Executions sit in Running status the whole time. +This will not clear up on its own. **Triage.** 1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures. 1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause. 1. **Determine whether this is a code change or a deploy artifact.** Common causes: - - A code change added, removed, or reordered commands — Activity scheduling, Timers, Signals, Child Workflows — without a versioning guard. In-flight Executions that built History under the old code fail on the new code. + - A code change added, removed, or reordered commands such as Activity scheduling, Timers, Signals, or Child Workflows, without a versioning guard. Executions that built their History under the old code fail on the new code. - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes. - Changed Activity or Timer parameters in existing Workflow code without versioning. -1. **Roll back if it is not resolving.** If the errors started after a deploy and are not clearing on their own, roll the Worker back to the previous version. Affected Executions resume on their next Workflow Task retry once compatible code is running. Then introduce a proper versioning guard before redeploying — see [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). -1. **Watch Worker pressure.** Continuous retries put sustained load on Workflow Workers. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high) — a high volume of retries can saturate capacity and affect healthy Executions on the same Task Queue. +1. **Roll back if it isn't clearing.** If the errors started after a deploy and aren't going away, roll the Worker back. Affected Executions pick up again on their next Workflow Task retry once compatible code is running. Then add a proper versioning guard before you redeploy. See [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). +1. **Watch Worker pressure.** All those retries add up. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high). Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue. ## gRPC message too large {/* #grpc-message-too-large */} **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge` -The Workflow Task response payload exceeded the gRPC message size limit. -The Worker attempted `RespondWorkflowTaskCompleted` and the response was rejected — by the gRPC library on the SDK side, by a proxy or load balancer in the path, or by the gRPC library on the Temporal Service side on receive. +The Workflow Task response was bigger than the gRPC message size limit. +The Worker tried `RespondWorkflowTaskCompleted` and something rejected it: the gRPC library on the SDK side, a proxy or load balancer in the path, or the gRPC library on the Temporal Service side when it went to receive. -Because the Temporal Service never saw the original request, the SDK sends a follow-up `RespondWorkflowTaskFailed` with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. -Since replay would produce the same oversized response on every attempt, the Temporal Service terminates the Workflow Execution rather than retrying it. +The Temporal Service never saw that original request, so the SDK follows up with `RespondWorkflowTaskFailed` and cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. +Replay would build the same oversized response every time, so the Service terminates the Workflow Execution instead of retrying it. -For the payload size limits themselves and the equivalent Activity Task behavior — which retries rather than terminating — see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). +For the payload size limits themselves, and for the Activity Task version of this (which retries instead of terminating), see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). **Why it matters.** -Affected Executions are terminated immediately and permanently, with `TERMINATED` status and no retry. -Any in-progress work in those Executions is lost, and they must be restarted manually. +Affected Executions end immediately and permanently, with `TERMINATED` status and no retry. +Whatever work was in progress is gone, and someone has to restart them by hand. -This is the one `failure_reason` on this page that ends Executions rather than retrying them, which is why it warrants a short `for` duration. +This is the one `failure_reason` on this page that kills Executions instead of retrying them, so give it a short `for` duration. **Triage.** 1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History. 1. **Find what is oversized.** The fix depends entirely on which part of the response is too large: - - **Oversized Activity inputs or outputs.** Move large payloads out of band — store them in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern. + - **Oversized Activity inputs or outputs.** Move the payload out of band: put it in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern. - **Accumulated Signals or Updates.** A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals. - **Too many commands in one response.** A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks. 1. **Fix and deploy before restarting anything.** Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI. :::note Self-hosted Temporal Service -Check the Workflow terminate rate on your server dashboard — a spike alongside this metric confirms Executions are being terminated at volume. +Check the Workflow terminate rate on your server dashboard. A spike alongside this metric confirms Executions are being terminated in bulk. ::: @@ -86,24 +86,24 @@ Check the Workflow terminate rate on your server dashboard — a spike alongside **Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError` -Sustained Workflow Task failures from unhandled exceptions and panics in Workflow code that the SDK catches and reports. -`WorkflowError` is the catch-all reason: it covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. +Workflow Tasks are failing steadily from unhandled exceptions and panics in Workflow code that the SDK catches and reports. +`WorkflowError` is the catch-all reason. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. **Why it matters.** The Temporal Service retries the Workflow Task. -If the error is deterministic and reproduces on every replay, the Execution is stuck retrying indefinitely, consuming Worker capacity and staying in a permanently unhealthy state. +If the error is deterministic and shows up on every replay, the Execution is stuck retrying forever, burning Worker capacity and never getting healthy. -At high rates the retry pressure saturates Workflow Worker slots and affects healthy Executions on the same Task Queue. -Unlike `GrpcMessageTooLarge`, the Temporal Service does not terminate the Execution, so the impact compounds until you resolve it. +At high rates the retry pressure fills up your Workflow Worker slots and starts affecting healthy Executions on the same Task Queue. +Unlike `GrpcMessageTooLarge`, the Temporal Service won't terminate the Execution for you, so this gets worse the longer you leave it. **Triage.** 1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. The `workflow_type` tag on the metric narrows which Workflow is failing. 1. **Determine which failure mode this is.** `WorkflowError` covers several: - - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool, caused by `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` being too low for the number of concurrent Executions. New Workflow Tasks are rejected before they can execute. Raise the thread count, and check whether the Worker pool needs to scale out as well. - - **Unhandled exception in Workflow code.** A bug or unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event identifies the error. - - **Data Converter error.** A failure serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration. -1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion often occur together under load, and a CPU-starved Worker completes Workflow Tasks more slowly, accelerating both. + - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool. `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` is too low for the number of concurrent Executions, so new Workflow Tasks get rejected before they run. Raise the thread count, and think about whether the Worker pool needs to scale out too. + - **Unhandled exception in Workflow code.** A bug or an unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event names the error. + - **Data Converter error.** Something failed serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration. +1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse. 1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running. ## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */} @@ -113,8 +113,8 @@ Unlike `GrpcMessageTooLarge`, the Temporal Service does not terminate the Execut Workflow Tasks are taking too long to execute on the Worker. The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks. -A batch workload where Workflow Tasks routinely run long may sit above this threshold without any problem. -Set the threshold against your own observed p99, and treat the default as a signal only if your Workflows are latency-sensitive. +A batch workload where Workflow Tasks routinely run long can sit above this threshold all day without anything being wrong. +Set the threshold from your own observed p99, and only treat the default as meaningful if your Workflows are latency-sensitive. **Why it matters.** The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue. @@ -127,34 +127,34 @@ At scale this compounds: more timeouts cause more cold replays, cold replays dri **Triage.** -1. **Check replay latency.** Check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-executing Event History rather than running new commands — usually caused by large histories, slow Data Converter execution during replay, or a high cache eviction rate forcing cold replays. +1. **Check replay latency.** Look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-running Event History rather than into new commands. Large histories, a slow Data Converter during replay, or a high cache eviction rate forcing cold replays are the usual culprits. 1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load). 1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker. 1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — the SDK holds the slot until the respond call succeeds, inflating this metric even when the Workflow code finished quickly. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly. ## Activity execution failures elevated {/* #activity-execution-failures-elevated */} **Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type` -Activities are explicitly failing — returning failures rather than timing out — at a sustained rate. +Activities are failing outright at a sustained rate: returning failures, not timing out. -`ApplicationFailure` instances marked with category `BENIGN` are excluded and do not increment this counter, so this metric tracks unexpected failures only to the extent your application uses benign failures correctly. +`ApplicationFailure` instances marked with category `BENIGN` don't increment this counter, so how well this metric tracks only the unexpected failures depends on how consistently your application marks the expected ones. **Why it matters.** -A high failure rate drives a burst of retry Tasks. -If Workers cannot keep up with the retry volume, the Activity Task backlog grows — see [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). -At scale, sustained retry bursts put significant pressure on Task matching and the underlying database. +A high failure rate means a burst of retry Tasks. +If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). +At scale, sustained retry bursts put real pressure on Task matching and the database underneath. **Triage.** 1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. -1. **Determine whether this is transient or a bug.** A downstream service outage, network partition, or database timeout recovers on its own — watch whether the rate falls. A persistent code bug does not. -1. **Check downstream service health.** A degraded dependency is a common cause of sustained failure bursts. If the dependency is throttling, confirm your [Retry Policy](/encyclopedia/retry-policies) has appropriate backoff — without it, retry bursts amplify the pressure you are already applying. +1. **Work out whether this is transient or a bug.** A downstream outage, a network partition, or a database timeout will recover on its own, so watch whether the rate falls. A code bug won't. +1. **Check downstream service health.** A struggling dependency is a common cause of sustained failure bursts. If it is throttling you, check that your [Retry Policy](/encyclopedia/retry-policies) has sensible backoff. Without it, your retries pile more pressure onto something that is already overloaded. 1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. -1. **Mark expected failures as benign.** If your design intentionally fails Activities — polling patterns, Saga compensations, flow control through exceptions — mark those `ApplicationFailure` instances with category `BENIGN`, which suppresses this metric for them and lets the alert track unexpected failures without per-`activity_type` threshold tuning. Confirm your SDK version supports the category before relying on it. +1. **Mark expected failures as benign.** If your design fails Activities on purpose, as polling patterns, Saga compensations, and flow control through exceptions all do, mark those `ApplicationFailure` instances with category `BENIGN`. That keeps them out of this metric and lets the alert track only the failures you didn't expect, without tuning a threshold per `activity_type`. Check that your SDK version supports the category before you rely on it. -Note that internal failures — a context propagation error or a context timeout, rather than an Activity returning a failure — increment this counter regardless of category. +One caveat: internal failures increment this counter no matter what category you set. Those are things like a context propagation error or a context timeout, rather than an Activity returning a failure. ## Unregistered Activity invocation {/* #unregistered-activity-invocation */} @@ -165,9 +165,9 @@ A Workflow scheduled an Activity that the Worker polling that Task Queue has no This metric is emitted by the Go SDK only. **Why it matters.** -The Activity cannot execute. It will keep being retried against a Worker that cannot run it until the Activity's `scheduleToClose` timeout expires, or indefinitely if no such timeout is set — so the Workflow Execution waiting on it makes no progress. +The Activity can't run. It keeps getting retried against a Worker that has no implementation for it, until the Activity's `scheduleToClose` timeout expires, or forever if you haven't set one. Meanwhile the Workflow Execution waiting on it goes nowhere. -This is almost always a deployment error rather than a runtime condition: Workflow code that schedules an Activity the deployed Worker does not register. It does not resolve on its own. +This is nearly always a deployment mistake rather than a runtime condition: Workflow code scheduling an Activity that the deployed Worker doesn't register. It won't fix itself. **Triage.** @@ -187,9 +187,9 @@ A [Local Activity](/local-activity) is running past the Workflow Task heartbeat A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task. That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout. -To keep the Task alive, the SDK sends Workflow Task heartbeats — repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still in progress and request more time. -The Service allows this up to the Workflow Task heartbeat timeout. -Past that, it times out the Task and reschedules it on the normal Task Queue. +To keep the Task alive, the SDK sends Workflow Task heartbeats: repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still going and ask for more time. +The Service goes along with this up to the Workflow Task heartbeat timeout. +After that it times the Task out and reschedules it on the normal Task Queue. Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats. So when the Task is rescheduled, every Local Activity in it runs again from the beginning. @@ -210,7 +210,7 @@ A single attempt running for 30 minutes is a design problem, not a tuning proble **Triage.** 1. **Identify the affected Local Activity.** The `activity_type` tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids. -1. **Find what it is blocked on.** A Local Activity running this long is almost always blocked on a downstream call — a slow service, a slow query, or a network call with a very long timeout. Fix the dependency, or shorten the timeout on the call so the Local Activity fails fast instead of hanging. +1. **Find what it is blocked on.** A Local Activity running this long is nearly always stuck on a downstream call: a slow service, a slow query, or a network call with a very generous timeout. Fix the dependency, or shorten that timeout so the Local Activity fails fast instead of hanging. 1. **Check whether a retry chain is accumulating.** A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check [`temporal_local_activity_execution_failed`](/references/sdk-metrics#local_activity_execution_failed) for the same `activity_type`, and fix the underlying failure first. -1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have already timed out heartbeating Workflow Tasks. Check Worker logs for timeout errors and Event History for `WorkflowTaskTimedOut` events. If they are present, Local Activities have already re-executed — verify idempotency and address any duplicate side effects. +1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have timed out heartbeating Workflow Tasks already. Look for timeout errors in Worker logs and `WorkflowTaskTimedOut` events in Event History. If they are there, your Local Activities have already run twice, so check whether they are idempotent and clean up any duplicate side effects. 1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed. diff --git a/docs/troubleshooting/sdk-request-failures.mdx b/docs/troubleshooting/sdk-request-failures.mdx index 88fcdaec49..be284a5508 100644 --- a/docs/troubleshooting/sdk-request-failures.mdx +++ b/docs/troubleshooting/sdk-request-failures.mdx @@ -16,22 +16,22 @@ tags: - Troubleshooting --- -This guide covers gRPC failures and elevated latency on requests that Temporal SDK Workers and Clients make to the Temporal Service. +This guide covers gRPC failures and slow requests between your Temporal SDK Workers or Clients and the Temporal Service. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). [`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. [`temporal_long_request_failure`](/references/sdk-metrics#long_request_failure) covers poll operations and long-poll `GetWorkflowExecutionHistory`. Both carry `namespace`, `operation`, and `status_code` tags. -Which status code appears on which operation determines how urgent the failure is. +How urgent a failure is depends on which status code turned up on which operation. :::note -Two naming details will affect your queries. -`UpdateWithStartWorkflowExecution` appears in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. -And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names — the tag can also be suppressed through Client options, so confirm it is present in your metrics endpoint before filtering on it. +Two naming details will trip up your queries. +`UpdateWithStartWorkflowExecution` shows up in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. +And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names. Client options can also turn the tag off, so check that it is present in your metrics endpoint before you filter on it. ::: @@ -46,26 +46,26 @@ There are three causes: - The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished. - The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up. -The second and third causes occur during normal operation, so isolated occurrences are not a problem. -What matters is a sustained rate. +The last two happen during normal operation, so a few of these are nothing to worry about. +A sustained rate is. **Why it matters.** -The result the Worker just produced was discarded. +The Temporal Service threw away the result your Worker just produced. For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a [Sticky Execution](/sticky-execution) cache eviction and a cold replay on the retry. -A sustained rate means Workers are consistently finishing too late. -Every discarded result is Worker capacity spent on work that was thrown away, and every rescheduled task adds to Workflow end-to-end latency. +A sustained rate means your Workers are finishing too late, over and over. +Every discarded result is Worker capacity you paid for and got nothing back from, and every rescheduled task adds to how long your Workflows take end to end. -If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried task. -Local Activity results are not written to Event History between Workflow Task heartbeats, so re-execution means running them again. -If they are not idempotent, this produces duplicate side effects with real business impact. +If you run [Local Activities](/local-activity), a Workflow Task timeout makes them run again from the start on the retried task. +Their results are not written to Event History between Workflow Task heartbeats, so there is nothing to resume from. +If they aren't idempotent, you get duplicate side effects, and that is a business problem rather than a monitoring one. **Triage.** 1. **Rule out the expected causes first.** Check the status of a few affected Executions in the Temporal UI or with `temporal workflow describe`. If they completed, were terminated, or hit their Run Timeout, the NOT_FOUND is expected. Check Worker restart counts in your infrastructure observability stack for the same reason. If either explains the volume, stop here. 1. **Check task execution latency.** For Workflow Tasks, check [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency). For Activities, check [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause. -1. **Check replay latency.** If Workflow Task execution latency is elevated, check [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. High replay latency means the Worker is spending its time re-executing Event History rather than running new commands — check for large histories and slow [Data Converter](/dataconversion) execution during replay. +1. **Check replay latency.** If Workflow Task execution latency is high, look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. When replay latency is high, the Worker is burning its time re-running Event History instead of getting to the new commands. Look for large histories and a slow [Data Converter](/dataconversion). 1. **Check Worker resources.** High CPU on the Worker slows task execution directly. Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays. 1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations). Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands. @@ -87,10 +87,10 @@ Cancellation returns `CancelRequested=true` in the heartbeat response body rathe **Why it matters.** If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. -The Activity re-executes from scratch on the next attempt, so a non-idempotent Activity produces duplicate side effects — which is why this is worth treating more seriously than its default severity suggests. +The Activity runs again from the start on the next attempt, so a non-idempotent Activity will duplicate its side effects. That is worth treating more seriously than the default Warning severity suggests. -A sustained rate means the Worker is consistently failing to heartbeat within the configured interval. -The Activity will keep timing out on every attempt until the cause is fixed, holding Task slots and generating retry tasks the whole time. +A sustained rate means the Worker keeps missing its heartbeat window. +The Activity will time out on every attempt until you fix the cause, holding Task slots and generating retry tasks the whole time. **Triage.** @@ -123,16 +123,17 @@ Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread **Triage.** -1. **Identify the throttle cause.** The cause determines the fix: a Namespace rate limit, a concurrency limit, system-wide overload, and an open circuit breaker are different problems with different remedies. On the Go SDK, [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) carries a `cause` tag with the reason — group by it to see which limit you are hitting. On other SDKs, use the guidance below to narrow it down. -1. **Check your traffic against your Namespace limits.** If the cause is a rate limit — `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT` — compare current throughput against your Namespace's [service limits](/cloud/limits) on Temporal Cloud, and open a support request if you need them raised. +1. **Find out which limit you hit.** A Namespace rate limit, a concurrency limit, system-wide overload, and an open circuit breaker are four different problems with four different fixes. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by its `cause` tag and it will tell you directly. On other SDKs, work through the steps below. +1. **Check your traffic against your Namespace limits.** For a rate limit cause (`RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT`), compare current throughput against your Namespace's [service limits](/cloud/limits) on Temporal Cloud, and open a support request if you need them raised. 1. **Treat overload and circuit-breaker causes as capacity problems.** `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED` and `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN` mean the Temporal Service is shedding load to protect itself. A higher limit will not help; the Service needs capacity, or your workload needs to slow down. 1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure. :::note Self-hosted Temporal Service -Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution` — slow persistence is the most common root cause of cascading throttling. -If the cause is a rate limit, `frontend.namespaceRPS` may be set too low for your traffic, but only raise it after confirming persistence is healthy. -If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself and needs capacity, not a higher limit. +Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`. +Slow persistence is the usual root cause of throttling that cascades like this. +If the cause is a rate limit, `frontend.namespaceRPS` may be too low for your traffic, but confirm persistence is healthy before you raise it. +If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself. It needs capacity, not a higher limit. ::: @@ -145,20 +146,20 @@ The SDK retries automatically, but a delayed respond call has a compounding cost **Why it matters.** This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations). -If throttling persists long enough, the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task is rescheduled — with the cache eviction, cold replay, and Local Activity re-execution consequences described in that section. +If throttling goes on long enough the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task gets rescheduled, with all the cache eviction, cold replay, and Local Activity re-execution that section describes. -Meanwhile every in-flight task holds its slot, reducing the concurrency available for new work. -Sustained throttling here escalates into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). +Meanwhile every in-flight task is still holding its slot, so you have less concurrency for new work. +Left alone, this turns into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). **Triage.** -1. **Identify the throttle cause,** as in the section above — on the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`. +1. **Find out which limit you hit,** as in the section above. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`. 1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. -1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) — slots are not released until the respond call succeeds, so throttling here drains the slot pool. +1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). Slots aren't released until the respond call succeeds, so throttling here drains the pool. :::note Self-hosted Temporal Service -Check persistence latency filtered to `UpdateWorkflowExecution` — slow persistence on this operation is the most common root cause of throttling on respond operations. +Check persistence latency filtered to `UpdateWorkflowExecution`. Slow persistence on that operation is the usual root cause of throttling on respond operations. ::: @@ -185,7 +186,8 @@ The Temporal Service throttles poll operations before it throttles respond or us :::note Self-hosted Temporal Service -If poll operations are being throttled at scale, the Namespace concurrent poller limit may need raising through `frontend.namespaceCount` or `frontend.globalNamespaceCount` — but scale Worker capacity first if schedule-to-start latency is the actual problem. +If poll operations are being throttled at scale, you may need to raise the Namespace concurrent poller limit through `frontend.namespaceCount` or `frontend.globalNamespaceCount`. +Scale Worker capacity first if schedule-to-start latency is the real problem. ::: @@ -201,7 +203,7 @@ By the time a Worker reaches steady-state polling it has already called `GetSyst Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down. **INTERNAL** means the Service encountered an error it could not attribute to the request. -Short bursts during Service restarts and rolling deploys are expected — set the `for` duration long enough that your own deploy process does not page you. +Short bursts during Service restarts and rolling deploys are normal, so set the `for` duration long enough that your own deploys don't page you. The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency. @@ -214,9 +216,9 @@ Workers receiving INTERNAL on poll operations back off and poll less frequently, :::note Self-hosted Temporal Service -Check service panics first — any panic is critical and is almost always the root cause of sustained INTERNAL errors. +Check service panics first. Any panic is critical, and it is almost always the root cause of sustained INTERNAL errors. Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. -For UNIMPLEMENTED, verify every Frontend, History, and Matching pod is running the intended binary — a wrong or corrupted binary on a subset of pods produces UNIMPLEMENTED on valid operations, usually alongside panics. +For UNIMPLEMENTED, check that every Frontend, History, and Matching pod is running the binary you intended. A wrong or corrupted binary on a few pods returns UNIMPLEMENTED on perfectly valid operations, usually alongside panics. ::: @@ -240,8 +242,9 @@ If throttling is the cause and retries exhaust the 60-second budget, the call fa :::note Self-hosted Temporal Service -Check Frontend Service latency filtered to the affected operations — server-side latency is the more precise signal, since the SDK metric includes serialization and network time. -Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, the usual driver of elevated Frontend latency on starts and Signals. -If persistence is healthy and there is no throttling, check Frontend pod CPU. +Check Frontend Service latency filtered to the affected operations. +Server-side latency is the sharper signal here, because the SDK metric also includes serialization and network time. +Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, which is what usually drives up Frontend latency on starts and Signals. +If persistence is healthy and nothing is being throttled, check Frontend pod CPU. ::: diff --git a/docs/troubleshooting/sdk-worker-capacity.mdx b/docs/troubleshooting/sdk-worker-capacity.mdx index 15f73567c6..c3ea7814a6 100644 --- a/docs/troubleshooting/sdk-worker-capacity.mdx +++ b/docs/troubleshooting/sdk-worker-capacity.mdx @@ -18,15 +18,15 @@ tags: - Workers --- -This guide covers the failure modes where Workers stop keeping up with their Task Queues: slots that never free up, pollers that stop polling, Tasks that sit in the queue, and a [Sticky Execution](/sticky-execution) cache that is not doing its job. +This guide covers what to do when your Workers stop keeping up with their Task Queues: slots that never free up, pollers that quit polling, Tasks sitting in the queue, and a [Sticky Execution](/sticky-execution) cache that isn't earning its keep. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For recommended alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). -These failure modes are causally linked. -Slots fill up, which stops polling, which lets Tasks accumulate, which stops completions. -When several fire at once, work backward to the earliest link in the chain rather than treating each as independent. +These failure modes are chained together. +Slots fill up, polling stops, Tasks pile up, completions go to zero. +When several fire at once, don't treat them as separate problems. Work backward to the earliest link in the chain. ## Worker Task slots exhausted {/* #worker-task-slots-exhausted */} @@ -38,24 +38,24 @@ The SDK blocks before issuing its next poll until a slot is released. :::note This metric reports meaningful values only with fixed-size slot suppliers. -It cannot be used with resource-based slot suppliers — see [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). -If your Workers use resource-based tuning, skip this section and use [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) as your signal that Worker capacity is short. +It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this section and let [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) tell you when capacity is short. ::: **Why it matters.** -Slots stay occupied because existing Tasks are not completing. -Until they free up, schedule-to-start latency rises and, in both the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) drops to zero as a secondary effect, because those SDKs block on slot acquisition before incrementing the poller gauge. +Slots stay occupied because the Tasks already holding them aren't finishing. +Until they do, schedule-to-start latency climbs. In the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) also drops to zero, because both block on getting a slot before they bump the poller gauge. -Impact and remediation differ by `worker_type`, so check that tag on the firing alert. -Values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`. +What this costs you and how you fix it both depend on `worker_type`, so check that tag on the firing alert. +The values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`. ### WorkflowWorker slots {/* #workflowworker-slots */} 1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). 1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer. 1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — slots are not released until the respond call succeeds. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). Slots aren't released until the respond call succeeds. To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options. @@ -66,17 +66,17 @@ To recover immediately, scale out Workflow Worker pods or raise the maximum conc 1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency. 1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). -To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size — but only after confirming a downstream dependency is not the constraint. +To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size. Confirm a downstream dependency isn't the real constraint first, or you will just push harder on something that is already struggling. ### LocalActivityWorker slots {/* #localactivityworker-slots */} -[Local Activities](/local-activity) run inside the Workflow Task execution loop, so blocked slots hold up the entire Workflow Task. -The SDK responds by sending repeated Workflow Task heartbeats to keep the Task alive. -If heartbeating continues past the Workflow Task heartbeat timeout — 30 minutes by default — the Temporal Service times out the Task and reschedules it, and the Local Activities re-execute from scratch. +[Local Activities](/local-activity) run inside the Workflow Task execution loop, so a blocked slot holds up the whole Workflow Task. +The SDK keeps the Task alive by sending repeated Workflow Task heartbeats. +If that goes on past the Workflow Task heartbeat timeout, 30 minutes by default, the Temporal Service times the Task out and reschedules it, and every Local Activity in it runs again from the start. 1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling. 1. **Check Worker CPU.** -1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) — if that is firing alongside this, the heartbeat timeout is imminent or already exceeded and re-execution has begun. +1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout). If that is firing too, you are close to the heartbeat timeout or already past it, and re-execution has started. ## All pollers disconnected {/* #all-pollers-disconnected */} @@ -100,17 +100,18 @@ Depending on your Workflow and Activity timeouts, Executions begin timing out wh **Triage.** -1. **Check whether Worker processes are running.** Check pod status, restart counts, and logs. Workers may have crashed, been evicted, or been OOM killed — this is the most common cause. +1. **Check whether Worker processes are running.** Look at pod status, restart counts, and logs. A crash, an eviction, or an OOM kill is the most common cause by a wide margin. 1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further. 1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. 1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry. -1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) — sustained INTERNAL errors cause Workers to back off and stop polling. +1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service). Sustained INTERNAL errors make Workers back off until they stop polling altogether. :::note Self-hosted Temporal Service Cross-check the server-side concurrent poller count for this Task Queue. If it has also dropped, Workers have fully disconnected from the Service's perspective. -For authentication failures, check unauthorized request counts and authorization system failures — a non-zero authorization system failure count means the auth plugin itself is failing, which is more urgent than an expired credential. +For authentication failures, check unauthorized request counts and authorization system failures. +A non-zero authorization system failure count means the auth plugin itself is broken, which is a bigger problem than an expired credential. ::: @@ -118,29 +119,29 @@ For authentication failures, check unauthorized request counts and authorization **Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted` -The SDK increments this counter only after a successful response, so a zero rate means no responses are landing. +The SDK only bumps this counter after a response succeeds, so a zero rate means nothing is landing. -**A zero rate on its own does not prove a problem.** -It is equally consistent with an idle Task Queue, a Namespace outside business hours, or a batch workload between runs. -This metric is tagged `namespace` and `operation`, not `task_queue`, so you cannot narrow it to a single Task Queue without correlating against another signal. +**Check for demand before you act on this one.** +A zero rate looks the same whether your Workers are broken or the Task Queue is simply quiet: outside business hours, or a batch workload between runs, zero is the correct answer. +The metric is tagged `namespace` and `operation`, not `task_queue`, so you can't narrow it to one Task Queue on its own either. -Treat it as actionable only alongside evidence of demand: a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency that keeps climbing. -Without one of those, there is nothing for Workers to complete and the metric is reporting the truth. +Look for a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency climbing. +If none of those are there, nothing is waiting to be completed and the metric is telling you the truth. **Why it matters.** -When there is demand, no Workflow Task completions means Workflow Executions are not making progress. +When there is demand and Workflow Task completions are still zero, your Workflow Executions aren't moving. Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them. No Activity Task completions means Workflows waiting on Activity results are stalled. -Activities begin timing out against their `scheduleToClose` timeouts, and the Temporal Service retries them within the Retry Policy — but with no Workers completing them, the retries accumulate too. +Activities start timing out against their `scheduleToClose` timeouts and the Temporal Service retries them within the Retry Policy, but with no Workers completing anything the retries just pile up alongside the originals. **Triage.** 1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed. -1. **Check whether polling also stopped.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are also at zero, Workers are down entirely — go to [All pollers disconnected](#all-pollers-disconnected). +1. **Check whether polling also stopped.** Look at [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are at zero too, your Workers are down entirely. Go to [All pollers disconnected](#all-pollers-disconnected). 1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). 1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) — sustained throttling suppresses this counter directly, because the SDK only counts successful responses. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). Sustained throttling suppresses this counter directly, since the SDK only counts responses that succeeded. 1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers. 1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers. @@ -158,7 +159,7 @@ Workflow Tasks are sitting in the queue before a Worker picks them up. Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. Two other pages give thresholds for this metric, for different purposes. -[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth investigating — use that when you are tuning Worker capacity, not paging on it. +[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth a look. Use that number when you are tuning Worker capacity, not when you are deciding whether to page someone. [Worker health](/cloud/worker-health#schedule-to-start-latency) supplies ready-made Prometheus queries for both this metric and the Activity equivalent. The thresholds on this page are set higher because they are meant to page someone. @@ -174,14 +175,15 @@ A backlog large enough can affect an entire self-hosted cluster, not just the af 1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else. 1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). -1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) — with every slot occupied, the SDK blocks before its next poll and the effective poll rate falls. -1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations) — throttled Workers back off and poll less frequently. -1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, the Worker pool is undersized for current throughput. Scale horizontally. Expect a delay before latency recovers — the accumulated backlog has to drain first. +1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). With every slot occupied the SDK blocks before its next poll, so the effective poll rate drops. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations). Throttled Workers back off and poll less often. +1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, your Worker pool is too small for current throughput. Scale horizontally, and expect latency to stay high for a while afterward. The backlog you already built up has to drain first. :::note Self-hosted Temporal Service -Check the approximate Task backlog in the Matching Service section of your server dashboard — a growing backlog confirms Tasks are accumulating faster than Workers pick them up. -Check the server-side concurrent poller count for this Task Queue to confirm whether your Worker pool has shrunk. +Check the approximate Task backlog in the Matching Service section of your server dashboard. +A growing backlog confirms Tasks are arriving faster than Workers pick them up. +Then check the server-side concurrent poller count for this Task Queue to see whether your Worker pool has shrunk. ::: @@ -203,7 +205,7 @@ At scale, an Activity Task backlog accumulates and puts pressure on Task matchin 1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected). 1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted). 1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations). -1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) — high failure churn generates retry Tasks that grow the backlog and keep this latency elevated even after Worker capacity is restored. +1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated). Lots of failures means lots of retry Tasks, which grow the backlog and hold this latency up even after you have restored Worker capacity. 1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks). :::note Self-hosted Temporal Service @@ -216,36 +218,37 @@ Check the approximate Task backlog and the server-side concurrent poller count f **Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 while the Worker is processing Workflow Tasks -This gauge reports how many Workflow Executions the Worker is **currently** holding in its [Sticky Execution](/sticky-execution) cache, not the configured cache capacity. +This gauge tells you how many Workflow Executions the Worker is holding in its [Sticky Execution](/sticky-execution) cache right now. +It is not the configured cache size. -**A zero value is normal on an idle Worker.** -A Worker that is not processing Workflow Tasks has nothing to cache, and a low-volume Task Queue may sit at zero for long stretches. -Alerting on this gauge alone will page you on idle Workers. +**Zero is normal on an idle Worker.** +A Worker that isn't processing Workflow Tasks has nothing to cache, and a low-volume Task Queue can sit at zero for hours. +Alert on this gauge by itself and you will get paged for Workers that are doing nothing wrong. -The condition worth alerting on is zero occupancy *while the Worker is doing work*. -Pair the gauge with a non-zero Workflow Task rate on the same Worker, and use a long `for` duration. -Corroborate with elevated [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate, either of which confirms Workflow Tasks are replaying from history rather than resuming from cache. +What you actually want to catch is zero occupancy while the Worker is working. +Pair the gauge with a non-zero Workflow Task rate on the same Worker and give it a long `for` duration. +High [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate will confirm it: both mean Workflow Tasks are replaying from history instead of picking up where they left off. -Tag coverage is uneven, which limits how precisely you can scope this alert: `namespace` is emitted only by the TypeScript and Java SDKs, and `task_queue` only by TypeScript. +Tag coverage is patchy, so there is a limit to how tightly you can scope this alert. Only the TypeScript and Java SDKs emit `namespace`, and only TypeScript emits `task_queue`. **Why it matters.** -When the cache holds nothing while Workflow Tasks are running, every Workflow Task fetches all History pages from the Temporal Service and re-executes every command from scratch. -Every Workflow Task becomes the equivalent of a cache miss. +With nothing in the cache, every Workflow Task pulls all the History pages from the Temporal Service and re-runs every command from the beginning. +Each one costs you what a cache miss costs. -At any meaningful scale this causes sustained pressure on the Temporal Service and elevated Workflow Task execution latency for every Execution on the Worker. +At any real scale that means sustained load on the Temporal Service and higher Workflow Task execution latency for every Execution on the Worker. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). **Triage.** 1. **Confirm the Worker is actually processing Workflow Tasks.** If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix. 1. **Check whether the cache is disabled by configuration.** A cache size of zero in Worker options means no Execution is ever cached: - - **Go SDK** — `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` disables the cache entirely. Restore a non-zero value. This must be called before any Worker starts; the cache is shared across all Workers in the process. - - **Java SDK** — `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets to the default rather than disabling the cache, and a negative value is rejected, so the Java SDK cannot reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead — a thread pool set too low starves Workflow execution and prevents the cache from being used effectively. -1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they are added. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction) — a high rate points at a cache too small for the number of concurrent Executions on the Worker, or at repeated Workflow Task timeouts forcing evictions. + - **Go SDK:** `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` turns the cache off completely, so restore a non-zero value. Call it before any Worker starts; the cache is shared across every Worker in the process. + - **Java SDK:** `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets it to the default instead of disabling it, and a negative value is rejected outright, so you can't reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead. A thread pool set too low starves Workflow execution and the cache never gets used properly. +1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they go in. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction). A high rate points at either a cache too small for the number of concurrent Executions on that Worker, or repeated Workflow Task timeouts forcing evictions. 1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero under load, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits. :::note Self-hosted Temporal Service -Check persistence latency filtered to `GetWorkflowExecution` — sustained high latency there confirms the Temporal Service is under pressure from repeated History reads. +Check persistence latency filtered to `GetWorkflowExecution`. Sustained high latency there confirms the Temporal Service is straining under all the repeated History reads. ::: From 8fb943be8d8e256b93c1fde9d5f203d84043fad3 Mon Sep 17 00:00:00 2001 From: Auston Li Date: Tue, 4 Aug 2026 19:05:01 -0700 Subject: [PATCH 5/5] Drop the redundant SDK prefix and restructure the alerting page Naming. The pages sat in Troubleshooting and Best practices with "SDK" in every title and sidebar label, which repeated what the section already says. The repo's own pattern is a descriptive title plus a short bare sidebar label: "Troubleshoot payload and gRPC message size limit errors" with "Message size limit errors", "Performance bottlenecks troubleshooting guide" with "Performance bottlenecks". Renamed to match, including file names, ids, and slugs, since none of these pages are published yet and the URLs are free to change now: Alerting on Worker metrics Worker alerting Troubleshoot request failures Request failures Troubleshoot Worker capacity Worker capacity Troubleshoot Workflow and Activity execution failures Execution failures Moved the setup section to the top, retitled "Before you start". It is a prerequisite: without metrics flowing there is nothing to alert on, and the metric-naming rules it carries (counter and histogram suffixes, and gauges taking none) are needed to write the queries in the tables further down. There was no reason for it to be second to last. Replaced "Alerts that fire during normal operation", which documented noise instead of fixing it. Split into the two cases: - Conditions that appear during a deploy and clear on their own once the rollout finishes. The fix is a `for` duration longer than a deploy takes, so the section now says to time your rollout and set it from that. - Conditions no `for` duration makes correct, because the alert is asking the wrong question. An idle Task Queue really does have zero Task completions and an empty sticky cache; an autoscaled-to-zero Worker really does have no pollers. Those need a demand signal or an exclusion, and the section points at the two troubleshooting sections that explain how. Restored list structure to "Route alerts by severity". The voice pass had flattened it into prose, which lost the scannability of what is really a routing table. Co-Authored-By: Claude Opus 5 --- docs/best-practices/sdk-worker-alerting.mdx | 167 ------------------ docs/best-practices/worker-alerting.mdx | 165 +++++++++++++++++ .../troubleshooting/blob-size-limit-error.mdx | 2 +- ...on-failures.mdx => execution-failures.mdx} | 22 +-- docs/troubleshooting/index.mdx | 6 +- ...uest-failures.mdx => request-failures.mdx} | 18 +- ...orker-capacity.mdx => worker-capacity.mdx} | 30 ++-- sidebars.js | 8 +- 8 files changed, 208 insertions(+), 210 deletions(-) delete mode 100644 docs/best-practices/sdk-worker-alerting.mdx create mode 100644 docs/best-practices/worker-alerting.mdx rename docs/troubleshooting/{sdk-execution-failures.mdx => execution-failures.mdx} (92%) rename docs/troubleshooting/{sdk-request-failures.mdx => request-failures.mdx} (95%) rename docs/troubleshooting/{sdk-worker-capacity.mdx => worker-capacity.mdx} (90%) diff --git a/docs/best-practices/sdk-worker-alerting.mdx b/docs/best-practices/sdk-worker-alerting.mdx deleted file mode 100644 index 4bed58f579..0000000000 --- a/docs/best-practices/sdk-worker-alerting.mdx +++ /dev/null @@ -1,167 +0,0 @@ ---- -id: sdk-worker-alerting -title: Alerting on Temporal SDK Worker metrics -sidebar_label: SDK Worker alerting -description: A recommended alert set for Temporal SDK Workers, with label filters, thresholds, and links to triage guidance -toc_max_heading_level: 4 -keywords: - - sdk metrics - - worker metrics - - alerting - - monitoring temporal metrics - - observability -tags: - - Metrics - - Observability - - Workers ---- - -Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window. -Nobody catches those for you. - -This page recommends a set of alerts that covers them. -It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. - -For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). -For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). - -## Start with these five {/* #start-with-these-five */} - -If you are setting up Worker alerts for the first time, start here. -These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing. - -1. [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected). Your Workers have stopped polling. -1. [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error). Workflow code no longer matches recorded history. -1. [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). Executions are being terminated and losing work. -1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up. -1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled. - -Add the rest once these are tuned and quiet. - -## Choose your thresholds {/* #choose-your-thresholds */} - -Every threshold on this page is a starting point, not a service level objective. -A high-throughput Task Queue needs different values than a bursty batch workload. - -There are two knobs per alert: - -- **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here. -- **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection. - -Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero. There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page. -All of those conditions show up briefly during normal operation, so set it generously. - -### Alerts that fire during normal operation {/* #expected-firing */} - -Several of these conditions show up in a perfectly healthy deployment. -Set the `for` duration long enough to ride them out, and check your value against how long your own deploys take: - -| Event | Alerts it can trigger | -| --- | --- | -| Worker deploy or rolling restart | NOT_FOUND on respond operations, all pollers disconnected, Task completions dropped to zero, non-determinism error | -| Temporal Service upgrade | INTERNAL from the Temporal Service | -| Normal Workflow completion or termination while a Task is in flight | NOT_FOUND on respond operations | -| Idle or low-volume Task Queue | Task completions dropped to zero, sticky cache holding zero entries | -| Scale-down or Worker autoscaling | All pollers disconnected, Task slots exhausted | - -## Recommended alert set {/* #recommended-alert-set */} - -The condition column gives the tag filters that tell each alert apart. -Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from. - -### Request failures {/* #request-failures */} - -These fire on gRPC responses coming back from the Temporal Service to your Worker or Client. -For triage, see [SDK request failures](/troubleshooting/sdk-request-failures). - -| Failure mode | Metric | Condition | Threshold | `for` | Default severity | -| --- | --- | --- | --- | --- | --- | -| [NOT_FOUND on respond operations](/troubleshooting/sdk-request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical | -| [NOT_FOUND on Activity heartbeat](/troubleshooting/sdk-request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning | -| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical | -| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical | -| [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning | -| [UNIMPLEMENTED from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical | -| [INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical | -| [Request latency high on user-facing operations](/troubleshooting/sdk-request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical | - -### Worker capacity {/* #worker-capacity */} - -These fire when your Workers stop keeping up with the Task Queue. -For triage, see [SDK Worker capacity](/troubleshooting/sdk-worker-capacity). - -| Failure mode | Metric | Condition | Threshold | `for` | Default severity | -| --- | --- | --- | --- | --- | --- | -| [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | -| [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | -| [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | -| [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical | -| [Workflow Task schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | -| [Activity schedule-to-start latency severe](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | -| [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | - -:::note - -`worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. -It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). -If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short. - -::: - -### Execution failures {/* #execution-failures */} - -These fire when your Workflow or Activity code fails on the Worker. -For triage, see [SDK execution failures](/troubleshooting/sdk-execution-failures). - -| Failure mode | Metric | Condition | Threshold | `for` | Default severity | -| --- | --- | --- | --- | --- | --- | -| [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | -| [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | -| [Workflow Task execution failures elevated](/troubleshooting/sdk-execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | -| [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | -| [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | -| [Unregistered Activity invocation](/troubleshooting/sdk-execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | -| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | - -## Set up the alerts {/* #set-up-the-alerts */} - -Before you can alert on these metrics, your Workers must be emitting them and your monitoring system must be scraping them: - -- **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). -- **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). - -### Resolve the metric names for your setup {/* #resolve-metric-names */} - -The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). -The names you query depend on your SDK and metrics reporter: - -- Every metric carries a `temporal_` prefix. -- **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`. -- **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. -- **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why. - -Tag coverage varies too. -`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely. -Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript. - -Confirm the exact names and tags in your own metrics endpoint before writing queries. - -## Route alerts by severity {/* #route-alerts-by-severity */} - -The severity in each table is a suggested routing default, not a property of the metric. -Tune it to your workload. - -Start by paging on the alerts that mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects. -Send the rest to a channel someone reads during the day. - -The defaults won't fit every deployment, for two reasons. - -Some Critical rows are latency thresholds or early warnings rather than confirmed stoppage. -High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. - -Some Warning rows can still lose you data. -A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start, which is a real problem if that Activity isn't idempotent. - -These alerts are also chained together. -Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. -When several fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/best-practices/worker-alerting.mdx b/docs/best-practices/worker-alerting.mdx new file mode 100644 index 0000000000..ee282854fe --- /dev/null +++ b/docs/best-practices/worker-alerting.mdx @@ -0,0 +1,165 @@ +--- +id: worker-alerting +title: Alerting on Worker metrics +sidebar_label: Worker alerting +description: A recommended alert set for Temporal Workers, with tag filters, thresholds, and links to triage guidance +toc_max_heading_level: 4 +keywords: + - sdk metrics + - worker metrics + - alerting + - monitoring temporal metrics + - observability +tags: + - Metrics + - Observability + - Workers +--- + +Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window. +Nobody catches those for you. + +This page recommends a set of alerts that covers them. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). +For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). + +## Before you start {/* #set-up-the-alerts */} + +None of this works until your Workers are emitting metrics and something is scraping them: + +- **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). +- **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). + +### Resolve the metric names for your setup {/* #resolve-metric-names */} + +The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). +The names you query depend on your SDK and metrics reporter: + +- Every metric carries a `temporal_` prefix. +- **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`. +- **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. +- **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why. + +Tag coverage varies too. +`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely. +Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript. + +Confirm the exact names and tags in your own metrics endpoint before writing queries. + +## Start with these five {/* #start-with-these-five */} + +If you are setting up Worker alerts for the first time, start here. +These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing. + +1. [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected). Your Workers have stopped polling. +1. [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error). Workflow code no longer matches recorded history. +1. [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). Executions are being terminated and losing work. +1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up. +1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled. + +Add the rest once these are tuned and quiet. + +## Choose your thresholds {/* #choose-your-thresholds */} + +Every threshold on this page is a starting point, not a service level objective. +A high-throughput Task Queue needs different values than a bursty batch workload. + +There are two knobs per alert: + +- **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here. +- **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection. + +Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero. +There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page. + +### Tune the `for` duration to your deploys {/* #expected-firing */} + +Several of these conditions appear briefly whenever you deploy. +A rolling restart drops pollers to zero on each pod as it cycles, produces NOT_FOUND on respond operations for Tasks that were in flight, and can produce short-lived non-determinism errors while two Worker versions overlap. +A Temporal Service upgrade does the same for INTERNAL. + +All of those clear on their own once the rollout finishes, so the fix is a `for` duration longer than a deploy takes, not a quieter threshold. +Time your own rollout and set it from that. + +Two conditions need a different fix, because no `for` duration makes them correct: + +- **An idle Task Queue** legitimately reports zero Task completions and an empty sticky cache. Pair both alerts with a demand signal instead, as [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) and [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) describe. +- **Autoscaling to zero** legitimately reports no pollers and no free slots. Exclude scaled-down Workers from those alerts, or scope them to Task Queues you keep warm. + +## Recommended alert set {/* #recommended-alert-set */} + +The condition column gives the tag filters that tell each alert apart. +Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from. + +### Request failures {/* #request-failures */} + +These fire on gRPC responses coming back from the Temporal Service to your Worker or Client. +For triage, see [Request failures](/troubleshooting/request-failures). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [NOT_FOUND on respond operations](/troubleshooting/request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical | +| [NOT_FOUND on Activity heartbeat](/troubleshooting/request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning | +| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical | +| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical | +| [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning | +| [UNIMPLEMENTED from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical | +| [INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical | +| [Request latency high on user-facing operations](/troubleshooting/request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical | + +### Worker capacity {/* #worker-capacity */} + +These fire when your Workers stop keeping up with the Task Queue. +For triage, see [Worker capacity](/troubleshooting/worker-capacity). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | +| [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | +| [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | +| [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical | +| [Workflow Task schedule-to-start latency severe](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Activity schedule-to-start latency severe](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | + +:::note + +`worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. +It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short. + +::: + +### Execution failures {/* #execution-failures */} + +These fire when your Workflow or Activity code fails on the Worker. +For triage, see [Execution failures](/troubleshooting/execution-failures). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | +| [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | +| [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | +| [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | +| [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | +| [Unregistered Activity invocation](/troubleshooting/execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | +| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | + +## Route alerts by severity {/* #route-alerts-by-severity */} + +The severity in each table is a suggested routing default, not a property of the metric. + +- **Page on Critical.** These mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects. +- **Send Warning to a channel someone reads during the day.** These mean something is degrading but Executions are still moving. + +Then adjust, because the defaults won't fit every deployment: + +- **Some Critical rows are early warnings, not confirmed stoppage.** High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. Downgrade the latency alerts if that describes you. +- **Some Warning rows can still lose you data.** A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start. If that Activity isn't idempotent, treat it as Critical. + +One last thing worth knowing before you wire up routing: these alerts are chained together. +Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. +Expect them to fire in clusters, and check the triage pages to find which one is the cause rather than responding to all four. +When several fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/troubleshooting/blob-size-limit-error.mdx b/docs/troubleshooting/blob-size-limit-error.mdx index 62c375f2b9..c42850b14f 100644 --- a/docs/troubleshooting/blob-size-limit-error.mdx +++ b/docs/troubleshooting/blob-size-limit-error.mdx @@ -109,7 +109,7 @@ The behavior depends on where the oversized message originates: with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Because replay would produce the same oversized request on every attempt, the Temporal Service ends the Execution with `TERMINATED` status rather than retrying it. Terminated Executions do not retry automatically, so fix the oversized payload and deploy the corrected Worker before restarting - them. See [SDK execution failures](/troubleshooting/sdk-execution-failures#grpc-message-too-large) for triage steps. + them. See [Execution failures](/troubleshooting/execution-failures#grpc-message-too-large) for triage steps. - **Activity Tasks:** The Activity gets stuck in a retry loop or exits with a `ScheduleToCloseTimeout`. The Activity executes successfully, but the Worker can't deliver the oversized result over gRPC. The server never receives the diff --git a/docs/troubleshooting/sdk-execution-failures.mdx b/docs/troubleshooting/execution-failures.mdx similarity index 92% rename from docs/troubleshooting/sdk-execution-failures.mdx rename to docs/troubleshooting/execution-failures.mdx index 3e4d9210a9..a1ae90862e 100644 --- a/docs/troubleshooting/sdk-execution-failures.mdx +++ b/docs/troubleshooting/execution-failures.mdx @@ -1,7 +1,7 @@ --- -id: sdk-execution-failures -title: Troubleshoot SDK execution failures -sidebar_label: SDK execution failures +id: execution-failures +title: Troubleshoot Workflow and Activity execution failures +sidebar_label: Execution failures description: Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency toc_max_heading_level: 3 keywords: @@ -19,7 +19,7 @@ tags: This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`. @@ -47,7 +47,7 @@ This will not clear up on its own. - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes. - Changed Activity or Timer parameters in existing Workflow code without versioning. 1. **Roll back if it isn't clearing.** If the errors started after a deploy and aren't going away, roll the Worker back. Affected Executions pick up again on their next Workflow Task retry once compatible code is running. Then add a proper versioning guard before you redeploy. See [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). -1. **Watch Worker pressure.** All those retries add up. Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high). Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue. +1. **Watch Worker pressure.** All those retries add up. Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high). Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue. ## gRPC message too large {/* #grpc-message-too-large */} @@ -103,7 +103,7 @@ Unlike `GrpcMessageTooLarge`, the Temporal Service won't terminate the Execution - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool. `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` is too low for the number of concurrent Executions, so new Workflow Tasks get rejected before they run. Raise the thread count, and think about whether the Worker pool needs to scale out too. - **Unhandled exception in Workflow code.** A bug or an unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event names the error. - **Data Converter error.** Something failed serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration. -1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse. +1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse. 1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running. ## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */} @@ -128,10 +128,10 @@ At scale this compounds: more timeouts cause more cold replays, cold replays dri **Triage.** 1. **Check replay latency.** Look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-running Event History rather than into new commands. Large histories, a slow Data Converter during replay, or a high cache eviction rate forcing cold replays are the usual culprits. -1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/sdk-worker-capacity#sticky-cache-holding-zero-entries-under-load). +1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load). 1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker. 1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly. ## Activity execution failures elevated {/* #activity-execution-failures-elevated */} @@ -143,7 +143,7 @@ Activities are failing outright at a sustained rate: returning failures, not tim **Why it matters.** A high failure rate means a burst of retry Tasks. -If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated). +If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated). At scale, sustained retry bursts put real pressure on Task matching and the database underneath. **Triage.** @@ -151,7 +151,7 @@ At scale, sustained retry bursts put real pressure on Task matching and the data 1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. 1. **Work out whether this is transient or a bug.** A downstream outage, a network partition, or a database timeout will recover on its own, so watch whether the rate falls. A code bug won't. 1. **Check downstream service health.** A struggling dependency is a common cause of sustained failure bursts. If it is throttling you, check that your [Retry Policy](/encyclopedia/retry-policies) has sensible backoff. Without it, your retries pile more pressure onto something that is already overloaded. -1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. +1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. 1. **Mark expected failures as benign.** If your design fails Activities on purpose, as polling patterns, Saga compensations, and flow control through exceptions all do, mark those `ApplicationFailure` instances with category `BENIGN`. That keeps them out of this metric and lets the alert track only the failures you didn't expect, without tuning a threshold per `activity_type`. Check that your SDK version supports the category before you rely on it. One caveat: internal failures increment this counter no matter what category you set. Those are things like a context propagation error or a context timeout, rather than an Activity returning a failure. @@ -202,7 +202,7 @@ Any pending Signals, Updates, or other events are delayed until the retried Work The Local Activity also occupies an executor slot for its entire duration. Several in this state at once can occupy every available slot, blocking new Local Activities from starting. -See [LocalActivityWorker slots](/troubleshooting/sdk-worker-capacity#localactivityworker-slots). +See [LocalActivityWorker slots](/troubleshooting/worker-capacity#localactivityworker-slots). Local Activities are designed for short, fast operations. A single attempt running for 30 minutes is a design problem, not a tuning problem. diff --git a/docs/troubleshooting/index.mdx b/docs/troubleshooting/index.mdx index 0308c5f68d..291c61ef56 100644 --- a/docs/troubleshooting/index.mdx +++ b/docs/troubleshooting/index.mdx @@ -24,7 +24,7 @@ Our troubleshooting guides are designed to help you quickly identify and resolve - [Troubleshoot the Failed Reaching Server Error](/troubleshooting/last-connection-error): The message "Failed reaching server: last connection error" often happens due to an expired TLS certificate or during the Server startup process when Client requests reach the Server before roles are fully initialized. - [Troubleshoot missed Schedule Actions](/troubleshooting/schedule-missed-actions): When a Schedule does not fire at its expected time, alert on the missed catchup window metric, then narrow down to the affected Schedule with `ListSchedules` and `DescribeSchedule`. - [Troubleshoot Serverless Workers](/troubleshooting/serverless-workers): Diagnose issues with Serverless Workers by tracing the flow from Task Queue to Worker execution, on [AWS Lambda](/troubleshooting/serverless-workers/aws-lambda) or [GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run). -- [Troubleshoot SDK request failures](/troubleshooting/sdk-request-failures): Diagnose gRPC failures and elevated latency on requests from Workers and Clients to the Temporal Service, including NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses. -- [Troubleshoot SDK Worker capacity](/troubleshooting/sdk-worker-capacity): Diagnose exhausted Task slots, disconnected pollers, Task completions dropping to zero, elevated schedule-to-start latency, and a disabled Sticky Execution cache. -- [Troubleshoot SDK execution failures](/troubleshooting/sdk-execution-failures): Diagnose non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout. +- [Troubleshoot request failures](/troubleshooting/request-failures): Diagnose gRPC failures and slow requests from Workers and Clients to the Temporal Service, including NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses. +- [Troubleshoot Worker capacity](/troubleshooting/worker-capacity): Diagnose exhausted Task slots, disconnected pollers, Task completions dropping to zero, rising schedule-to-start latency, and an empty Sticky Execution cache. +- [Troubleshoot Workflow and Activity execution failures](/troubleshooting/execution-failures): Diagnose non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout. - [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows): Recover pinned Workflows that have failed or are stuck retrying tasks after rolling out a faulty Worker Deployment Version. diff --git a/docs/troubleshooting/sdk-request-failures.mdx b/docs/troubleshooting/request-failures.mdx similarity index 95% rename from docs/troubleshooting/sdk-request-failures.mdx rename to docs/troubleshooting/request-failures.mdx index be284a5508..95234c50ab 100644 --- a/docs/troubleshooting/sdk-request-failures.mdx +++ b/docs/troubleshooting/request-failures.mdx @@ -1,8 +1,8 @@ --- -id: sdk-request-failures -title: Troubleshoot SDK request failures -sidebar_label: SDK request failures -description: Diagnose gRPC failures and elevated latency on requests from Temporal SDK Workers and Clients +id: request-failures +title: Troubleshoot request failures +sidebar_label: Request failures +description: Diagnose gRPC failures and slow requests from Temporal Workers and Clients toc_max_heading_level: 3 keywords: - sdk metrics @@ -19,7 +19,7 @@ tags: This guide covers gRPC failures and slow requests between your Temporal SDK Workers or Clients and the Temporal Service. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). [`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. @@ -149,13 +149,13 @@ This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-r If throttling goes on long enough the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task gets rescheduled, with all the cache eviction, cold replay, and Local Activity re-execution that section describes. Meanwhile every in-flight task is still holding its slot, so you have less concurrency for new work. -Left alone, this turns into [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). +Left alone, this turns into [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted). **Triage.** 1. **Find out which limit you hit,** as in the section above. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`. 1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. -1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/sdk-worker-capacity#worker-task-slots-exhausted). Slots aren't released until the respond call succeeds, so throttling here drains the pool. +1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted). Slots aren't released until the respond call succeeds, so throttling here drains the pool. :::note Self-hosted Temporal Service @@ -181,7 +181,7 @@ The Temporal Service throttles poll operations before it throttles respond or us 1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. On the Go SDK, [`temporal_long_request_resource_exhausted`](/references/sdk-metrics#long_request_resource_exhausted) carries the `cause` tag for poll operations. 1. **Check whether you are over-polling.** A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against [Worker performance](/develop/worker-performance) guidance before assuming the limit is too low. -1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/sdk-worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet. +1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet. 1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare against your Namespace's [service limits](/cloud/limits). :::note Self-hosted Temporal Service @@ -212,7 +212,7 @@ Workers receiving INTERNAL on poll operations back off and poll less frequently, 1. **Check SDK and Temporal Service version compatibility.** For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version. 1. **Check whether recent deploys correlate.** Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate. 1. **Check whether the errors are Namespace-scoped or cluster-wide.** Errors isolated to one Namespace point at Namespace configuration. Cluster-wide errors point at infrastructure. -1. **Check downstream effects.** Sustained errors on poll operations cause Workers to back off. Cross-check [All pollers disconnected](/troubleshooting/sdk-worker-capacity#all-pollers-disconnected) and [Task completions dropped to zero](/troubleshooting/sdk-worker-capacity#task-completions-dropped-to-zero). +1. **Check downstream effects.** Sustained errors on poll operations cause Workers to back off. Cross-check [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) and [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero). :::note Self-hosted Temporal Service diff --git a/docs/troubleshooting/sdk-worker-capacity.mdx b/docs/troubleshooting/worker-capacity.mdx similarity index 90% rename from docs/troubleshooting/sdk-worker-capacity.mdx rename to docs/troubleshooting/worker-capacity.mdx index c3ea7814a6..45ed439fc2 100644 --- a/docs/troubleshooting/sdk-worker-capacity.mdx +++ b/docs/troubleshooting/worker-capacity.mdx @@ -1,7 +1,7 @@ --- -id: sdk-worker-capacity -title: Troubleshoot SDK Worker capacity problems -sidebar_label: SDK Worker capacity +id: worker-capacity +title: Troubleshoot Worker capacity +sidebar_label: Worker capacity description: Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems toc_max_heading_level: 3 keywords: @@ -21,7 +21,7 @@ tags: This guide covers what to do when your Workers stop keeping up with their Task Queues: slots that never free up, pollers that quit polling, Tasks sitting in the queue, and a [Sticky Execution](/sticky-execution) cache that isn't earning its keep. It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. -For alert thresholds and `for` durations, see [SDK Worker alerting](/best-practices/sdk-worker-alerting). +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). These failure modes are chained together. @@ -52,10 +52,10 @@ The values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`. ### WorkflowWorker slots {/* #workflowworker-slots */} -1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). +1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high). 1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer. 1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). Slots aren't released until the respond call succeeds. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Slots aren't released until the respond call succeeds. To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options. @@ -64,7 +64,7 @@ To recover immediately, scale out Workflow Worker pods or raise the maximum conc 1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected. 1. **Check Worker CPU.** High CPU slows Activity execution directly. 1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size. Confirm a downstream dependency isn't the real constraint first, or you will just push harder on something that is already struggling. @@ -76,7 +76,7 @@ If that goes on past the Workflow Task heartbeat timeout, 30 minutes by default, 1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling. 1. **Check Worker CPU.** -1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/sdk-execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout). If that is firing too, you are close to the heartbeat timeout or already past it, and re-execution has started. +1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout). If that is firing too, you are close to the heartbeat timeout or already past it, and re-execution has started. ## All pollers disconnected {/* #all-pollers-disconnected */} @@ -104,7 +104,7 @@ Depending on your Workflow and Activity timeouts, Executions begin timing out wh 1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further. 1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. 1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry. -1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/sdk-request-failures#unimplemented-or-internal-from-the-temporal-service). Sustained INTERNAL errors make Workers back off until they stop polling altogether. +1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service). Sustained INTERNAL errors make Workers back off until they stop polling altogether. :::note Self-hosted Temporal Service @@ -139,9 +139,9 @@ Activities start timing out against their `scheduleToClose` timeouts and the Tem 1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed. 1. **Check whether polling also stopped.** Look at [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are at zero too, your Workers are down entirely. Go to [All pollers disconnected](#all-pollers-disconnected). -1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/sdk-execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/sdk-execution-failures#grpc-message-too-large). +1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). 1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed. -1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-respond-operations). Sustained throttling suppresses this counter directly, since the SDK only counts responses that succeeded. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Sustained throttling suppresses this counter directly, since the SDK only counts responses that succeeded. 1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers. 1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers. @@ -176,7 +176,7 @@ A backlog large enough can affect an entire self-hosted cluster, not just the af 1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else. 1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). 1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). With every slot occupied the SDK blocks before its next poll, so the effective poll rate drops. -1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations). Throttled Workers back off and poll less often. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations). Throttled Workers back off and poll less often. 1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, your Worker pool is too small for current throughput. Scale horizontally, and expect latency to stay high for a while afterward. The backlog you already built up has to drain first. :::note Self-hosted Temporal Service @@ -204,8 +204,8 @@ At scale, an Activity Task backlog accumulates and puts pressure on Task matchin 1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting. 1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected). 1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted). -1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/sdk-request-failures#resource_exhausted-on-poll-operations). -1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/sdk-execution-failures#activity-execution-failures-elevated). Lots of failures means lots of retry Tasks, which grow the backlog and hold this latency up even after you have restored Worker capacity. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations). +1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated). Lots of failures means lots of retry Tasks, which grow the backlog and hold this latency up even after you have restored Worker capacity. 1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks). :::note Self-hosted Temporal Service @@ -236,7 +236,7 @@ With nothing in the cache, every Workflow Task pulls all the History pages from Each one costs you what a cache miss costs. At any real scale that means sustained load on the Temporal Service and higher Workflow Task execution latency for every Execution on the Worker. -See [Workflow Task execution latency high](/troubleshooting/sdk-execution-failures#workflow-task-execution-latency-high). +See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high). **Triage.** diff --git a/sidebars.js b/sidebars.js index 09f81c7ddd..a21f6eeec1 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1703,9 +1703,9 @@ module.exports = { 'troubleshooting/last-connection-error', 'troubleshooting/performance-bottlenecks', 'troubleshooting/schedule-missed-actions', - 'troubleshooting/sdk-request-failures', - 'troubleshooting/sdk-worker-capacity', - 'troubleshooting/sdk-execution-failures', + 'troubleshooting/request-failures', + 'troubleshooting/worker-capacity', + 'troubleshooting/execution-failures', { type: 'category', label: 'Serverless Workers', @@ -1731,7 +1731,7 @@ module.exports = { }, items: [ 'best-practices/worker', - 'best-practices/sdk-worker-alerting', + 'best-practices/worker-alerting', 'best-practices/pre-production-testing', 'production-deployment/multi-tenant-patterns', 'best-practices/managing-namespace',