diff --git a/bin/check-metrics-reference.js b/bin/check-metrics-reference.js index 77208c38f3..bc121c18e9 100644 --- a/bin/check-metrics-reference.js +++ b/bin/check-metrics-reference.js @@ -27,12 +27,14 @@ const SDKS = ['Core', 'Go', 'Java']; // against this, so a hyphen/underscore slip in either place is a failure. const TAGS = [ 'activity_type', + 'cause', 'failure_reason', 'namespace', 'nexus_operation', 'nexus_service', 'operation', 'poller_type', + 'status_code', 'task_queue', 'worker_type', 'workflow_type', diff --git a/bin/metrics-baseline.json b/bin/metrics-baseline.json index 55a5ab3c8d..1ed0b2b7a8 100644 --- a/bin/metrics-baseline.json +++ b/bin/metrics-baseline.json @@ -26,16 +26,6 @@ "sdks": "Go, Java", "note": "Deprecated in both SDKs. Superseded by local_activity_execution_failed." }, - { - "name": "long_request_resource_exhausted", - "sdks": "Go", - "note": "" - }, - { - "name": "request_resource_exhausted", - "sdks": "Go", - "note": "" - }, { "name": "resource_slots_cpu_pid_output", "sdks": "Core, Java", diff --git a/docs/best-practices/worker-alerting.mdx b/docs/best-practices/worker-alerting.mdx new file mode 100644 index 0000000000..ee282854fe --- /dev/null +++ b/docs/best-practices/worker-alerting.mdx @@ -0,0 +1,165 @@ +--- +id: worker-alerting +title: Alerting on Worker metrics +sidebar_label: Worker alerting +description: A recommended alert set for Temporal Workers, with tag filters, thresholds, and links to triage guidance +toc_max_heading_level: 4 +keywords: + - sdk metrics + - worker metrics + - alerting + - monitoring temporal metrics + - observability +tags: + - Metrics + - Observability + - Workers +--- + +Your Worker processes emit metrics that the Temporal Service has no view into: Workflow code failing on replay, Workers that have stopped polling, Task slots that never free up, Local Activities that run past the Workflow Task heartbeat window. +Nobody catches those for you. + +This page recommends a set of alerts that covers them. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For metric definitions and tag sets, see the [Temporal SDK metrics reference](/references/sdk-metrics). +For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). + +## Before you start {/* #set-up-the-alerts */} + +None of this works until your Workers are emitting metrics and something is scraping them: + +- **Temporal Cloud:** [Set up SDK metrics](/cloud/metrics/sdk-metrics-setup), then [configure Prometheus and Grafana](/cloud/metrics/prometheus-grafana). +- **Self-hosted:** [Monitor Temporal Platform metrics](/self-hosted-guide/monitoring). + +### Resolve the metric names for your setup {/* #resolve-metric-names */} + +The metric names in the tables above are the base names from the [SDK metrics reference](/references/sdk-metrics). +The names you query depend on your SDK and metrics reporter: + +- Every metric carries a `temporal_` prefix. +- **Counters** pick up a `_total` suffix when scraped through Prometheus, so `request_failure` becomes `temporal_request_failure_total`. +- **Histograms** pick up `_seconds_bucket` on the bucket series, so you query `temporal_workflow_task_schedule_to_start_latency_seconds_bucket`. +- **Gauges take no suffix at all.** `temporal_num_pollers`, `temporal_worker_task_slots_available`, and `temporal_sticky_cache_size` are gauges. Add `_total` to any of them and you get back nothing, with no hint as to why. + +Tag coverage varies too. +`status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names (`NOT_FOUND`, `RESOURCE_EXHAUSTED`), and Client options can turn the tag off entirely. +Not every SDK emits every tag. `sticky_cache_size`, for example, carries `namespace` only in the TypeScript and Java SDKs, and `task_queue` only in TypeScript. + +Confirm the exact names and tags in your own metrics endpoint before writing queries. + +## Start with these five {/* #start-with-these-five */} + +If you are setting up Worker alerts for the first time, start here. +These five catch the failure modes that stop Workflow Executions outright, and they are the least likely to wake you up for nothing. + +1. [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected). Your Workers have stopped polling. +1. [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error). Workflow code no longer matches recorded history. +1. [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). Executions are being terminated and losing work. +1. [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated). Tasks are backing up. +1. [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations). Starts, Signals, and Updates are being throttled. + +Add the rest once these are tuned and quiet. + +## Choose your thresholds {/* #choose-your-thresholds */} + +Every threshold on this page is a starting point, not a service level objective. +A high-throughput Task Queue needs different values than a bursty batch workload. + +There are two knobs per alert: + +- **The threshold** sets what counts as unhealthy. Pick it from your own observed p99 during a period you know was healthy, not from the value listed here. +- **The `for` duration** sets how long the condition has to hold before the alert fires. Short durations catch problems faster but fire on transient spikes. Long durations stay quiet but delay detection. + +Some of these alerts fire on a binary condition: any occurrence of a gRPC status code, or a gauge hitting zero. +There is no threshold to tune on those, so the `for` duration is the only thing standing between you and a page. + +### Tune the `for` duration to your deploys {/* #expected-firing */} + +Several of these conditions appear briefly whenever you deploy. +A rolling restart drops pollers to zero on each pod as it cycles, produces NOT_FOUND on respond operations for Tasks that were in flight, and can produce short-lived non-determinism errors while two Worker versions overlap. +A Temporal Service upgrade does the same for INTERNAL. + +All of those clear on their own once the rollout finishes, so the fix is a `for` duration longer than a deploy takes, not a quieter threshold. +Time your own rollout and set it from that. + +Two conditions need a different fix, because no `for` duration makes them correct: + +- **An idle Task Queue** legitimately reports zero Task completions and an empty sticky cache. Pair both alerts with a demand signal instead, as [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) and [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) describe. +- **Autoscaling to zero** legitimately reports no pollers and no free slots. Exclude scaled-down Workers from those alerts, or scope them to Task Queues you keep warm. + +## Recommended alert set {/* #recommended-alert-set */} + +The condition column gives the tag filters that tell each alert apart. +Group every alert by `namespace` plus the tags in its condition, so that when one fires you already know which Namespace, operation, or Task Queue it came from. + +### Request failures {/* #request-failures */} + +These fire on gRPC responses coming back from the Temporal Service to your Worker or Client. +For triage, see [Request failures](/troubleshooting/request-failures). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [NOT_FOUND on respond operations](/troubleshooting/request-failures#not_found-on-respond-operations) | `request_failure` | `status_code=NOT_FOUND`, `operation` in `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, `RespondActivityTaskFailed` | Any occurrence | 5m | Critical | +| [NOT_FOUND on Activity heartbeat](/troubleshooting/request-failures#not_found-on-activity-heartbeat) | `request_failure` | `status_code=NOT_FOUND`, `operation=RecordActivityTaskHeartbeat` | Any occurrence | 5m | Warning | +| [RESOURCE_EXHAUSTED on user-facing operations](/troubleshooting/request-failures#resource_exhausted-on-user-facing-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, `ExecuteMultiOperation` | Any occurrence | 1m | Critical | +| [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations) | `request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in the four respond operations above | Any occurrence | 5m | Critical | +| [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations) | `long_request_failure` | `status_code=RESOURCE_EXHAUSTED`, `operation` in `PollWorkflowTaskQueue`, `PollActivityTaskQueue` | Any occurrence | 5m | Warning | +| [UNIMPLEMENTED from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=UNIMPLEMENTED`, any operation | Any occurrence | 2m | Critical | +| [INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service) | `request_failure` | `status_code=INTERNAL`, any operation | Any occurrence | 2m | Critical | +| [Request latency high on user-facing operations](/troubleshooting/request-failures#request-latency-high-on-user-facing-operations) | `request_latency` | `operation` in the five user-facing operations above | p99 above 2s | 5m | Critical | + +### Worker capacity {/* #worker-capacity */} + +These fire when your Workers stop keeping up with the Task Queue. +For triage, see [Worker capacity](/troubleshooting/worker-capacity). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) | `worker_task_slots_available` | `worker_type` in `WorkflowWorker`, `ActivityWorker`, `LocalActivityWorker` | Reaches 0 | 2m | Critical | +| [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) | `num_pollers` | `poller_type` in `workflow_task`, `workflow_sticky_task`, `activity_task` | Reaches 0 | 5m | Critical | +| [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero) | `request` | `operation` in `RespondWorkflowTaskCompleted`, `RespondActivityTaskCompleted` | Rate reaches 0 while the Task Queue has demand | 5m | Critical | +| [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 5s | 5m | Critical | +| [Workflow Task schedule-to-start latency severe](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) | `workflow_task_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Activity schedule-to-start latency severe](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) | `activity_schedule_to_start_latency` | `task_queue` | p99 above 30m | 5m | Critical | +| [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load) | `sticky_cache_size` | Paired with a non-zero Workflow Task rate on the same Worker | Reaches 0 | 15m | Warning | + +:::note + +`worker_task_slots_available` reports meaningful values only with fixed-size slot suppliers. +It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this one and let schedule-to-start latency tell you when capacity is short. + +::: + +### Execution failures {/* #execution-failures */} + +These fire when your Workflow or Activity code fails on the Worker. +For triage, see [Execution failures](/troubleshooting/execution-failures). + +| Failure mode | Metric | Condition | Threshold | `for` | Default severity | +| --- | --- | --- | --- | --- | --- | +| [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) | `workflow_task_execution_failed` | `failure_reason=NonDeterminismError` | Any occurrence | 1m | Critical | +| [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large) | `workflow_task_execution_failed` | `failure_reason=GrpcMessageTooLarge` | Any occurrence | 1m | Critical | +| [Workflow Task execution failures elevated](/troubleshooting/execution-failures#workflow-task-execution-failures-elevated) | `workflow_task_execution_failed` | `failure_reason=WorkflowError` | Rate above 10/s | 2m | Warning | +| [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high) | `workflow_task_execution_latency` | `task_queue`, `workflow_type` | p99 above 10s | 5m | Critical | +| [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated) | `activity_execution_failed` | `activity_type` | Rate above 10/s | 2m | Warning | +| [Unregistered Activity invocation](/troubleshooting/execution-failures#unregistered-activity-invocation) (Go SDK only) | `unregistered_activity_invocation` | `activity_type`, `task_queue`, `workflow_type` | Any occurrence | 1m | Critical | +| [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout) | `local_activity_execution_latency` | `activity_type` | p99 above 30m | 5m | Critical | + +## Route alerts by severity {/* #route-alerts-by-severity */} + +The severity in each table is a suggested routing default, not a property of the metric. + +- **Page on Critical.** These mean Workflow Executions have stopped, or are still running but losing work or duplicating side effects. +- **Send Warning to a channel someone reads during the day.** These mean something is degrading but Executions are still moving. + +Then adjust, because the defaults won't fit every deployment: + +- **Some Critical rows are early warnings, not confirmed stoppage.** High Workflow Task execution latency matters a lot on a latency-sensitive Namespace and barely at all on a batch workload where Tasks routinely run long. Downgrade the latency alerts if that describes you. +- **Some Warning rows can still lose you data.** A NOT_FOUND on Activity heartbeat means the attempt already timed out and will run again from the start. If that Activity isn't idempotent, treat it as Critical. + +One last thing worth knowing before you wire up routing: these alerts are chained together. +Task slots fill up, which drops pollers to zero, which drives schedule-to-start latency up, which drives Task completions to zero. +Expect them to fire in clusters, and check the triage pages to find which one is the cause rather than responding to all four. +When several fire together, the triage pages identify which is the root cause and which are symptoms. diff --git a/docs/references/sdk-metrics.mdx b/docs/references/sdk-metrics.mdx index ccf4939ec9..4726ecab4b 100644 --- a/docs/references/sdk-metrics.mdx +++ b/docs/references/sdk-metrics.mdx @@ -64,9 +64,9 @@ Each metric may have some combination of the following [tags](/glossary#tag) att - `namespace`: Namespace the Worker is bound to - `poller_type`: One of the following: - `workflow_task` + - `workflow_sticky_task` - `activity_task` - `nexus_task` - - `sticky_workflow_task` - `worker_type`: One of the following: - `ActivityWorker` - `WorkflowWorker` @@ -75,6 +75,8 @@ Each metric may have some combination of the following [tags](/glossary#tag) att - `activity_type`: The name of the Activity Function the metric is associated with - `workflow_type`: The name of the Workflow Function the metric is associated with - `operation`: RPC method name; available for metrics related to Temporal Client gRPC requests +- `status_code`: gRPC status code returned by the Temporal Service, in `UPPER_SNAKE_CASE` (for example, `NOT_FOUND`); available on Temporal Client gRPC request failure metrics +- `cause`: reason the Temporal Service rejected a request as resource exhausted (for example, `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`); available on the resource-exhausted metrics Some tags may not be available in every SDK, and Histogram metrics may have different buckets in each SDK. @@ -96,6 +98,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff | [temporal_long_request](#long_request) | Service Client | Counter | Core, Go, Java | | [temporal_long_request_failure](#long_request_failure) | Service Client | Counter | Core, Go, Java | | [temporal_long_request_latency](#long_request_latency) | Service Client | Histogram | Core, Go, Java | +| [temporal_long_request_resource_exhausted](#long_request_resource_exhausted) | Service Client | Counter | Go | | [temporal_nexus_poll_no_task](#nexus_poll_no_task) | Worker | Counter | Core, Go, Java | | [temporal_nexus_task_schedule_to_start_latency](#nexus_task_schedule_to_start_latency) | Worker | Histogram | Core, Go, Java | | [temporal_nexus_task_execution_failed](#nexus_task_execution_failed) | Worker | Counter | Core, Go, Java | @@ -106,6 +109,7 @@ Some tags may not be available in every SDK, and Histogram metrics may have diff | [temporal_request](#request) | Service Client | Counter | Core, Go, Java | | [temporal_request_failure](#request_failure) | Service Client | Counter | Core, Go, Java | | [temporal_request_latency](#request_latency) | Service Client | Histogram | Core, Go, Java | +| [temporal_request_resource_exhausted](#request_resource_exhausted) | Service Client | Counter | Go | | [temporal_resource_slots_cpu_usage](#resource_slots_cpu_usage) | Worker | Gauge | Core, Java | | [temporal_resource_slots_mem_usage](#resource_slots_mem_usage) | Worker | Gauge | Core, Java | | [temporal_sticky_cache_hit](#sticky_cache_hit) | Worker | Counter | Core, Go, Java | @@ -253,7 +257,11 @@ This number is included into the total `long_request` counter for long poll RPC - Type: Counter - Available in: Core, Go, Java -- Tags: `namespace`, `operation` +- Tags: `namespace`, `operation`, `status_code` + +The `status_code` tag carries the gRPC status code returned by the Temporal Service. +Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on. +The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `long_request_latency` @@ -263,6 +271,18 @@ Latency of a Temporal Client gRPC long poll request. - Available in: Core, Go, Java - Tags: `namespace`, `operation` +### `long_request_resource_exhausted` + +Temporal Client made an RPC long poll request that the Temporal Service rejected as resource exhausted. +These failures are also counted in `long_request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection. + +- Type: Counter +- Available in: Go +- Tags: `namespace`, `operation`, `cause` + +The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`. +The cause determines the remedy, so prefer this metric over `long_request_failure` when investigating throttling. + ### `nexus_poll_no_task` A Nexus Worker poll for a Nexus Task timed out, and no Nexus Task is available to pick from the Task Queue. @@ -352,7 +372,11 @@ This number is included into the total `request` counter for RPC requests. - Type: Counter - Available in: Core, Go, Java -- Tags: `namespace`, `operation` +- Tags: `namespace`, `operation`, `status_code` + +The `status_code` tag carries the gRPC status code returned by the Temporal Service. +Values are `UPPER_SNAKE_CASE`, matching the gRPC status code names: `NOT_FOUND`, `RESOURCE_EXHAUSTED`, `UNIMPLEMENTED`, `INTERNAL`, and so on. +The tag can be suppressed through Client options, so confirm it is present in your own metrics endpoint before writing queries against it. ### `request_latency` @@ -362,6 +386,18 @@ Latency of a Temporal Client gRPC request. - Available in: Core, Go, Java - Tags: `namespace`, `operation` +### `request_resource_exhausted` + +Temporal Client made an RPC request that the Temporal Service rejected as resource exhausted. +These failures are also counted in `request_failure` with `status_code=RESOURCE_EXHAUSTED`; this metric adds the reason for the rejection. + +- Type: Counter +- Available in: Go +- Tags: `namespace`, `operation`, `cause` + +The `cause` tag carries the reason the Temporal Service rejected the request, such as `RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_CONCURRENT_LIMIT`, `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED`, or `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN`. +The cause determines the remedy, so prefer this metric over `request_failure` when investigating throttling. + ### `resource_slots_cpu_usage` CPU usage as a value between 0 and 100. As perceived by the resource-based slots tuner, if @@ -506,6 +542,7 @@ A Workflow Task Execution failed. Valid values for the `failure_reason` tag: - `NonDeterminismError`: The Workflow Task failed due to a non-determinism error. +- `GrpcMessageTooLarge`: The Workflow Task response exceeded the gRPC message size limit and could not be delivered. The Temporal Service terminates the Workflow Execution in response. See [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). - `WorkflowError`: The Workflow Task failed for any other reason. ### `workflow_task_execution_latency` diff --git a/docs/troubleshooting/blob-size-limit-error.mdx b/docs/troubleshooting/blob-size-limit-error.mdx index a9c67f527b..c42850b14f 100644 --- a/docs/troubleshooting/blob-size-limit-error.mdx +++ b/docs/troubleshooting/blob-size-limit-error.mdx @@ -103,11 +103,13 @@ The error message depends on which operation carried the oversized gRPC message The behavior depends on where the oversized message originates: -- **Workflow Tasks:** The Workflow gets stuck in a retry loop that isn't visible in the Event History. This happens - because when the Worker completes a Workflow Task, it sends all the commands the Workflow produced (such as Activity - schedules and their inputs) back to the Temporal Service. If the combined size exceeds 4 MB, the SDK catches the gRPC - error and sends a failed Workflow Task response with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Replay - produces the same oversized request every time, so the Workflow never makes progress. +- **Workflow Tasks:** The Temporal Service terminates the Workflow Execution. When the Worker completes a Workflow Task, + it sends all the commands the Workflow produced (such as Activity schedules and their inputs) back to the Temporal + Service. If the combined size exceeds 4 MB, the SDK catches the gRPC error and sends a failed Workflow Task response + with cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. Because replay would produce the same oversized request + on every attempt, the Temporal Service ends the Execution with `TERMINATED` status rather than retrying it. Terminated + Executions do not retry automatically, so fix the oversized payload and deploy the corrected Worker before restarting + them. See [Execution failures](/troubleshooting/execution-failures#grpc-message-too-large) for triage steps. - **Activity Tasks:** The Activity gets stuck in a retry loop or exits with a `ScheduleToCloseTimeout`. The Activity executes successfully, but the Worker can't deliver the oversized result over gRPC. The server never receives the diff --git a/docs/troubleshooting/execution-failures.mdx b/docs/troubleshooting/execution-failures.mdx new file mode 100644 index 0000000000..a1ae90862e --- /dev/null +++ b/docs/troubleshooting/execution-failures.mdx @@ -0,0 +1,216 @@ +--- +id: execution-failures +title: Troubleshoot Workflow and Activity execution failures +sidebar_label: Execution failures +description: Diagnose non-determinism errors, oversized payloads, failing Workflow and Activity code, and Local Activity latency +toc_max_heading_level: 3 +keywords: + - non determinism error + - workflow task failed + - activity execution failed + - local activity + - grpc message too large +tags: + - Metrics + - Observability + - Troubleshooting +--- + +This guide covers failures that happen while your Workflow and Activity code is running on a Worker: replay mismatches, oversized responses, unhandled exceptions, and Local Activities that outrun the Workflow Task heartbeat timeout. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). +For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). + +[`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed) carries a `failure_reason` tag along with `namespace`, `task_queue`, and `workflow_type`. +The reason decides what the Temporal Service does next, and the difference is worth knowing: two of the three retry forever, one terminates the Execution on the spot. +Alert on each `failure_reason` separately. + +## Non-determinism error {/* #non-determinism-error */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=NonDeterminismError` + +Replay produced a different sequence of commands than the one recorded in Event History. +The Worker noticed that the Workflow code it is running no longer matches what the Execution has already done. + +**Why it matters.** +Affected Executions stop making progress. +The Temporal Service retries the Workflow Task over and over, loading up your Workflow Workers, and by default those Executions sit in Running status the whole time. +This will not clear up on its own. + +**Triage.** + +1. **Identify the affected Workflow Executions.** This metric does not carry a Workflow Id. Worker logs record the error with the Workflow Id and Run Id. In the Temporal UI you can also find affected Executions by querying the `TemporalReportedProblems` Search Attribute, which the Temporal Service sets on Executions experiencing repeated Workflow Task failures. +1. **Read the error.** The `WorkflowTaskFailed` event in an affected Execution's Event History contains the message identifying exactly where replay diverged and which command was expected versus produced. This is the most direct signal for root cause. +1. **Determine whether this is a code change or a deploy artifact.** Common causes: + - A code change added, removed, or reordered commands such as Activity scheduling, Timers, Signals, or Child Workflows, without a versioning guard. Executions that built their History under the old code fail on the new code. + - A rolling restart with old and new Worker versions briefly running together. Some Executions fail transiently and recover once the rollout completes. If your deploys routinely trigger this alert, lengthen its `for` duration past how long a rollout takes. + - Changed Activity or Timer parameters in existing Workflow code without versioning. +1. **Roll back if it isn't clearing.** If the errors started after a deploy and aren't going away, roll the Worker back. Affected Executions pick up again on their next Workflow Task retry once compatible code is running. Then add a proper versioning guard before you redeploy. See [Versioning Workflows](/workflow-definition#workflow-versioning) and [Worker Versioning](/production-deployment/worker-deployments/worker-versioning). +1. **Watch Worker pressure.** All those retries add up. Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker` and [Workflow Task execution latency high](#workflow-task-execution-latency-high). Enough retry volume will saturate your capacity and start hurting healthy Executions on the same Task Queue. + +## gRPC message too large {/* #grpc-message-too-large */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=GrpcMessageTooLarge` + +The Workflow Task response was bigger than the gRPC message size limit. +The Worker tried `RespondWorkflowTaskCompleted` and something rejected it: the gRPC library on the SDK side, a proxy or load balancer in the path, or the gRPC library on the Temporal Service side when it went to receive. + +The Temporal Service never saw that original request, so the SDK follows up with `RespondWorkflowTaskFailed` and cause `WORKFLOW_TASK_FAILED_CAUSE_GRPC_MESSAGE_TOO_LARGE`. +Replay would build the same oversized response every time, so the Service terminates the Workflow Execution instead of retrying it. + +For the payload size limits themselves, and for the Activity Task version of this (which retries instead of terminating), see [Troubleshoot the BlobSizeLimitError](/troubleshooting/blob-size-limit-error). + +**Why it matters.** +Affected Executions end immediately and permanently, with `TERMINATED` status and no retry. +Whatever work was in progress is gone, and someone has to restart them by hand. + +This is the one `failure_reason` on this page that kills Executions instead of retrying them, so give it a short `for` duration. + +**Triage.** + +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Check Worker logs for Workflow Ids and Run Ids, then confirm the cause from the `WorkflowTaskFailed` and `WorkflowExecutionTerminated` events in Event History. +1. **Find what is oversized.** The fix depends entirely on which part of the response is too large: + - **Oversized Activity inputs or outputs.** Move the payload out of band: put it in blob storage and pass a reference through Event History instead. See [External Storage](/external-storage) for the pattern. + - **Accumulated Signals or Updates.** A large number buffered into a single Workflow Task. Rate-limit senders or batch Signals. + - **Too many commands in one response.** A Workflow scheduling a very large fan-out of Activities or Child Workflows in a single step. Break the fan-out into smaller batches across multiple Workflow Tasks. +1. **Fix and deploy before restarting anything.** Terminated Executions do not retry. Restarting them before the cause is fixed means hitting the same limit and being terminated again. Once the corrected Worker is deployed and verified, restart the affected Executions from the Temporal UI or CLI. + +:::note Self-hosted Temporal Service + +Check the Workflow terminate rate on your server dashboard. A spike alongside this metric confirms Executions are being terminated in bulk. + +::: + +## Workflow Task execution failures elevated {/* #workflow-task-execution-failures-elevated */} + +**Metric:** `temporal_workflow_task_execution_failed` with `failure_reason=WorkflowError` + +Workflow Tasks are failing steadily from unhandled exceptions and panics in Workflow code that the SDK catches and reports. +`WorkflowError` is the catch-all reason. It covers thread pool exhaustion, unhandled exceptions thrown inside the Workflow function, and Data Converter errors. + +**Why it matters.** +The Temporal Service retries the Workflow Task. +If the error is deterministic and shows up on every replay, the Execution is stuck retrying forever, burning Worker capacity and never getting healthy. + +At high rates the retry pressure fills up your Workflow Worker slots and starts affecting healthy Executions on the same Task Queue. +Unlike `GrpcMessageTooLarge`, the Temporal Service won't terminate the Execution for you, so this gets worse the longer you leave it. + +**Triage.** + +1. **Identify the affected Executions.** This metric does not carry a Workflow Id. Worker logs carry the Workflow Id, Run Id, and full stack trace. The `WorkflowTaskFailed` event in Event History carries the error message and type. The `workflow_type` tag on the metric narrows which Workflow is failing. +1. **Determine which failure mode this is.** `WorkflowError` covers several: + - **Thread pool exhaustion (Java SDK).** A `RejectedExecutionException` from a saturated Workflow thread pool. `setMaxWorkflowThreadCount` on `WorkerFactoryOptions` is too low for the number of concurrent Executions, so new Workflow Tasks get rejected before they run. Raise the thread count, and think about whether the Worker pool needs to scale out too. + - **Unhandled exception in Workflow code.** A bug or an unexpected condition throws. If it reproduces on every replay, the Execution is stuck. The `WorkflowTaskFailed` event names the error. + - **Data Converter error.** Something failed serializing or deserializing Workflow inputs, outputs, or Memo fields. Check your [Data Converter](/dataconversion) and [Payload Codec](/payload-codec) configuration. +1. **Check Worker thread and slot pressure.** Cross-check [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted) for `worker_type=WorkflowWorker`. Slot exhaustion and thread pool exhaustion tend to show up together under load, and a CPU-starved Worker is slower to finish Workflow Tasks, which makes both worse. +1. **Fix and redeploy.** Affected Executions resume on their next Workflow Task retry once compatible code is running. + +## Workflow Task execution latency high {/* #workflow-task-execution-latency-high */} + +**Metric:** [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency), tagged `namespace`, `task_queue`, and `workflow_type` + +Workflow Tasks are taking too long to execute on the Worker. +The default Workflow Task timeout is 10 seconds, so at or above that value the Temporal Service is actively timing out Workflow Tasks. + +A batch workload where Workflow Tasks routinely run long can sit above this threshold all day without anything being wrong. +Set the threshold from your own observed p99, and only treat the default as meaningful if your Workflows are latency-sensitive. + +**Why it matters.** +The Temporal Service writes `WorkflowTaskTimedOut` events to Event History and reschedules timed-out Tasks on the normal Task Queue. +Each timeout forces a [Sticky Execution](/sticky-execution) cache eviction on the Worker holding the Execution, so the next Workflow Task for it requires a full cold replay. + +If you run [Local Activities](/local-activity), a Workflow Task timeout causes them to re-execute from scratch on the retried Task, because their results are not checkpointed between Workflow Task heartbeats. +Non-idempotent Local Activities produce duplicate side effects with real business impact. + +At scale this compounds: more timeouts cause more cold replays, cold replays drive latency higher, and higher latency causes more timeouts. + +**Triage.** + +1. **Check replay latency.** Look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency). If it is high, the time is going into re-running Event History rather than into new commands. Large histories, a slow Data Converter during replay, or a high cache eviction rate forcing cold replays are the usual culprits. +1. **Check the Sticky Execution cache.** A high forced-eviction rate causes a cold replay on every Workflow Task. See [Sticky cache holding zero entries under load](/troubleshooting/worker-capacity#sticky-cache-holding-zero-entries-under-load). +1. **Check Worker CPU.** If replay latency is normal but execution latency is high, the time is going into new command execution. High CPU slows all code on the Worker. +1. **Check for blocking Workflow code.** Workflow code must not perform blocking I/O, heavy computation, or synchronous non-Temporal calls. Any blocking call holds the Task slot and inflates this metric. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). The SDK holds the slot until the respond call succeeds, which inflates this metric even when your Workflow code finished quickly. + +## Activity execution failures elevated {/* #activity-execution-failures-elevated */} + +**Metric:** [`temporal_activity_execution_failed`](/references/sdk-metrics#activity_execution_failed), tagged `activity_type` + +Activities are failing outright at a sustained rate: returning failures, not timing out. + +`ApplicationFailure` instances marked with category `BENIGN` don't increment this counter, so how well this metric tracks only the unexpected failures depends on how consistently your application marks the expected ones. + +**Why it matters.** +A high failure rate means a burst of retry Tasks. +If your Workers can't keep up with the retry volume, the Activity Task backlog grows. See [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated). +At scale, sustained retry bursts put real pressure on Task matching and the database underneath. + +**Triage.** + +1. **Identify which Activity is failing.** The `activity_type` tag narrows it down. Worker logs for that type carry the error messages, stack traces, and associated Workflow Ids. +1. **Work out whether this is transient or a bug.** A downstream outage, a network partition, or a database timeout will recover on its own, so watch whether the rate falls. A code bug won't. +1. **Check downstream service health.** A struggling dependency is a common cause of sustained failure bursts. If it is throttling you, check that your [Retry Policy](/encyclopedia/retry-policies) has sensible backoff. Without it, your retries pile more pressure onto something that is already overloaded. +1. **Check schedule-to-start latency.** A growing retry backlog shows up as elevated [Activity schedule-to-start latency](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) even after the failure rate drops. +1. **Mark expected failures as benign.** If your design fails Activities on purpose, as polling patterns, Saga compensations, and flow control through exceptions all do, mark those `ApplicationFailure` instances with category `BENIGN`. That keeps them out of this metric and lets the alert track only the failures you didn't expect, without tuning a threshold per `activity_type`. Check that your SDK version supports the category before you rely on it. + +One caveat: internal failures increment this counter no matter what category you set. Those are things like a context propagation error or a context timeout, rather than an Activity returning a failure. + +## Unregistered Activity invocation {/* #unregistered-activity-invocation */} + +**Metric:** [`temporal_unregistered_activity_invocation`](/references/sdk-metrics#unregistered_activity_invocation), tagged `activity_type`, `task_queue`, and `workflow_type` + +A Workflow scheduled an Activity that the Worker polling that Task Queue has no registered implementation for. + +This metric is emitted by the Go SDK only. + +**Why it matters.** +The Activity can't run. It keeps getting retried against a Worker that has no implementation for it, until the Activity's `scheduleToClose` timeout expires, or forever if you haven't set one. Meanwhile the Workflow Execution waiting on it goes nowhere. + +This is nearly always a deployment mistake rather than a runtime condition: Workflow code scheduling an Activity that the deployed Worker doesn't register. It won't fix itself. + +**Triage.** + +1. **Identify the Activity and the Task Queue.** The `activity_type` and `task_queue` tags name both. The `workflow_type` tag identifies which Workflow is scheduling it. +1. **Check whether the Activity is registered on the right Worker.** Confirm the Worker polling that Task Queue registers that Activity type. A common cause is registering the Activity on a Worker polling a different Task Queue. +1. **Check for a partial rollout.** If Workflow code that schedules a new Activity deployed ahead of the Worker that implements it, some Workers will be running without the registration. Complete the rollout. +1. **Check for a renamed Activity.** Changing an Activity's registered name while Executions are in flight leaves those Executions scheduling the old name. Register both names until the in-flight Executions drain, or use a versioning guard. + +## Local Activity latency exceeds the heartbeat timeout {/* #local-activity-latency-exceeds-the-heartbeat-timeout */} + +**Metric:** [`temporal_local_activity_execution_latency`](/references/sdk-metrics#local_activity_execution_latency), tagged `activity_type` + +A [Local Activity](/local-activity) is running past the Workflow Task heartbeat timeout, which defaults to 30 minutes. + +### How Workflow Task heartbeating works {/* #how-workflow-task-heartbeating-works */} + +A Local Activity executes inside the Workflow Task rather than as a separately scheduled Activity Task. +That means the Workflow Task stays open for as long as the Local Activity runs, which would normally exceed the Workflow Task timeout. + +To keep the Task alive, the SDK sends Workflow Task heartbeats: repeated `RespondWorkflowTaskCompleted` calls that tell the Temporal Service work is still going and ask for more time. +The Service goes along with this up to the Workflow Task heartbeat timeout. +After that it times the Task out and reschedules it on the normal Task Queue. + +Local Activities cannot heartbeat individually the way regular Activities can, and their results are not recorded in Event History between Workflow Task heartbeats. +So when the Task is rescheduled, every Local Activity in it runs again from the beginning. + +**Why it matters.** +When the Temporal Service times out the heartbeating Workflow Task, the Local Activity re-executes from scratch. +A non-idempotent Local Activity produces duplicate side effects with real business impact. + +Any pending Signals, Updates, or other events are delayed until the retried Workflow Task completes, so end-to-end Execution latency rises significantly. + +The Local Activity also occupies an executor slot for its entire duration. +Several in this state at once can occupy every available slot, blocking new Local Activities from starting. +See [LocalActivityWorker slots](/troubleshooting/worker-capacity#localactivityworker-slots). + +Local Activities are designed for short, fast operations. +A single attempt running for 30 minutes is a design problem, not a tuning problem. + +**Triage.** + +1. **Identify the affected Local Activity.** The `activity_type` tag narrows it down. Worker logs for that type show what it is doing, how long individual attempts run, and the associated Workflow Ids. +1. **Find what it is blocked on.** A Local Activity running this long is nearly always stuck on a downstream call: a slow service, a slow query, or a network call with a very generous timeout. Fix the dependency, or shorten that timeout so the Local Activity fails fast instead of hanging. +1. **Check whether a retry chain is accumulating.** A high failure rate paired with an aggressive Retry Policy can push total elapsed time past the heartbeat timeout even when every individual attempt is short. Check [`temporal_local_activity_execution_failed`](/references/sdk-metrics#local_activity_execution_failed) for the same `activity_type`, and fix the underlying failure first. +1. **Check whether timeouts have already happened.** By the time this fires, the Temporal Service may have timed out heartbeating Workflow Tasks already. Look for timeout errors in Worker logs and `WorkflowTaskTimedOut` events in Event History. If they are there, your Local Activities have already run twice, so check whether they are idempotent and clean up any duplicate side effects. +1. **Fix the design.** If the work genuinely takes this long, convert it to a regular Activity with heartbeating, which is the correct primitive for long-running work. If it must stay a Local Activity, set a `scheduleToCloseTimeout` below the Workflow Task heartbeat timeout so it fails with a timeout error the Workflow can handle, rather than having the entire Workflow Task re-executed. diff --git a/docs/troubleshooting/index.mdx b/docs/troubleshooting/index.mdx index 6a3bcad470..291c61ef56 100644 --- a/docs/troubleshooting/index.mdx +++ b/docs/troubleshooting/index.mdx @@ -24,4 +24,7 @@ Our troubleshooting guides are designed to help you quickly identify and resolve - [Troubleshoot the Failed Reaching Server Error](/troubleshooting/last-connection-error): The message "Failed reaching server: last connection error" often happens due to an expired TLS certificate or during the Server startup process when Client requests reach the Server before roles are fully initialized. - [Troubleshoot missed Schedule Actions](/troubleshooting/schedule-missed-actions): When a Schedule does not fire at its expected time, alert on the missed catchup window metric, then narrow down to the affected Schedule with `ListSchedules` and `DescribeSchedule`. - [Troubleshoot Serverless Workers](/troubleshooting/serverless-workers): Diagnose issues with Serverless Workers by tracing the flow from Task Queue to Worker execution, on [AWS Lambda](/troubleshooting/serverless-workers/aws-lambda) or [GCP Cloud Run](/troubleshooting/serverless-workers/cloud-run). +- [Troubleshoot request failures](/troubleshooting/request-failures): Diagnose gRPC failures and slow requests from Workers and Clients to the Temporal Service, including NOT_FOUND on respond operations, RESOURCE_EXHAUSTED throttling, and UNIMPLEMENTED or INTERNAL responses. +- [Troubleshoot Worker capacity](/troubleshooting/worker-capacity): Diagnose exhausted Task slots, disconnected pollers, Task completions dropping to zero, rising schedule-to-start latency, and an empty Sticky Execution cache. +- [Troubleshoot Workflow and Activity execution failures](/troubleshooting/execution-failures): Diagnose non-determinism errors, oversized Workflow Task responses, unhandled exceptions in Workflow and Activity code, and Local Activities running past the Workflow Task heartbeat timeout. - [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows): Recover pinned Workflows that have failed or are stuck retrying tasks after rolling out a faulty Worker Deployment Version. diff --git a/docs/troubleshooting/request-failures.mdx b/docs/troubleshooting/request-failures.mdx new file mode 100644 index 0000000000..95234c50ab --- /dev/null +++ b/docs/troubleshooting/request-failures.mdx @@ -0,0 +1,250 @@ +--- +id: request-failures +title: Troubleshoot request failures +sidebar_label: Request failures +description: Diagnose gRPC failures and slow requests from Temporal Workers and Clients +toc_max_heading_level: 3 +keywords: + - sdk metrics + - request failure + - resource exhausted + - grpc errors + - monitoring temporal metrics +tags: + - Metrics + - Observability + - Troubleshooting +--- + +This guide covers gRPC failures and slow requests between your Temporal SDK Workers or Clients and the Temporal Service. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). +For metric definitions, see the [Temporal SDK metrics reference](/references/sdk-metrics). + +[`temporal_request_failure`](/references/sdk-metrics#request_failure) increments when the Temporal Service returns a non-OK gRPC status code on a standard operation. +[`temporal_long_request_failure`](/references/sdk-metrics#long_request_failure) covers poll operations and long-poll `GetWorkflowExecutionHistory`. +Both carry `namespace`, `operation`, and `status_code` tags. +How urgent a failure is depends on which status code turned up on which operation. + +:::note + +Two naming details will trip up your queries. +`UpdateWithStartWorkflowExecution` shows up in SDK metrics under the gRPC operation name `ExecuteMultiOperation`. +And `status_code` values are `UPPER_SNAKE_CASE` in every SDK, matching the gRPC status code names. Client options can also turn the tag off, so check that it is present in your metrics endpoint before you filter on it. + +::: + +## NOT_FOUND on respond operations {/* #not_found-on-respond-operations */} + +**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` + +A Worker finished a Workflow Task or Activity Task and reported the result, and the Temporal Service replied that the task no longer exists. +There are three causes: + +- The task timed out. The Worker ran past the Workflow Task timeout, or past the Activity `startToClose` or `scheduleToClose` timeout, and the Service discarded the in-flight task. +- The Workflow Execution is no longer running. It completed, was terminated, or hit its Workflow Run Timeout before the task finished. +- The Worker restarted mid-execution. The in-flight Task Token was lost, the Service rescheduled the task, and the original Worker still attempted to respond after coming back up. + +The last two happen during normal operation, so a few of these are nothing to worry about. +A sustained rate is. + +**Why it matters.** +The Temporal Service threw away the result your Worker just produced. +For Activities, the Service has already rescheduled the Activity for retry if the Retry Policy allows it. +For Workflow Tasks, the Service writes a `WorkflowTaskTimedOut` event to Event History and reschedules the task on the normal Task Queue, which forces a [Sticky Execution](/sticky-execution) cache eviction and a cold replay on the retry. + +A sustained rate means your Workers are finishing too late, over and over. +Every discarded result is Worker capacity you paid for and got nothing back from, and every rescheduled task adds to how long your Workflows take end to end. + +If you run [Local Activities](/local-activity), a Workflow Task timeout makes them run again from the start on the retried task. +Their results are not written to Event History between Workflow Task heartbeats, so there is nothing to resume from. +If they aren't idempotent, you get duplicate side effects, and that is a business problem rather than a monitoring one. + +**Triage.** + +1. **Rule out the expected causes first.** Check the status of a few affected Executions in the Temporal UI or with `temporal workflow describe`. If they completed, were terminated, or hit their Run Timeout, the NOT_FOUND is expected. Check Worker restart counts in your infrastructure observability stack for the same reason. If either explains the volume, stop here. +1. **Check task execution latency.** For Workflow Tasks, check [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency). For Activities, check [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type`. If p99 is at or above the corresponding timeout, that is the direct cause. +1. **Check replay latency.** If Workflow Task execution latency is high, look at [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) next. When replay latency is high, the Worker is burning its time re-running Event History instead of getting to the new commands. Look for large histories and a slow [Data Converter](/dataconversion). +1. **Check Worker resources.** High CPU on the Worker slows task execution directly. Look at the `identity` field in the `WorkflowTaskStarted` or `ActivityTaskStarted` event to identify which Worker ran the task, then check that pod for CPU saturation and cold-start delays. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](#resource_exhausted-on-respond-operations). Sustained throttling can delay a respond call long enough for the Service to time out the task before the response lands. + +:::note Self-hosted Temporal Service + +If SDK-side metrics look normal and the Execution was not terminated or timed out, check server-side latency: Frontend Service latency filtered to the affected respond operation, and persistence latency filtered to `UpdateWorkflowExecution`. + +::: + +## NOT_FOUND on Activity heartbeat {/* #not_found-on-activity-heartbeat */} + +**Metric:** `temporal_request_failure` with `status_code=NOT_FOUND` on `RecordActivityTaskHeartbeat` + +A Worker heartbeated a running Activity and the Temporal Service replied that the task no longer exists. +The Service has already cancelled the in-flight Activity Task: either the `heartbeatTimeout` fired before the next heartbeat call arrived, the `startToClose` timeout expired while the Activity was still executing, or the Workflow Execution is no longer running. + +Normal Workflow-side cancellation is not a cause. +Cancellation returns `CancelRequested=true` in the heartbeat response body rather than a gRPC error, so NOT_FOUND on this operation is a reliable signal of a timeout or forced closure. + +**Why it matters.** +If `heartbeatTimeout` is the cause, the Service has already timed out this Activity attempt and scheduled a retry if the Retry Policy allows it. +The Activity runs again from the start on the next attempt, so a non-idempotent Activity will duplicate its side effects. That is worth treating more seriously than the default Warning severity suggests. + +A sustained rate means the Worker keeps missing its heartbeat window. +The Activity will time out on every attempt until you fix the cause, holding Task slots and generating retry tasks the whole time. + +**Triage.** + +1. **Compare the heartbeat interval against `heartbeatTimeout`.** The Worker must call heartbeat more frequently than the timeout. If the Activity slows down between heartbeat calls because of CPU pressure, blocking I/O, or downstream throttling, the effective interval grows past the timeout even though the code is calling heartbeat. +1. **Check Worker CPU.** A CPU-starved Worker slows down between heartbeat calls even when the Activity is making progress. If utilization is consistently high, reduce per-Worker concurrency or scale out horizontally. +1. **Check the `startToClose` timeout.** If the Activity has run longer than `startToClose`, the Service times it out while the Activity is still executing, and the next heartbeat returns NOT_FOUND. Compare [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` against the configured timeout. +1. **Check for throttling on heartbeat calls.** Query `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` and `operation=RecordActivityTaskHeartbeat`. If the Temporal Service is throttling these calls, the effective heartbeat interval grows past `heartbeatTimeout` even when the Worker calls on time. See [RESOURCE_EXHAUSTED on poll operations](#resource_exhausted-on-poll-operations) for how to work through a throttling cause. +1. **Check heartbeat payload size.** The last heartbeat details payload is held in memory for the life of the Activity attempt. Large payloads on high-throughput Activity Workers contribute to memory pressure on the Temporal Service. Store only the minimum progress state needed to resume on retry. + +## RESOURCE_EXHAUSTED on user-facing operations {/* #resource_exhausted-on-user-facing-operations */} + +**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, `UpdateWorkflowExecution`, or `ExecuteMultiOperation` + +The Temporal Service is throttling the operations your application code uses to start Workflows and deliver Signals and Updates. +The SDK retries these automatically for up to 60 seconds. +Beyond that, the call fails and the error propagates to your caller. + +**Why it matters.** +These operations are on your application's critical path. +Within the retry window, callers experience elevated latency. +Past it, calls fail outright and your application must handle the error. + +If it does not, starts and Signals are silently dropped. +A dropped start means the Workflow never runs. +A dropped Signal or Update means a running Workflow never receives input it is waiting on, and may stall indefinitely. +Log these failures in your application code so you can backfill starts and Signals afterward. + +The Temporal Service throttles these operations last. +Seeing RESOURCE_EXHAUSTED here means throttling is already severe and widespread. + +**Triage.** + +1. **Find out which limit you hit.** A Namespace rate limit, a concurrency limit, system-wide overload, and an open circuit breaker are four different problems with four different fixes. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by its `cause` tag and it will tell you directly. On other SDKs, work through the steps below. +1. **Check your traffic against your Namespace limits.** For a rate limit cause (`RESOURCE_EXHAUSTED_CAUSE_RPS_LIMIT` or `RESOURCE_EXHAUSTED_CAUSE_APS_LIMIT`), compare current throughput against your Namespace's [service limits](/cloud/limits) on Temporal Cloud, and open a support request if you need them raised. +1. **Treat overload and circuit-breaker causes as capacity problems.** `RESOURCE_EXHAUSTED_CAUSE_SYSTEM_OVERLOADED` and `RESOURCE_EXHAUSTED_CAUSE_CIRCUIT_BREAKER_OPEN` mean the Temporal Service is shedding load to protect itself. A higher limit will not help; the Service needs capacity, or your workload needs to slow down. +1. **Add backoff in your application.** If throttling is expected during traffic peaks, ensure calling code retries with backoff rather than tight-looping, which amplifies the pressure. + +:::note Self-hosted Temporal Service + +Check the resource-exhausted cause on your server dashboard, then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`. +Slow persistence is the usual root cause of throttling that cascades like this. +If the cause is a rate limit, `frontend.namespaceRPS` may be too low for your traffic, but confirm persistence is healthy before you raise it. +If the cause is system overload or an open circuit breaker, the Temporal Service is shedding load to protect itself. It needs capacity, not a higher limit. + +::: + +## RESOURCE_EXHAUSTED on respond operations {/* #resource_exhausted-on-respond-operations */} + +**Metric:** `temporal_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `RespondWorkflowTaskCompleted`, `RespondWorkflowTaskFailed`, `RespondActivityTaskCompleted`, or `RespondActivityTaskFailed` + +The Temporal Service is throttling Workers reporting task results. +The SDK retries automatically, but a delayed respond call has a compounding cost: the Worker holds the Task slot until the call succeeds, and the Service-side task stays in-flight until the response lands. + +**Why it matters.** +This is a leading indicator of [NOT_FOUND on respond operations](#not_found-on-respond-operations). +If throttling goes on long enough the task times out, the Service writes a `WorkflowTaskTimedOut` or `ActivityTaskTimedOut` event, and the task gets rescheduled, with all the cache eviction, cold replay, and Local Activity re-execution that section describes. + +Meanwhile every in-flight task is still holding its slot, so you have less concurrency for new work. +Left alone, this turns into [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted). + +**Triage.** + +1. **Find out which limit you hit,** as in the section above. On the Go SDK, group [`temporal_request_resource_exhausted`](/references/sdk-metrics#request_resource_exhausted) by `cause`. +1. **Check whether timeouts have already started.** If [NOT_FOUND on respond operations](#not_found-on-respond-operations) is also firing, throttling has already cascaded into task timeouts and Executions are losing work. +1. **Check Task slot availability.** See [Worker Task slots exhausted](/troubleshooting/worker-capacity#worker-task-slots-exhausted). Slots aren't released until the respond call succeeds, so throttling here drains the pool. + +:::note Self-hosted Temporal Service + +Check persistence latency filtered to `UpdateWorkflowExecution`. Slow persistence on that operation is the usual root cause of throttling on respond operations. + +::: + +## RESOURCE_EXHAUSTED on poll operations {/* #resource_exhausted-on-poll-operations */} + +**Metric:** `temporal_long_request_failure` with `status_code=RESOURCE_EXHAUSTED` on `PollWorkflowTaskQueue` or `PollActivityTaskQueue` + +The Temporal Service is throttling Worker poll calls. +Poll operations are long-poll requests, so they increment `temporal_long_request_failure` rather than `temporal_request_failure`. + +**Why it matters.** +Throttled Workers back off and poll less frequently, which lowers the effective poll rate for the Task Queue even when every Worker is healthy. +That shows up as rising schedule-to-start latency and, if it persists, as a growing Task backlog. + +This is often a symptom rather than a cause. +The Temporal Service throttles poll operations before it throttles respond or user-facing operations, so throttling here can be the first visible sign of pressure that has nothing to do with your Workers. + +**Triage.** + +1. **Identify the throttle cause.** As with the other throttling sections, a Namespace rate limit, a concurrency limit, and system-wide overload need different responses. On the Go SDK, [`temporal_long_request_resource_exhausted`](/references/sdk-metrics#long_request_resource_exhausted) carries the `cause` tag for poll operations. +1. **Check whether you are over-polling.** A large number of Workers each configured with many concurrent pollers can exceed the Namespace poller limit without processing any more work. Check your configured poller counts against [Worker performance](/develop/worker-performance) guidance before assuming the limit is too low. +1. **Check downstream effects.** See [Workflow Task schedule-to-start latency elevated](/troubleshooting/worker-capacity#workflow-task-schedule-to-start-latency-elevated) and [Activity schedule-to-start latency elevated](/troubleshooting/worker-capacity#activity-schedule-to-start-latency-elevated) to gauge whether throttling is affecting Task dispatch yet. +1. **Check your traffic against your Namespace limits.** On Temporal Cloud, compare against your Namespace's [service limits](/cloud/limits). + +:::note Self-hosted Temporal Service + +If poll operations are being throttled at scale, you may need to raise the Namespace concurrent poller limit through `frontend.namespaceCount` or `frontend.globalNamespaceCount`. +Scale Worker capacity first if schedule-to-start latency is the real problem. + +::: + +## UNIMPLEMENTED or INTERNAL from the Temporal Service {/* #unimplemented-or-internal-from-the-temporal-service */} + +**Metric:** `temporal_request_failure` with `status_code=UNIMPLEMENTED` or `status_code=INTERNAL`, on any operation + +These two status codes point at the Temporal Service rather than at your application, and they behave differently in the SDK. +Alert on them separately. + +**UNIMPLEMENTED** means the Service does not recognize an operation the Worker called. +By the time a Worker reaches steady-state polling it has already called `GetSystemInfo` and `DescribeNamespace` successfully, so this is rarely a plain version mismatch on a freshly deployed Worker. +Most SDK versions treat UNIMPLEMENTED as non-retryable: the Worker surfaces it as a fatal error and may shut down. + +**INTERNAL** means the Service encountered an error it could not attribute to the request. +Short bursts during Service restarts and rolling deploys are normal, so set the `for` duration long enough that your own deploys don't page you. +The SDK retries INTERNAL, but sustained errors exhaust the retry budget and surface to callers. +Workers receiving INTERNAL on poll operations back off and poll less frequently, which raises schedule-to-start latency. + +**Triage.** + +1. **Check SDK and Temporal Service version compatibility.** For UNIMPLEMENTED, confirm your SDK version is not calling an API that has been removed or changed in your Service version. +1. **Check whether recent deploys correlate.** Both codes commonly appear immediately after a Service upgrade or a Worker deploy. If the timing lines up, consider rolling back while you investigate. +1. **Check whether the errors are Namespace-scoped or cluster-wide.** Errors isolated to one Namespace point at Namespace configuration. Cluster-wide errors point at infrastructure. +1. **Check downstream effects.** Sustained errors on poll operations cause Workers to back off. Cross-check [All pollers disconnected](/troubleshooting/worker-capacity#all-pollers-disconnected) and [Task completions dropped to zero](/troubleshooting/worker-capacity#task-completions-dropped-to-zero). + +:::note Self-hosted Temporal Service + +Check service panics first. Any panic is critical, and it is almost always the root cause of sustained INTERNAL errors. +Then check persistence errors and availability; the Temporal Service wraps database errors as INTERNAL. +For UNIMPLEMENTED, check that every Frontend, History, and Matching pod is running the binary you intended. A wrong or corrupted binary on a few pods returns UNIMPLEMENTED on perfectly valid operations, usually alongside panics. + +::: + +## Request latency high on user-facing operations {/* #request-latency-high-on-user-facing-operations */} + +**Metric:** [`temporal_request_latency`](/references/sdk-metrics#request_latency) on `StartWorkflowExecution`, `SignalWithStartWorkflowExecution`, `SignalWorkflowExecution`, or `ExecuteMultiOperation` + +p99 latency on the operations your application calls synchronously has risen above your threshold. + +**Why it matters.** +These calls block your application code while they wait on the Temporal Service, so the latency is felt directly by your users and by anything downstream of the call completing. + +The SDK retries transient errors but does not hide the latency cost: every retry adds to the total time this metric observes. +If throttling is the cause and retries exhaust the 60-second budget, the call fails outright. + +**Triage.** + +1. **Check for throttling on the same operations.** See [RESOURCE_EXHAUSTED on user-facing operations](#resource_exhausted-on-user-facing-operations). If both are firing, throttling is the cause of the latency and retries are what you are measuring. +1. **Check payload sizes.** This metric includes serialization and network time. Large Workflow inputs or Signal payloads, or an expensive [Payload Codec](/payload-codec), raise it without any Service-side slowdown. +1. **Check network path and region.** Clients in a different region from the Temporal Service pay that round trip on every call. + +:::note Self-hosted Temporal Service + +Check Frontend Service latency filtered to the affected operations. +Server-side latency is the sharper signal here, because the SDK metric also includes serialization and network time. +Then check persistence latency filtered to `CreateWorkflowExecution` and `UpdateWorkflowExecution`, which is what usually drives up Frontend latency on starts and Signals. +If persistence is healthy and nothing is being throttled, check Frontend pod CPU. + +::: diff --git a/docs/troubleshooting/worker-capacity.mdx b/docs/troubleshooting/worker-capacity.mdx new file mode 100644 index 0000000000..45ed439fc2 --- /dev/null +++ b/docs/troubleshooting/worker-capacity.mdx @@ -0,0 +1,254 @@ +--- +id: worker-capacity +title: Troubleshoot Worker capacity +sidebar_label: Worker capacity +description: Diagnose exhausted Task slots, disconnected pollers, Task backlog, and Sticky Execution cache problems +toc_max_heading_level: 3 +keywords: + - worker capacity + - task slots + - pollers + - schedule to start latency + - sticky cache + - task backlog +tags: + - Metrics + - Observability + - Troubleshooting + - Workers +--- + +This guide covers what to do when your Workers stop keeping up with their Task Queues: slots that never free up, pollers that quit polling, Tasks sitting in the queue, and a [Sticky Execution](/sticky-execution) cache that isn't earning its keep. +It applies to Workers connected to Temporal Cloud and to a self-hosted Temporal Service. + +For alert thresholds and `for` durations, see [Worker alerting](/best-practices/worker-alerting). +For Worker sizing and tuning, see [Worker performance](/develop/worker-performance) and [Worker deployment and performance](/best-practices/worker). + +These failure modes are chained together. +Slots fill up, polling stops, Tasks pile up, completions go to zero. +When several fire at once, don't treat them as separate problems. Work backward to the earliest link in the chain. + +## Worker Task slots exhausted {/* #worker-task-slots-exhausted */} + +**Metric:** [`temporal_worker_task_slots_available`](/references/sdk-metrics#worker_task_slots_available) reaching 0, tagged `worker_type` + +Every Task execution slot for this `worker_type` and Task Queue is occupied, so no new Tasks can be picked up. +The SDK blocks before issuing its next poll until a slot is released. + +:::note + +This metric reports meaningful values only with fixed-size slot suppliers. +It can't be used with resource-based slot suppliers. See [Slot availability metrics](/develop/worker-performance#slot-availability-metrics). +If your Workers use resource-based tuning, skip this section and let [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated) tell you when capacity is short. + +::: + +**Why it matters.** +Slots stay occupied because the Tasks already holding them aren't finishing. +Until they do, schedule-to-start latency climbs. In the Go and Java SDKs, [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) also drops to zero, because both block on getting a slot before they bump the poller gauge. + +What this costs you and how you fix it both depend on `worker_type`, so check that tag on the firing alert. +The values are `WorkflowWorker`, `ActivityWorker`, and `LocalActivityWorker`. + +### WorkflowWorker slots {/* #workflowworker-slots */} + +1. **Check Workflow Task execution latency.** Sustained high [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) confirms something is holding slots. See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high). +1. **Check Worker CPU.** High CPU slows Workflow Task execution directly and keeps slots occupied longer. +1. **Check for blocking calls in Workflow code.** A slot is held until the Workflow Task completes. Blocking I/O, heavy computation, or synchronous non-Temporal API calls inside a Workflow hold the slot far longer than expected. In the Python SDK, verify that no `async def` Workflow code is blocking the event loop. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Slots aren't released until the respond call succeeds. + +To recover immediately, scale out Workflow Worker pods or raise the maximum concurrent Workflow Task execution size in your Worker options. + +### ActivityWorker slots {/* #activityworker-slots */} + +1. **Check Activity execution latency.** Sustained high [`temporal_activity_execution_latency`](/references/sdk-metrics#activity_execution_latency) for the affected `activity_type` confirms Activities are holding slots longer than expected. +1. **Check Worker CPU.** High CPU slows Activity execution directly. +1. **Check what the Activities are waiting on.** If slots are exhausted because Activities are blocked on a downstream service, raising the slot count increases pressure on that service and makes the problem worse. Investigate the dependency before scaling concurrency. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). + +To recover immediately, scale out Activity Worker pods or raise the maximum concurrent Activity execution size. Confirm a downstream dependency isn't the real constraint first, or you will just push harder on something that is already struggling. + +### LocalActivityWorker slots {/* #localactivityworker-slots */} + +[Local Activities](/local-activity) run inside the Workflow Task execution loop, so a blocked slot holds up the whole Workflow Task. +The SDK keeps the Task alive by sending repeated Workflow Task heartbeats. +If that goes on past the Workflow Task heartbeat timeout, 30 minutes by default, the Temporal Service times the Task out and reschedules it, and every Local Activity in it runs again from the start. + +1. **Check what the Local Activities are waiting on.** The cause is almost always Local Activity code that is blocking and not returning. Check Worker logs for the affected `activity_type`. If it calls a downstream service, check whether that service is slow or throttling. +1. **Check Worker CPU.** +1. **Check the Local Activity latency alert.** See [Local Activity latency exceeds the heartbeat timeout](/troubleshooting/execution-failures#local-activity-latency-exceeds-the-heartbeat-timeout). If that is firing too, you are close to the heartbeat timeout or already past it, and re-execution has started. + +## All pollers disconnected {/* #all-pollers-disconnected */} + +**Metric:** [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) reaching 0, tagged `poller_type` and `task_queue` + +No active pollers remain for this poller type and Task Queue. +Workers have stopped polling entirely and Tasks are accumulating with nothing to process them. + +:::note + +This metric is tagged `poller_type`, not `worker_type`. +The values are `workflow_task`, `workflow_sticky_task`, `activity_task`, and `nexus_task`. +Alert on each poller type your Workers use, and note that a Worker with an active Sticky Execution cache reports `workflow_sticky_task` alongside `workflow_task`. + +::: + +**Why it matters.** +Workflow and Activity Tasks accumulate on the Temporal Service. +At scale this grows into a large backlog that puts pressure on Task matching and persistence. +Depending on your Workflow and Activity timeouts, Executions begin timing out while waiting for Tasks to be processed. + +**Triage.** + +1. **Check whether Worker processes are running.** Look at pod status, restart counts, and logs. A crash, an eviction, or an OOM kill is the most common cause by a wide margin. +1. **Rule out scale-down.** If you autoscale Workers or scale to zero on an idle Task Queue, a zero poller count is expected. Confirm against your scaling events before investigating further. +1. **Check whether slots are exhausted.** See [Worker Task slots exhausted](#worker-task-slots-exhausted) for the same Task Queue. Because the Go and Java SDKs block on slot acquisition before incrementing the poller gauge, exhausted slots drive this gauge to zero as a symptom. If both are firing, fix the slots first. +1. **Check for authentication failures.** Expired or revoked credentials are a common cause of pollers disconnecting. Check Worker logs for authentication errors and verify certificate or API key expiry. +1. **Check for INTERNAL errors.** See [UNIMPLEMENTED or INTERNAL from the Temporal Service](/troubleshooting/request-failures#unimplemented-or-internal-from-the-temporal-service). Sustained INTERNAL errors make Workers back off until they stop polling altogether. + +:::note Self-hosted Temporal Service + +Cross-check the server-side concurrent poller count for this Task Queue. +If it has also dropped, Workers have fully disconnected from the Service's perspective. +For authentication failures, check unauthorized request counts and authorization system failures. +A non-zero authorization system failure count means the auth plugin itself is broken, which is a bigger problem than an expired credential. + +::: + +## Task completions dropped to zero {/* #task-completions-dropped-to-zero */} + +**Metric:** [`temporal_request`](/references/sdk-metrics#request) rate reaching 0 on `RespondWorkflowTaskCompleted` or `RespondActivityTaskCompleted` + +The SDK only bumps this counter after a response succeeds, so a zero rate means nothing is landing. + +**Check for demand before you act on this one.** +A zero rate looks the same whether your Workers are broken or the Task Queue is simply quiet: outside business hours, or a batch workload between runs, zero is the correct answer. +The metric is tagged `namespace` and `operation`, not `task_queue`, so you can't narrow it to one Task Queue on its own either. + +Look for a non-zero Workflow or Activity start rate, a growing Task backlog, or schedule-to-start latency climbing. +If none of those are there, nothing is waiting to be completed and the metric is telling you the truth. + +**Why it matters.** +When there is demand and Workflow Task completions are still zero, your Workflow Executions aren't moving. +Signals, Updates, Timers, and Activity results accumulate in Event History with nothing processing them. + +No Activity Task completions means Workflows waiting on Activity results are stalled. +Activities start timing out against their `scheduleToClose` timeouts and the Temporal Service retries them within the Retry Policy, but with no Workers completing anything the retries just pile up alongside the originals. + +**Triage.** + +1. **Confirm there is demand.** Check the Workflow and Activity start rate, or the Task backlog on the Task Queue. If neither shows demand, the Task Queue is idle and no further action is needed. +1. **Check whether polling also stopped.** Look at [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for the relevant `poller_type`. If pollers are at zero too, your Workers are down entirely. Go to [All pollers disconnected](#all-pollers-disconnected). +1. **Check whether Workers are failing Tasks instead of completing them.** If polling is active but completions are zero, check [`temporal_workflow_task_execution_failed`](/references/sdk-metrics#workflow_task_execution_failed). Workers may be failing every Task. See [Non-determinism error](/troubleshooting/execution-failures#non-determinism-error) and [gRPC message too large](/troubleshooting/execution-failures#grpc-message-too-large). +1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). If every slot is occupied, no new Tasks are picked up or completed. +1. **Check for throttling on respond operations.** See [RESOURCE_EXHAUSTED on respond operations](/troubleshooting/request-failures#resource_exhausted-on-respond-operations). Sustained throttling suppresses this counter directly, since the SDK only counts responses that succeeded. +1. **Check schedule-to-start latency.** See [Workflow Task schedule-to-start latency elevated](#workflow-task-schedule-to-start-latency-elevated). If Tasks are not being dispatched, completions drop to zero even with healthy Workers. +1. **For Activity completions, check Workflow Task health first.** Activity completions drop to zero when no new Activities are being scheduled, which happens when Workflow Tasks are not completing. Rule out the Workflow Task side before investigating Activity Workers. + +:::note Self-hosted Temporal Service + +Check overall cluster health: service errors by Namespace, persistence availability, and resource-exhausted counts by cause. + +::: + +## Workflow Task schedule-to-start latency elevated {/* #workflow-task-schedule-to-start-latency-elevated */} + +**Metric:** [`temporal_workflow_task_schedule_to_start_latency`](/references/sdk-metrics#workflow_task_schedule_to_start_latency), tagged `namespace` and `task_queue` + +Workflow Tasks are sitting in the queue before a Worker picks them up. +Alert at two levels: a lower threshold that catches meaningful delay, and a much higher one that catches a stalled Task Queue. + +Two other pages give thresholds for this metric, for different purposes. +[Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_workflow_task_schedule_to_start_latency-spike) treats P95 above one second as worth a look. Use that number when you are tuning Worker capacity, not when you are deciding whether to page someone. +[Worker health](/cloud/worker-health#schedule-to-start-latency) supplies ready-made Prometheus queries for both this metric and the Activity equivalent. +The thresholds on this page are set higher because they are meant to page someone. + +**Why it matters.** +Schedule-to-start latency adds directly to Workflow end-to-end latency. +Every second a Task waits is a second added to how long your Workflows take. + +Past roughly the Workflow Task heartbeat timeout, Executions are effectively stalled rather than slow: anything waiting on a Workflow Task is making no progress at all. +At scale a large Task backlog accumulates, putting significant pressure on Task matching and the underlying database. +A backlog large enough can affect an entire self-hosted cluster, not just the affected Namespace and Task Queue. + +**Triage.** + +1. **Check Worker health.** Confirm Worker processes are running and not restarting. If they are down, bring them back before anything else. +1. **Check poller counts.** Check [`temporal_num_pollers`](/references/sdk-metrics#num_pollers) for `poller_type=workflow_task`. Fewer pollers means fewer Workers competing for Tasks, which raises this metric directly. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check Task slots.** See [Worker Task slots exhausted](#worker-task-slots-exhausted). With every slot occupied the SDK blocks before its next poll, so the effective poll rate drops. +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations). Throttled Workers back off and poll less often. +1. **Scale out if the backlog is real.** If Workers are healthy and slots are free but latency stays high, your Worker pool is too small for current throughput. Scale horizontally, and expect latency to stay high for a while afterward. The backlog you already built up has to drain first. + +:::note Self-hosted Temporal Service + +Check the approximate Task backlog in the Matching Service section of your server dashboard. +A growing backlog confirms Tasks are arriving faster than Workers pick them up. +Then check the server-side concurrent poller count for this Task Queue to see whether your Worker pool has shrunk. + +::: + +## Activity schedule-to-start latency elevated {/* #activity-schedule-to-start-latency-elevated */} + +**Metric:** [`temporal_activity_schedule_to_start_latency`](/references/sdk-metrics#activity_schedule_to_start_latency), tagged `namespace` and `task_queue` + +Activity Tasks are sitting in the queue before a Worker picks them up. + +As with the Workflow Task equivalent above, [Performance bottlenecks](/troubleshooting/performance-bottlenecks#temporal_activity_schedule_to_start_latency-spike) gives a lower investigation threshold for this metric, and [Worker health](/cloud/worker-health#schedule-to-start-latency) supplies the Prometheus query. + +**Why it matters.** +Any Workflow Execution waiting on an Activity result is stalled for the duration of this latency. +At scale, an Activity Task backlog accumulates and puts pressure on Task matching and the underlying database. + +**Triage.** + +1. **Check Worker health.** Confirm Activity Worker processes are running and not restarting. +1. **Check poller counts** for `poller_type=activity_task`. See [All pollers disconnected](#all-pollers-disconnected). +1. **Check Task slots** for `worker_type=ActivityWorker`. See [Worker Task slots exhausted](#worker-task-slots-exhausted). +1. **Check for throttling on poll operations.** See [RESOURCE_EXHAUSTED on poll operations](/troubleshooting/request-failures#resource_exhausted-on-poll-operations). +1. **Check the Activity failure rate.** See [Activity execution failures elevated](/troubleshooting/execution-failures#activity-execution-failures-elevated). Lots of failures means lots of retry Tasks, which grow the backlog and hold this latency up even after you have restored Worker capacity. +1. **Check for Task Queue rate limits.** A low `TaskQueueActivitiesPerSecond` caps how fast Activities are dispatched regardless of Worker capacity. See [Performance bottlenecks](/troubleshooting/performance-bottlenecks). + +:::note Self-hosted Temporal Service + +Check the approximate Task backlog and the server-side concurrent poller count for this Task Queue, as in the Workflow Task section above. + +::: + +## Sticky cache holding zero entries under load {/* #sticky-cache-holding-zero-entries-under-load */} + +**Metric:** [`temporal_sticky_cache_size`](/references/sdk-metrics#sticky_cache_size) reaching 0 while the Worker is processing Workflow Tasks + +This gauge tells you how many Workflow Executions the Worker is holding in its [Sticky Execution](/sticky-execution) cache right now. +It is not the configured cache size. + +**Zero is normal on an idle Worker.** +A Worker that isn't processing Workflow Tasks has nothing to cache, and a low-volume Task Queue can sit at zero for hours. +Alert on this gauge by itself and you will get paged for Workers that are doing nothing wrong. + +What you actually want to catch is zero occupancy while the Worker is working. +Pair the gauge with a non-zero Workflow Task rate on the same Worker and give it a long `for` duration. +High [`temporal_workflow_task_replay_latency`](/references/sdk-metrics#workflow_task_replay_latency) or a high [`temporal_sticky_cache_miss`](/references/sdk-metrics#sticky_cache_miss) rate will confirm it: both mean Workflow Tasks are replaying from history instead of picking up where they left off. + +Tag coverage is patchy, so there is a limit to how tightly you can scope this alert. Only the TypeScript and Java SDKs emit `namespace`, and only TypeScript emits `task_queue`. + +**Why it matters.** +With nothing in the cache, every Workflow Task pulls all the History pages from the Temporal Service and re-runs every command from the beginning. +Each one costs you what a cache miss costs. + +At any real scale that means sustained load on the Temporal Service and higher Workflow Task execution latency for every Execution on the Worker. +See [Workflow Task execution latency high](/troubleshooting/execution-failures#workflow-task-execution-latency-high). + +**Triage.** + +1. **Confirm the Worker is actually processing Workflow Tasks.** If the Workflow Task rate is zero, the Worker is idle and there is nothing to fix. +1. **Check whether the cache is disabled by configuration.** A cache size of zero in Worker options means no Execution is ever cached: + - **Go SDK:** `worker.SetStickyWorkflowCacheSize(int)` sets the size, defaulting to 10,000. Passing `0` turns the cache off completely, so restore a non-zero value. Call it before any Worker starts; the cache is shared across every Worker in the process. + - **Java SDK:** `WorkerFactoryOptions.Builder.setWorkflowCacheSize(int)` sets the size, defaulting to 600. Passing `0` resets it to the default instead of disabling it, and a negative value is rejected outright, so you can't reach a zero-size cache this way. Check `setMaxWorkflowThreadCount(int)` instead. A thread pool set too low starves Workflow execution and the cache never gets used properly. +1. **Check the eviction rate.** If the cache is configured correctly but occupancy stays at zero, entries are being evicted as fast as they go in. Check [`temporal_sticky_cache_total_forced_eviction`](/references/sdk-metrics#sticky_cache_total_forced_eviction). A high rate points at either a cache too small for the number of concurrent Executions on that Worker, or repeated Workflow Task timeouts forcing evictions. +1. **Verify the fix.** After redeploying, `temporal_sticky_cache_size` should climb from zero under load, and [`temporal_workflow_task_execution_latency`](/references/sdk-metrics#workflow_task_execution_latency) should fall as cold replays are replaced by cache hits. + +:::note Self-hosted Temporal Service + +Check persistence latency filtered to `GetWorkflowExecution`. Sustained high latency there confirms the Temporal Service is straining under all the repeated History reads. + +::: diff --git a/sidebars.js b/sidebars.js index 6e7f9a9e92..7f751809b1 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1704,6 +1704,9 @@ module.exports = { 'troubleshooting/last-connection-error', 'troubleshooting/performance-bottlenecks', 'troubleshooting/schedule-missed-actions', + 'troubleshooting/request-failures', + 'troubleshooting/worker-capacity', + 'troubleshooting/execution-failures', { type: 'category', label: 'Serverless Workers', @@ -1729,6 +1732,7 @@ module.exports = { }, items: [ 'best-practices/worker', + 'best-practices/worker-alerting', 'best-practices/pre-production-testing', 'production-deployment/multi-tenant-patterns', 'best-practices/managing-namespace',