From 28ff50c76621351758910be28314e3ae6102232a Mon Sep 17 00:00:00 2001 From: Jwahir Sundai Date: Wed, 5 Aug 2026 17:14:47 -0500 Subject: [PATCH 1/3] break out worker versioning section --- docs/best-practices/worker.mdx | 2 +- .../kubernetes-controller.mdx | 2 +- .../recover-pinned-workflows.mdx | 4 +- .../worker-deployments/worker-versioning.mdx | 1079 ----------------- .../worker-versioning/configure-worker.mdx | 264 ++++ .../worker-versioning/index.mdx | 212 ++++ .../worker-versioning/roll-out-and-pin.mdx | 232 ++++ .../worker-versioning/sunset-and-gc.mdx | 179 +++ .../upgrade-on-continue-as-new.mdx | 266 ++++ sidebars.js | 20 +- 10 files changed, 1174 insertions(+), 1086 deletions(-) delete mode 100644 docs/production-deployment/worker-deployments/worker-versioning.mdx create mode 100644 docs/production-deployment/worker-deployments/worker-versioning/configure-worker.mdx create mode 100644 docs/production-deployment/worker-deployments/worker-versioning/index.mdx create mode 100644 docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx create mode 100644 docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx create mode 100644 docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx index 128f469712..9a4d7f9787 100644 --- a/docs/best-practices/worker.mdx +++ b/docs/best-practices/worker.mdx @@ -174,7 +174,7 @@ safer rollouts, clearer rollback behavior, and a cleaner operational model than Workflow definition. To learn more about versioning Workflows, see the -[Workflow Versioning](/production-deployment/worker-deployments/worker-versioning.mdx) guide or take our +[Worker Versioning](/production-deployment/worker-deployments/worker-versioning) guide or take our [Worker Versioning course](https://learn.temporal.io/courses/worker_versioning/). :::tip diff --git a/docs/production-deployment/worker-deployments/kubernetes-controller.mdx b/docs/production-deployment/worker-deployments/kubernetes-controller.mdx index afad15ffa1..97408df898 100644 --- a/docs/production-deployment/worker-deployments/kubernetes-controller.mdx +++ b/docs/production-deployment/worker-deployments/kubernetes-controller.mdx @@ -31,7 +31,7 @@ Over time these checks can become a source of technical debt, as safely removing Using pinning, you will not need to add branching to your Workflows to avoid non-determinism errors. This allows you to bypass the other Versioning APIs. -The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the [Temporal CLI](/production-deployment/worker-deployments/worker-versioning#rolling-out-changes-with-the-cli). +The Worker Controller gives you direct, programmatic control over your Worker deployments, and integrates with the [Temporal CLI](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin#rolling-out-changes-with-the-cli). You do not need to use the Worker Controller to use Worker Versioning, but when used together, Worker Versioning and the Worker Controller can provide more graceful deployments and upgrades, and less need to manually tune your Workers. Note that in Temporal, **Worker Deployment** is sometimes referred to as **Deployment**, but since the Worker Controller makes significant references to Kubernetes Deployment resource, within this page we will stick to these terms: diff --git a/docs/production-deployment/worker-deployments/recover-pinned-workflows.mdx b/docs/production-deployment/worker-deployments/recover-pinned-workflows.mdx index b3f740eb58..01125aadc4 100644 --- a/docs/production-deployment/worker-deployments/recover-pinned-workflows.mdx +++ b/docs/production-deployment/worker-deployments/recover-pinned-workflows.mdx @@ -125,7 +125,7 @@ For Workflows still retrying without state corruption, you may need to use the [ ## Recover Workflows -Temporal exposes two recovery primitives, both available through the CLI or directly through the Worker Versioning APIs (see [Moving a pinned Workflow](/production-deployment/worker-deployments/worker-versioning#moving-a-pinned-workflow)): +Temporal exposes two recovery primitives, both available through the CLI or directly through the Worker Versioning APIs (see [Moving a pinned Workflow](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin#moving-a-pinned-workflow)): - **Versioning Override** — forces the next retried Workflow Task to execute on a different pinned Version. Use [`temporal workflow update-options`](/cli/command-reference/workflow#update-options). - **Reset-with-Move** — atomically resets a Workflow's Event History and applies a Versioning Override. Use [`temporal workflow reset with-workflow-update-options`](/cli/command-reference/workflow#with-workflow-update-options). @@ -234,7 +234,7 @@ Recommended approach: 1. Repeat the idempotent recovery commands on `Running` Workflows until the drainage status reports `drained`. The Temporal Service refreshes drainage status periodically, so it may take a few minutes after the last running Workflow closes. 2. Once the Version is drained, run the non-idempotent Reset-with-Move command against closed Workflows once. -See [Sunsetting an old Deployment Version](/production-deployment/worker-deployments/worker-versioning#sunsetting-an-old-deployment-version) for more on drainage states. +See [Sunsetting an old Deployment Version](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc#sunsetting-an-old-deployment-version) for more on drainage states. ## Clean up the drained Version diff --git a/docs/production-deployment/worker-deployments/worker-versioning.mdx b/docs/production-deployment/worker-deployments/worker-versioning.mdx deleted file mode 100644 index 3601f900ff..0000000000 --- a/docs/production-deployment/worker-deployments/worker-versioning.mdx +++ /dev/null @@ -1,1079 +0,0 @@ ---- -id: worker-versioning -title: Worker Versioning -sidebar_label: Worker Versioning -description: - Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for - patching to support multiple code paths. -slug: /production-deployment/worker-deployments/worker-versioning -toc_max_heading_level: 4 -keywords: - - scaling - - workers - - versioning - - deploys -tags: - - Temporal Service - - Durable Execution ---- - -import { SdkTabs, AnnotatedCode } from '@site/src/components'; - -Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on -your Workers without breaking them. Temporal enables this by helping you manage different builds or versions, formally -called [Worker Deployment Versions](/worker-versioning#deployment-versions). - -For most teams, Worker Versioning should be the default recommendation for deploying Workflow code changes in -production. If you can run versioned worker deployments, prefer Worker Versioning over patching. - -Worker Versioning unlocks important benefits for users of [blue-green or rainbow deployments](#deployment-systems). - -- Ramping traffic gradually to a new Worker Deployment Version. -- Verifying a new Deployment Version with tests before sending production traffic to it. -- Instant rollback when you detect that a new Deployment Version is broken. -- Improved error rates when adopting it. - -In addition, Worker Versioning introduces **Workflow Pinning**. For pinned Workflow Types, each execution runs entirely -on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running, -pinned Workflows. - -To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems). - -:::tip - -Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo. - -
- -
- -::: - -:::note - -Minimum versions: - -- Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0) -- Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0) -- Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0) -- Typescript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0) -- .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0) -- Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0) - -Self-hosted users: - -- Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1) -- Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1) -- Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) - -::: - -## Getting Started with Worker Versioning {/* #definition */} - -To get started with Worker Versioning, you should understand some concepts around versioning and deployments. - -- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two - active Deployment Versions can run at once. -- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on - multiple Task Queues, but they all run the same build. -- A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version. -- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's - Worker Deployment controls how the task queue matches Workflow Tasks with Workers. -- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or - Auto-Upgrade. - - A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. - - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its - Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single - Deployment Version and need to be kept replay-safe manually, i.e. with - [patching](/workflow-definition#workflow-versioning). - - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their - Worker Deployment. - - Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](#deployment-systems). - - Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete. - - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark - your Workflow Type as Auto-Upgrade. -- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where - Workflows are routed to unless they were previously pinned on a different version. -- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a - configurable percentage of Workflows are routed to unless they were previously pinned on a different version. -- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the - version it will move to next. - -## Setting up your deployment system {/* #deployment-systems */} - -If you haven't already, you'll want to pick a container deployment solution for your Workers. - -You also need to pick among three common deployment strategies: - -- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and - only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower - availability than the other strategies and are incompatible with Worker Versioning. -- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can - control how traffic is routed between them. This allows you to maximize your uptime with features like instant - rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need. -- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy - new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know - when all the Workflows of a given version are drained so that you can sunset it. - -:::note - -You also have the option to use the -[Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable -rainbow deployments of your Workers if you're using Kubernetes. - -::: - -If you cannot yet support blue-green or rainbow style deployments, use [patching](/patching) as a fallback while you -work toward a versioned deployment model. - -## Configuring a Worker for Versioning - -You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. There are -three new parameters, with different names depending on the language: - -- `UseVersioning`: This enables the Versioning functionality for this Worker. -- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a - deployment name and a build ID number. -- (Optional) The [Default Versioning Behavior](#definition). If unset, you'll be required to specify the behavior on - each Workflow. Or you can default to Pinned or Auto-Upgrade. - -Follow the example for your SDK below. Select a concept to highlight the matching lines: - - - - -```go -buildID := mustGetEnv("MY_BUILD_ID") -w := worker.New(c, myTaskQueue, worker.Options{ - DeploymentOptions: worker.DeploymentOptions{ - UseVersioning: true, - Version: worker.WorkerDeploymentVersion{ - DeploymentName: "llm_srv", - BuildID: buildID, - }, - DefaultVersioningBehavior: workflow.VersioningBehaviorUnspecified, - }, -}) -``` - - - - -```java -import io.temporal.worker.WorkerOptions; -import io.temporal.common.VersioningBehavior; -import io.temporal.common.WorkerDeploymentVersion; -import io.temporal.worker.WorkerDeploymentOptions; - -WorkerOptions.newBuilder() - .setDeploymentOptions( - WorkerDeploymentOptions.newBuilder() - .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0")) - .setUseVersioning(true) - .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) - .build()) - .build(); -``` - - - - -```python -from temporalio.common import WorkerDeploymentVersion, VersioningBehavior -from temporalio.worker import Worker, WorkerDeploymentConfig - -Worker( - client, - task_queue="mytaskqueue", - workflows=workflows, - activities=activities, - deployment_config=WorkerDeploymentConfig( - version=WorkerDeploymentVersion( - deployment_name="llm_srv", - build_id=my_env.build_id), - use_worker_versioning=True, - default_versioning_behavior=VersioningBehavior.UNSPECIFIED - ), -) -``` - - - - -```ts -const myWorker = await Worker.create({ - workflowsPath: require.resolve('./workflows'), - taskQueue, - workerDeploymentOptions: { - useWorkerVersioning: true, - version: { buildId: '1.0', deploymentName: 'llm_srv' }, - }, - connection: nativeConnection, -}); -``` - - - - -```csharp -var myWorker = new TemporalWorker( - Client, - new TemporalWorkerOptions(taskQueue) - { - DeploymentOptions = new(new("llm_srv", "1.0"), true) - { - DefaultVersioningBehavior = VersioningBehavior.Unspecified, - }, - }.AddWorkflow()); -``` - - - - -```ruby -worker = Temporalio::Worker.new( - client: client, - task_queue: task_queue, - workflows: [MyWorkflow], - deployment_options: Temporalio::Worker::DeploymentOptions.new( - version: Temporalio::WorkerDeploymentVersion.new( - deployment_name: 'llm_srv', - build_id: '1.0' - ), - use_worker_versioning: true, - default_versioning_behavior: Temporalio::VersioningBehavior::UNSPECIFIED - ) -) -``` - - - - -## Choosing a Versioning Behavior {/* #choosing-behavior */} - -The right versioning behavior depends on how long your Workflows run relative to your deployment frequency. - -### Decision guide {/* #decision-guide */} - -| Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | -| ---------------------------------------- | --------------------- | -------------------------------------------------------- | ------------------ | -| **Short** (completes before next deploy) | N/A | `PINNED` | Never | -| **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | -| **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](#upgrade-on-continue-as-new) | Never | -| **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | - -### Examples by Workflow type {/* #behavior-examples */} - -| Workflow Type | Duration | Recommended Behavior | Notes | -| -------------------- | ------------ | -------------------------- | ----------------------------------- | -| Order processing | Minutes | `PINNED` | Completes before next deploy | -| Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | -| Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | -| Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | -| AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | -| Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | - -:::info Long-running Workflows with Continue-as-New - -If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the -CaN boundary without patching. See [Upgrading on Continue-as-New](#upgrade-on-continue-as-new) below. - -::: - -### Default Versioning Behavior Considerations - -If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. -Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. - -In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be -short-running for the foreseeable future, you can default to Pinned. - -Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to -annotate their Workflows. This default is the most similar to the legacy behavior. Once each Workflow Type is annotated, -you can remove the `DefaultVersioningBehavior`. - -There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of -the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks -or slowing down. - -For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, -you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the -Current Version are blocking the queue. - -:::note - -Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not -Unspecified. - -::: - -You also want to make sure you understand how your Activities are going to work across different Worker Deployment -Versions. Refer to the [Worker Versioning Activity behavior docs](/worker-versioning#activity-behavior-across-versions) -for more details. - -If you run [Serverless Workers](/serverless-workers), each Worker Deployment Version's compute provider points at a -specific Lambda function. In production, configure it with a qualified versioned ARN so Pinned Workflows remain truly -pinned to the code they started against, regardless of whether your default behavior is Pinned or Auto-Upgrade. See -[Worker Versioning with Serverless Workers](/serverless-workers#worker-versioning-with-serverless-workers). - -## Rolling out changes with the CLI - -Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use -the `temporal` CLI to check which of your Worker versions are currently polling: - -You can view the Versions that are part of a Deployment with `temporal worker deployment describe`: - -```bash -temporal worker deployment describe --name="$MY_DEPLOYMENT" -``` - -To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name -and a Build ID: - -```bash -temporal worker deployment set-current-version \ - --deployment-name "YourDeploymentName" \ - --build-id "YourBuildID" -``` - -To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the -same parameters and a ramping percentage: - -```bash -temporal worker deployment set-ramping-version \ - --deployment-name "YourDeploymentName" \ - --build-id "YourBuildID" \ - --percentage=5 -``` - -You can verify that Workflows are cutting over to that version with `describe -w YourWorkflowID`: - -```bash -temporal workflow describe -w YourWorkflowID -``` - -That returns the new Version that the workflow is running on: - -``` -Versioning Info: - - Behavior AutoUpgrade - Version llm_srv.2.0 - OverrideBehavior Unspecified -``` - -## Marking a Workflow Type as Pinned - -You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. This will cause it -to remain on its original deployed version: - - - -```go -// w is the Worker configured as in the previous example -w.RegisterWorkflowWithOptions(HelloWorld, workflow.RegisterOptions{ - // or workflow.VersioningBehaviorAutoUpgrade - VersioningBehavior: workflow.VersioningBehaviorPinned, -}) -``` - - -```java -@WorkflowInterface -public interface HelloWorld { - @WorkflowMethod - String hello(); -} - -public static class HelloWorldImpl implements HelloWorld { - @Override - @WorkflowVersioningBehavior(VersioningBehavior.PINNED) - public String hello() { - return "Hello, World!"; - } -} - -``` - - -```python -@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) -class HelloWorld: - @workflow.run - async def run(self): - return "hello world!" - -``` - - - - -```ts -setWorkflowOptions({ versioningBehavior: 'PINNED' }, helloWorld); -export async function helloWorld(): Promise { - return 'hello world!'; -} -``` - - - - -```csharp -[Workflow(VersioningBehavior = VersioningBehavior.Pinned)] -public class HelloWorld -{ - [WorkflowRun] - public async Task RunAsync() - { - return "hello world!"; - } -} -``` - - - - -```ruby -class HelloWorld < Temporalio::Workflow::Definition - workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED - - def execute - 'hello world!' - end -end -``` - - - - -## Moving a pinned Workflow - -Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the -fix. - -If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`: - -```bash -temporal workflow update-options \ - --workflow-id "$WORKFLOW_ID" \ - --versioning-override-behavior pinned \ - --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ - --versioning-override-build-id "$TARGET_BUILD_ID" -``` - -You can move several Workflows at once matching a `--query` parameter: - -```bash -temporal workflow update-options \ - --query="TemporalWorkerDeploymentVersion=$TARGET_DEPLOYMENT:$BAD_BUILD_ID" \ - --versioning-override-behavior pinned \ - --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ - --versioning-override-build-id "$FIXED_BUILD_ID" -``` - -In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to -patch your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. If you -made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you want to -roll back to an earlier version, it's not possible to patch it. Considering using -[Workflow Reset](/workflow-execution/event#reset) along with your move. - -"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset -Workflow, so when it resumes execution, all new Workflow Tasks will be executed on your new Worker. - -```bash -temporal workflow reset with-workflow-update-options \ - --workflow-id "$WORKFLOW_ID" \ - --event-id "$EVENT_ID" \ - --reason "$REASON" \ - --versioning-override-behavior pinned \ - --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ - --versioning-override-build-id "$TARGET_BUILD_ID" -``` - -For a complete runbook covering batch recovery, drainage handling, and how to choose between Versioning Override and Reset-with-Move based on each Workflow's state, see [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows). - -## Migrating a Workflow from Pinned to Auto-Upgrade - -There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your -Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake. - -Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a -new build, but that would just push the problem to the next build. - -In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. -You can use `temporal workflow update-options` for this: - -```bash -temporal workflow update-options \ - --workflow-id "$WORKFLOW_ID" \ - --versioning-override-behavior auto_upgrade -``` - -If you want to move all your Workflows of a certain type to this new configuration, you can do it with this command: - -```bash -temporal workflow update-options \ - --query="WorkflowType='$WORKFLOW_TYPE'" \ - --versioning-override-behavior auto_upgrade -``` - -You can also filter on a certain build ID to limit the number of Workflows you apply it to: - -```bash -temporal workflow update-options \ - --query="WorkflowType='$WORKFLOW_TYPE' AND TemporalWorkerDeploymentVersion='$TARGET_DEPLOYMENT:$OLD_VERSION'" \ - --versioning-override-behavior auto_upgrade -``` - -:::note - -When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code. - -::: - -## Upgrading on Continue-as-New {/* #upgrade-on-continue-as-new */} - -Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker -Deployment Versions at Continue-as-New boundaries without requiring patching. - -This pattern is ideal for: - -- **Entity Workflows** that run for months or years -- **Batch processing** Workflows that checkpoint with Continue-as-New -- **AI agent Workflows** with long sleeps waiting for user input - -:::note Public Preview - -This feature is in Public Preview as an experimental SDK-level option. - -::: - -### How it works {/* #upgrade-on-can-how-it-works */} - -By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. With the -upgrade option enabled: - -1. Each Workflow run remains pinned to its version (no patching needed during a run) -2. The Temporal Server tells the workflow when a new [Target Version](/worker-versioning#versioning-definitions) becomes available -3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the [Target Version](/worker-versioning#versioning-definitions) - -### Checking for new versions {/* #checking-for-new-versions */} - -When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through -`target_worker_deployment_version_changed`: - - - - -```go -func (w *Workflows) ContinueAsNewWithVersionUpgradeV1( - ctx workflow.Context, - attempt int, -) (string, error) { - if attempt > 0 { - return "v1.0", nil - } - - // Check GetTargetWorkerDeploymentVersionChanged periodically. - // GetTargetWorkerDeploymentVersionChanged is refreshed after each WFT completes. - for { - // Trigger a WFT when timer expires, thereby refreshing the GetTargetWorkerDeploymentVersionChanged flag. - // Since this is just a test workflow, we aren't doing any real work. In a real workflow regularly - // doing non-sleep workflow tasks, you would not need to artificially trigger a WFT to refresh the - // GetTargetWorkerDeploymentVersionChanged flag. You could choose to check the field periodically, or you - // might want to check before accepting updates, starting activities, or starting child workflows. - err := workflow.Sleep(ctx, 10*time.Millisecond) - if err != nil { - return "", err - } - info := workflow.GetInfo(ctx) - if info.GetTargetWorkerDeploymentVersionChanged() { - return "", workflow.NewContinueAsNewErrorWithOptions( - ctx, - workflow.ContinueAsNewErrorOptions{ - // Pass InitialVersioningBehavior=workflow.ContinueAsNewVersioningBehaviorAutoUpgrade - // to make the new run start with AutoUpgrade behavior and use the Target Version of - // its Worker Deployment. - InitialVersioningBehavior: workflow.ContinueAsNewVersioningBehaviorAutoUpgrade, - }, - "ContinueAsNewWithVersionUpgrade", - attempt+1, - ) - } - } -} - -func (w *Workflows) ContinueAsNewWithVersionUpgradeV2( - ctx workflow.Context, - attempt int, -) (string, error) { - return "v2.0", nil -} -``` - - - - -```java -public class ContinueAsNewWithVersionUpgradeImpl implements ContinueAsNewWithVersionUpgrade { - @Override - public String run(int attempt) { - if (attempt > 0) { - return "v1.0"; - } - - // isTargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. - // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial - // timer; you could check the flag periodically, or before accepting Updates, starting - // Activities, or starting child Workflows. - while (true) { - Workflow.sleep(Duration.ofMillis(10)); - if (Workflow.getInfo().isTargetWorkerDeploymentVersionChanged()) { - // Set InitialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade - // behavior and uses the Target Version of its Worker Deployment. - Workflow.continueAsNew( - ContinueAsNewOptions.newBuilder() - .setInitialVersioningBehavior(InitialVersioningBehavior.AUTO_UPGRADE) - .build(), - attempt + 1); - } - } - } -} -``` - - - - -```python -@workflow.defn -class ContinueAsNewWithVersionUpgrade: - @workflow.run - async def run(self, attempt: int) -> str: - if attempt > 0: - return "v1.0" - - # is_target_worker_deployment_version_changed() is refreshed after each Workflow Task - # completes. In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need - # an artificial timer; you could check the flag periodically, or before accepting Updates, - # starting Activities, or starting child Workflows. - while True: - await workflow.sleep(timedelta(milliseconds=10)) - if workflow.info().is_target_worker_deployment_version_changed(): - # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with - # AutoUpgrade behavior and uses the Target Version of its Worker Deployment. - workflow.continue_as_new( - attempt + 1, - initial_versioning_behavior=ContinueAsNewVersioningBehavior.AUTO_UPGRADE, - ) -``` - - - - -```ts -import * as wf from '@temporalio/workflow'; -import { InitialVersioningBehavior } from '@temporalio/common'; - -export async function continueAsNewWithVersionUpgrade(attempt: number): Promise { - if (attempt > 0) { - return 'v1.0'; - } - - // targetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. - // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial - // timer; you could check the flag periodically, or before accepting Updates, starting - // Activities, or starting child Workflows. - for (;;) { - await wf.sleep('10ms'); - if (wf.workflowInfo().targetWorkerDeploymentVersionChanged) { - // Set initialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade - // behavior and uses the Target Version of its Worker Deployment. - return await wf.makeContinueAsNewFunc({ - initialVersioningBehavior: InitialVersioningBehavior.AUTO_UPGRADE, - })(attempt + 1); - } - } -} -``` - - - - -```csharp -[Workflow] -public class ContinueAsNewWithVersionUpgrade -{ - [WorkflowRun] - public async Task RunAsync(int attempt) - { - if (attempt > 0) - { - return "v1.0"; - } - - // TargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. - // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an - // artificial timer; you could check the flag periodically, or before accepting Updates, - // starting Activities, or starting child Workflows. - while (true) - { - await Workflow.DelayAsync(TimeSpan.FromMilliseconds(10)); - if (Workflow.TargetWorkerDeploymentVersionChanged) - { - // Set InitialVersioningBehavior to AutoUpgrade so the new run starts with - // AutoUpgrade behavior and uses the Target Version of its Worker Deployment. - throw Workflow.CreateContinueAsNewException( - (ContinueAsNewWithVersionUpgrade wf) => wf.RunAsync(attempt + 1), - new ContinueAsNewOptions - { - InitialVersioningBehavior = InitialVersioningBehavior.AutoUpgrade, - }); - } - } - } -} -``` - - - - -```ruby -class ContinueAsNewWithVersionUpgrade < Temporalio::Workflow::Definition - workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED - - def execute(attempt) - return 'v1.0' if attempt.positive? - - # target_worker_deployment_version_changed? is refreshed after each Workflow Task completes. - # In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial - # timer; you could check the flag periodically, or before accepting Updates, starting - # Activities, or starting child Workflows. - loop do - Temporalio::Workflow.sleep(0.01) - next unless Temporalio::Workflow.target_worker_deployment_version_changed? - - # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with AutoUpgrade - # behavior and uses the Target Version of its Worker Deployment. - raise Temporalio::Workflow::ContinueAsNewError.new( - attempt + 1, - initial_versioning_behavior: Temporalio::ContinueAsNewVersioningBehavior::AUTO_UPGRADE - ) - end - end -end -``` - - - - -### Limitations {/* #upgrade-on-can-limitations */} - -:::caution Current Limitations - -- **Lazy moving only:** Workflows must execute a step to receive the target-version-changed information. Sleeping - Workflows won't proactively get it. If you have idle Workflows that you want to wake up so they can check the - target-version-changed flag, you can send them a Signal. -- **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input provided by the - previous version's workflow definition is compatible with the new version's workflow definition. If incompatible, the - new run may fail on its first Workflow Task. - -::: - -## Sunsetting an old Deployment Version - -A Worker Deployment Version moves through the following states: - -1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never - becomes Active, it will never be Draining or Drained. -2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade - Workflows. -3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is - possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated - periodically. -4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. - -You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each -`VersionSummary`, or by describing the version directly. When a version is Draining or Drained, that is displayed in a -value called `DrainageStatus`. Periodically, the Temporal Service will refresh this status by counting any open pinned -Workflows using that version. - -On each refresh, `DrainageInfo.last_checked_time` is updated. Eventually, `DrainageInfo` will report that the version is -fully drained. At this point, no Workflows are still running on that version and no more will be automatically routed to -it, so you can consider shutting down the running Workers. - -You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with -`temporal worker deployment describe-version`: - -```bash -temporal worker deployment describe-version \ - --deployment-name "YourDeploymentName" \ - --build-id "YourBuildID" -``` - -``` -Worker Deployment Version: - Version llm_srv.1.0 - CreateTime 5 hours ago - RoutingChangedTime 32 seconds ago - RampPercentage 0 - DrainageStatus draining - DrainageLastChangedTime 31 seconds ago - DrainageLastCheckedTime 31 seconds ago - -Task Queues: - Name Type - hello-world activity - hello-world workflow -``` - -If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep -some Workers running to handle them. - -### Adding a pre-deployment test - -Before deploying a new Workflow revision, you can test it with synthetic traffic. - -To do this, use pinning in your tests, following the examples below - - - -```go -workflowOptions := client.StartWorkflowOptions{ - ID: "MyWorkflowId", - TaskQueue: "MyTaskQueue", - VersioningOverride: &client.PinnedVersioningOverride{ - Version: worker.WorkerDeploymentVersion{ - DeploymentName: "DeployName", - BuildID: "1.0", - }, - }, -} -// c is an initialized Client -we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, HelloWorld, "Hello") -``` - - -```java -MyWorkflow handle = client.newWorkflowStub( - MyWorkflow.class, - WorkflowOptions.newBuilder() - .setWorkflowId("MyWorkflowId") - .setTaskQueue("MyTaskQueue") - .setVersioningOverride(new VersioningOverride.PinnedVersioningOverride( - new WorkerDeploymentVersion("DeployName", "1.0"))) - .build() -); -WorkflowExecution we = WorkflowClient.start(handle::execute, "Hello"); -``` - - -```python -handle = client.start_workflow( - MyWorkflow.run, - "Hello", - id="MyWorkflowId", - task_queue="MyTaskQueue", - versioning_override=PinnedVersioningOverride( - WorkerDeploymentVersion("DeployName", "1.0") - ), -) -``` - - -```ts -const handle = await client.workflow.start('helloWorld', { - taskQueue: 'MyTaskQueue', - workflowId: 'MyWorkflowId', - versioningOverride: { - pinnedTo: { buildId: '1.0', deploymentName: 'deploy-name' }, - }, -}); -``` - - -```csharp -var workerV1 = new WorkerDeploymentVersion("deploy-name", "1.0"); -var handle = await Client.StartWorkflowAsync( - (HelloWorld wf) => wf.RunAsync(), - new(id: "MyWorkflowId", taskQueue: "MyTaskQueue") - { - VersioningOverride = new VersioningOverride.Pinned(workerV1), - } -); -``` - - -```ruby -worker_v1 = Temporalio::WorkerDeploymentVersion.new( - deployment_name: 'deploy-name', - build_id: '1.0' -) -handle = env.client.start_workflow( - HelloWorld, - id: 'MyWorkflowId', - task_queue: 'MyTaskQueue', - versioning_override: Temporalio::VersioningOverride.pinned(worker_v1) -) -``` - - - -## Garbage collection - -Worker Deployments are never garbage collected, but _Worker Deployment Versions_ (often referred to as Versions, Worker -Versions, Deployment Versions) are. - -Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to -[`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L1501-L1505), -which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed. - -For example, when you deploy your 101st Worker Version in a Worker Deployment, the server looks at the oldest drained -version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version -still has pollers, the server will try the next oldest version. If none of the 100 versions are eligible for deletion -(ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version -would fail. - -At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` -or stop polling from one of the old drained versions to make it eligible for clean up. - -If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and -Deployment Name as the deleted version and the server will recreate it. - -This covers the complete lifecycle of working with Worker Versioning. We are continuing to improve this feature, and we -welcome any feedback or feature requests using the sidebar link! diff --git a/docs/production-deployment/worker-deployments/worker-versioning/configure-worker.mdx b/docs/production-deployment/worker-deployments/worker-versioning/configure-worker.mdx new file mode 100644 index 0000000000..67074bbc3f --- /dev/null +++ b/docs/production-deployment/worker-deployments/worker-versioning/configure-worker.mdx @@ -0,0 +1,264 @@ +--- +id: configure-worker +title: Configure a Worker for Versioning +sidebar_label: Configure a Worker +description: Enable Worker Versioning on your Workers with UseVersioning, Deployment Version, and default Versioning Behavior. +toc_max_heading_level: 4 +keywords: + - workers + - versioning + - deploys +tags: + - Temporal Service + - Durable Execution +--- + +import { SdkTabs, AnnotatedCode } from '@site/src/components'; + +You'll need to add a few additional configuration parameters to your Workers to toggle on Worker Versioning. There are +three new parameters, with different names depending on the language: + +- `UseVersioning`: This enables the Versioning functionality for this Worker. +- A `Version` to identify the revision that this Worker will be allowed to execute. This is a combination of a + deployment name and a build ID number. +- (Optional) The [Default Versioning Behavior](/production-deployment/worker-deployments/worker-versioning#definition). If unset, you'll be required to specify the behavior on + each Workflow. Or you can default to Pinned or Auto-Upgrade. + +Follow the example for your SDK below. Select a concept to highlight the matching lines: + + + + +```go +buildID := mustGetEnv("MY_BUILD_ID") +w := worker.New(c, myTaskQueue, worker.Options{ + DeploymentOptions: worker.DeploymentOptions{ + UseVersioning: true, + Version: worker.WorkerDeploymentVersion{ + DeploymentName: "llm_srv", + BuildID: buildID, + }, + DefaultVersioningBehavior: workflow.VersioningBehaviorUnspecified, + }, +}) +``` + + + + +```java +import io.temporal.worker.WorkerOptions; +import io.temporal.common.VersioningBehavior; +import io.temporal.common.WorkerDeploymentVersion; +import io.temporal.worker.WorkerDeploymentOptions; + +WorkerOptions.newBuilder() + .setDeploymentOptions( + WorkerDeploymentOptions.newBuilder() + .setVersion(new WorkerDeploymentVersion("llm_srv", "1.0")) + .setUseVersioning(true) + .setDefaultVersioningBehavior(VersioningBehavior.AUTO_UPGRADE) + .build()) + .build(); +``` + + + + +```python +from temporalio.common import WorkerDeploymentVersion, VersioningBehavior +from temporalio.worker import Worker, WorkerDeploymentConfig + +Worker( + client, + task_queue="mytaskqueue", + workflows=workflows, + activities=activities, + deployment_config=WorkerDeploymentConfig( + version=WorkerDeploymentVersion( + deployment_name="llm_srv", + build_id=my_env.build_id), + use_worker_versioning=True, + default_versioning_behavior=VersioningBehavior.UNSPECIFIED + ), +) +``` + + + + +```ts +const myWorker = await Worker.create({ + workflowsPath: require.resolve('./workflows'), + taskQueue, + workerDeploymentOptions: { + useWorkerVersioning: true, + version: { buildId: '1.0', deploymentName: 'llm_srv' }, + }, + connection: nativeConnection, +}); +``` + + + + +```csharp +var myWorker = new TemporalWorker( + Client, + new TemporalWorkerOptions(taskQueue) + { + DeploymentOptions = new(new("llm_srv", "1.0"), true) + { + DefaultVersioningBehavior = VersioningBehavior.Unspecified, + }, + }.AddWorkflow()); +``` + + + + +```ruby +worker = Temporalio::Worker.new( + client: client, + task_queue: task_queue, + workflows: [MyWorkflow], + deployment_options: Temporalio::Worker::DeploymentOptions.new( + version: Temporalio::WorkerDeploymentVersion.new( + deployment_name: 'llm_srv', + build_id: '1.0' + ), + use_worker_versioning: true, + default_versioning_behavior: Temporalio::VersioningBehavior::UNSPECIFIED + ) +) +``` + + + + diff --git a/docs/production-deployment/worker-deployments/worker-versioning/index.mdx b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx new file mode 100644 index 0000000000..5551265175 --- /dev/null +++ b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx @@ -0,0 +1,212 @@ +--- +id: index +title: Worker Versioning +sidebar_label: Worker Versioning +description: + Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for + patching to support multiple code paths. +slug: /production-deployment/worker-deployments/worker-versioning +toc_max_heading_level: 4 +keywords: + - scaling + - workers + - versioning + - deploys +tags: + - Temporal Service + - Durable Execution +--- + +Worker Versioning is a Temporal feature that allows you to confidently deploy new changes to the Workflows running on +your Workers without breaking them. Temporal enables this by helping you manage different builds or versions, formally +called [Worker Deployment Versions](/worker-versioning#deployment-versions). + +For most teams, Worker Versioning should be the default recommendation for deploying Workflow code changes in +production. If you can run versioned worker deployments, prefer Worker Versioning over patching. + +Worker Versioning unlocks important benefits for users of [blue-green or rainbow deployments](#deployment-systems). + +- Ramping traffic gradually to a new Worker Deployment Version. +- Verifying a new Deployment Version with tests before sending production traffic to it. +- Instant rollback when you detect that a new Deployment Version is broken. +- Improved error rates when adopting it. + +In addition, Worker Versioning introduces **Workflow Pinning**. For pinned Workflow Types, each execution runs entirely +on the Worker Deployment Version where it started. You need not worry about making breaking code changes to running, +pinned Workflows. + +To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems). + +:::tip + +Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo. + +
+ +
+ +::: + +:::note + +Minimum versions: + +- Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0) +- Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0) +- Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0) +- Typescript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0) +- .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0) +- Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0) + +Self-hosted users: + +- Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1) +- Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1) +- Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) + +::: + +## Getting Started with Worker Versioning {/* #definition */} + +To get started with Worker Versioning, you should understand some concepts around versioning and deployments. + +- A **Worker Deployment** is a deployment or service across multiple versions. In a rainbow deployment, more than two + active Deployment Versions can run at once. +- A **Worker Deployment Version** is a version of a deployment or service. It can have multiple Workers polling on + multiple Task Queues, but they all run the same build. +- A **Build ID**, in combination with a Worker Deployment name, identifies a single Worker Deployment Version. +- When a versioned worker polls on a task queue, that task queue becomes part of that Worker's version. That version's + Worker Deployment controls how the task queue matches Workflow Tasks with Workers. +- Using **Workflow Pinning**, you can declare each Workflow type to have a **Versioning Behavior**, either Pinned or + Auto-Upgrade. + - A **Pinned** Workflow is guaranteed to complete on a single Worker Deployment Version. + - An **Auto-Upgrade** Workflow will automatically move to a new code version as you roll it out, specifically its + Target Worker Deployment Version (defined below). Therefore, Auto-Upgrade Workflows are not restricted to a single + Deployment Version and need to be kept replay-safe manually, i.e. with + [patching](/workflow-definition#workflow-versioning). + - Both Pinned and Auto-Upgrade Workflows are guaranteed to start only on the Current or Ramping Version of their + Worker Deployment. + - Pinned Workflows are designed for use with rainbow deployments. See [Deployment Systems](#deployment-systems). + - Pinned Workflows don't need to be patched, as they run on the same worker and build until they complete. + - If you expect your Workflow to run longer than you want your Worker Deployment Versions to exist, you should mark + your Workflow Type as Auto-Upgrade. +- Each Worker Deployment has a single [**Current Version**](/worker-versioning#versioning-definitions) which is where + Workflows are routed to unless they were previously pinned on a different version. +- Each Worker Deployment can have a [**Ramping Version**](/worker-versioning#versioning-definitions) which is where a + configurable percentage of Workflows are routed to unless they were previously pinned on a different version. +- For a given Workflow, its [**Target Worker Deployment Version**](/worker-versioning#versioning-definitions) is the + version it will move to next. + +## Setting up your deployment system {/* #deployment-systems */} + +If you haven't already, you'll want to pick a container deployment solution for your Workers. + +You also need to pick among three common deployment strategies: + +- A **rolling deployment** strategy upgrades Workers in place with little control over how quickly they cut over and + only a slow ability to roll Workers back. Rolling deploys have minimal footprint but tend to provide lower + availability than the other strategies and are incompatible with Worker Versioning. +- A **blue-green deployment** strategy maintains two "colors," or Worker Deployment Versions simultaneously and can + control how traffic is routed between them. This allows you to maximize your uptime with features like instant + rollback and ramping. Worker Versioning enables the routing control that blue-green deployments need. +- A **rainbow deployment** strategy is like blue-green but with more colors, allowing Workflow Pinning. You can deploy + new revisions of your Workflows freely while older versions drain. Using Worker Versioning, Temporal lets you know + when all the Workflows of a given version are drained so that you can sunset it. + +:::note + +You also have the option to use the +[Temporal Worker Controller](/production-deployment/worker-deployments/kubernetes-controller) to automatically enable +rainbow deployments of your Workers if you're using Kubernetes. + +::: + +If you cannot yet support blue-green or rainbow style deployments, use [patching](/patching) as a fallback while you +work toward a versioned deployment model. + +## Choosing a Versioning Behavior {/* #choosing-behavior */} + +The right versioning behavior depends on how long your Workflows run relative to your deployment frequency. + +### Decision guide {/* #decision-guide */} + +| Workflow Duration | Uses Continue-as-New? | Recommended Behavior | Patching Required? | +| ---------------------------------------- | --------------------- | -------------------------------------------------------- | ------------------ | +| **Short** (completes before next deploy) | N/A | `PINNED` | Never | +| **Medium** (spans multiple deploys) | No | `AUTO_UPGRADE` | Yes | +| **Long** (weeks to years) | Yes | `PINNED` + [upgrade on CaN](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) | Never | +| **Long** (weeks to years) | No | `AUTO_UPGRADE` + patching | Yes | + +### Examples by Workflow type {/* #behavior-examples */} + +| Workflow Type | Duration | Recommended Behavior | Notes | +| -------------------- | ------------ | -------------------------- | ----------------------------------- | +| Order processing | Minutes | `PINNED` | Completes before next deploy | +| Payment retry | Hours | `PINNED` or `AUTO_UPGRADE` | Depends on deploy frequency | +| Subscription billing | Days | `AUTO_UPGRADE` | May span multiple deploys | +| Customer entity | Months-Years | `PINNED` + upgrade on CaN | Uses Continue-as-New pattern | +| AI agent / Chatbot | Weeks | `PINNED` + upgrade on CaN | Long sleeps, uses CaN | +| Compliance audit | Months | `AUTO_UPGRADE` + patching | Cannot use CaN (needs full history) | + +:::info Long-running Workflows with Continue-as-New + +If your Workflow uses Continue-as-New to manage history size, you can upgrade to new Worker Deployment Versions at the +CaN boundary without patching. See [Upgrading on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new). + +::: + +### Default Versioning Behavior Considerations + +If you are using blue-green deployments, you should default to Auto-Upgrade and should not use Workflow Pinning. +Otherwise, if your Worker and Workflows are new, we suggest not providing a `DefaultVersioningBehavior`. + +In general, each Workflow Type should be annotated as Auto-Upgrade or Pinned. If all of your Workflows will be +short-running for the foreseeable future, you can default to Pinned. + +Many users who are migrating to Worker Versioning will start by defaulting to Auto-Upgrade until they have had time to +annotate their Workflows. This default is the most similar to the legacy behavior. Once each Workflow Type is annotated, +you can remove the `DefaultVersioningBehavior`. + +There is a possibility of a queue blocking limitation for new or Auto-Upgrade Workflows if there is a ramp, but one of +the Current or Ramping versions is down or doesn't have enough capacity. This leads to other versions not getting Tasks +or slowing down. + +For example, you have a Current Version and a Ramping Version at 50%. If all of your Current Version Workers go down, +you would expect at least 50% of new Workflows to go to the Ramping Version. This won't happen because the Tasks for the +Current Version are blocking the queue. + +:::note + +Keep in mind that Child Workflows of a parent or previous Auto-Upgrade Workflow default to Auto-Upgrade behavior and not +Unspecified. + +::: + +You also want to make sure you understand how your Activities are going to work across different Worker Deployment +Versions. Refer to the [Worker Versioning Activity behavior docs](/worker-versioning#activity-behavior-across-versions) +for more details. + +If you run [Serverless Workers](/serverless-workers), each Worker Deployment Version's compute provider points at a +specific Lambda function. In production, configure it with a qualified versioned ARN so Pinned Workflows remain truly +pinned to the code they started against, regardless of whether your default behavior is Pinned or Auto-Upgrade. See +[Worker Versioning with Serverless Workers](/serverless-workers#worker-versioning-with-serverless-workers). + + +## In this guide + +- [Configure a Worker for Versioning](/production-deployment/worker-deployments/worker-versioning/configure-worker) +- [Roll out and pin Workflows](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin) +- [Upgrade on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) +- [Sunset and garbage collection](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc) +- [Recover pinned Workflows](/production-deployment/worker-deployments/recover-pinned-workflows) +- [Unversioned to versioned migration](/production-deployment/worker-deployments/unversioned-to-versioned-migration) + diff --git a/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx b/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx new file mode 100644 index 0000000000..678f76215b --- /dev/null +++ b/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx @@ -0,0 +1,232 @@ +--- +id: roll-out-and-pin +title: Roll out and pin Workflows +sidebar_label: Roll out and pin +description: Roll out Worker Deployment Versions with the CLI, mark Workflow Types as Pinned, move pinned Workflows, and migrate to Auto-Upgrade. +toc_max_heading_level: 4 +keywords: + - workers + - versioning + - deploys +tags: + - Temporal Service + - Durable Execution +--- + +import { SdkTabs, AnnotatedCode } from '@site/src/components'; + +## Rolling out changes with the CLI {/* #rolling-out-changes-with-the-cli */} + +Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use +the `temporal` CLI to check which of your Worker versions are currently polling: + +You can view the Versions that are part of a Deployment with `temporal worker deployment describe`: + +```bash +temporal worker deployment describe --name="$MY_DEPLOYMENT" +``` + +To activate a Deployment Version, use `temporal worker deployment set-current-version`, specifying the deployment name +and a Build ID: + +```bash +temporal worker deployment set-current-version \ + --deployment-name "YourDeploymentName" \ + --build-id "YourBuildID" +``` + +To ramp a Deployment Version up to some percentage of your overall Worker fleet, use `set-ramping version`, with the +same parameters and a ramping percentage: + +```bash +temporal worker deployment set-ramping-version \ + --deployment-name "YourDeploymentName" \ + --build-id "YourBuildID" \ + --percentage=5 +``` + +You can verify that Workflows are cutting over to that version with `describe -w YourWorkflowID`: + +```bash +temporal workflow describe -w YourWorkflowID +``` + +That returns the new Version that the workflow is running on: + +``` +Versioning Info: + + Behavior AutoUpgrade + Version llm_srv.2.0 + OverrideBehavior Unspecified +``` + +## Marking a Workflow Type as Pinned + +You can mark a Workflow Type as pinned when you register it by adding an additional Pinned parameter. This will cause it +to remain on its original deployed version: + + + +```go +// w is the Worker configured as in the previous example +w.RegisterWorkflowWithOptions(HelloWorld, workflow.RegisterOptions{ + // or workflow.VersioningBehaviorAutoUpgrade + VersioningBehavior: workflow.VersioningBehaviorPinned, +}) +``` + + +```java +@WorkflowInterface +public interface HelloWorld { + @WorkflowMethod + String hello(); +} + +public static class HelloWorldImpl implements HelloWorld { + @Override + @WorkflowVersioningBehavior(VersioningBehavior.PINNED) + public String hello() { + return "Hello, World!"; + } +} + +``` + + +```python +@workflow.defn(versioning_behavior=VersioningBehavior.PINNED) +class HelloWorld: + @workflow.run + async def run(self): + return "hello world!" + +``` + + + + +```ts +setWorkflowOptions({ versioningBehavior: 'PINNED' }, helloWorld); +export async function helloWorld(): Promise { + return 'hello world!'; +} +``` + + + + +```csharp +[Workflow(VersioningBehavior = VersioningBehavior.Pinned)] +public class HelloWorld +{ + [WorkflowRun] + public async Task RunAsync() + { + return "hello world!"; + } +} +``` + + + + +```ruby +class HelloWorld < Temporalio::Workflow::Definition + workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED + + def execute + 'hello world!' + end +end +``` + + + + +## Moving a pinned Workflow {/* #moving-a-pinned-workflow */} + +Sometimes you'll need to manually move a set of pinned Workflows off of a version that has a bug to a version with the +fix. + +If you need to move a pinned Workflow to a new version, use `temporal workflow update-options`: + +```bash +temporal workflow update-options \ + --workflow-id "$WORKFLOW_ID" \ + --versioning-override-behavior pinned \ + --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ + --versioning-override-build-id "$TARGET_BUILD_ID" +``` + +You can move several Workflows at once matching a `--query` parameter: + +```bash +temporal workflow update-options \ + --query="TemporalWorkerDeploymentVersion=$TARGET_DEPLOYMENT:$BAD_BUILD_ID" \ + --versioning-override-behavior pinned \ + --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ + --versioning-override-build-id "$FIXED_BUILD_ID" +``` + +In this scenario, you may also need to use the other [Versioning APIs](/workflow-definition#workflow-versioning) to +patch your Workflow in the "fixed" build, so that your target Worker can handle the moved Workflows correctly. If you +made a [version-incompatible change](/workflow-definition#deterministic-constraints) to your Workflow, and you want to +roll back to an earlier version, it's not possible to patch it. Considering using +[Workflow Reset](/workflow-execution/event#reset) along with your move. + +"Reset-with-Move" allows you to atomically Reset your Workflow and set a Versioning Override on the newly reset +Workflow, so when it resumes execution, all new Workflow Tasks will be executed on your new Worker. + +```bash +temporal workflow reset with-workflow-update-options \ + --workflow-id "$WORKFLOW_ID" \ + --event-id "$EVENT_ID" \ + --reason "$REASON" \ + --versioning-override-behavior pinned \ + --versioning-override-deployment-name "$TARGET_DEPLOYMENT" \ + --versioning-override-build-id "$TARGET_BUILD_ID" +``` + +For a complete runbook covering batch recovery, drainage handling, and how to choose between Versioning Override and Reset-with-Move based on each Workflow's state, see [Recover pinned Workflows after a bad rollout](/production-deployment/worker-deployments/recover-pinned-workflows). + +## Migrating a Workflow from Pinned to Auto-Upgrade + +There may be times when you need to migrate your Workflow from Pinned to Auto-Upgrade because you configured your +Workflow Type with the wrong behavior or you've pinned a really long-running Workflow by mistake. + +Pinned Workflows can block version drainage, especially when they run for a long time. You could move the Workflow to a +new build, but that would just push the problem to the next build. + +In order to make this change, you need to change the versioning behavior for your Workflow from Pinned to Auto-Upgrade. +You can use `temporal workflow update-options` for this: + +```bash +temporal workflow update-options \ + --workflow-id "$WORKFLOW_ID" \ + --versioning-override-behavior auto_upgrade +``` + +If you want to move all your Workflows of a certain type to this new configuration, you can do it with this command: + +```bash +temporal workflow update-options \ + --query="WorkflowType='$WORKFLOW_TYPE'" \ + --versioning-override-behavior auto_upgrade +``` + +You can also filter on a certain build ID to limit the number of Workflows you apply it to: + +```bash +temporal workflow update-options \ + --query="WorkflowType='$WORKFLOW_TYPE' AND TemporalWorkerDeploymentVersion='$TARGET_DEPLOYMENT:$OLD_VERSION'" \ + --versioning-override-behavior auto_upgrade +``` + +:::note + +When you change the behavior to Auto-Upgrade, the Workflow will resume work on the Workflow's Target Version. So if the Workflow's Target Version is different from the earlier Pinned Version, you should make sure you [patch](/patching#patching) the Workflow code. + +::: + diff --git a/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx b/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx new file mode 100644 index 0000000000..9c9a4c316b --- /dev/null +++ b/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx @@ -0,0 +1,179 @@ +--- +id: sunset-and-gc +title: Sunset and garbage collection +sidebar_label: Sunset and GC +description: Drain and sunset old Worker Deployment Versions, then garbage-collect unused versions. +toc_max_heading_level: 4 +keywords: + - workers + - versioning + - deploys +tags: + - Temporal Service + - Durable Execution +--- + +import { SdkTabs } from '@site/src/components'; + +## Sunsetting an old Deployment Version {/* #sunsetting-an-old-deployment-version */} + +A Worker Deployment Version moves through the following states: + +1. **Inactive**: The version exists because a Worker with that version has polled the server. If this version never + becomes Active, it will never be Draining or Drained. +2. **Active**: The version is either Current or Ramping, so it is accepting new Workflows and existing Auto-Upgrade + Workflows. +3. **Draining**: The version stopped being Current or Ramping, and it has open pinned Workflows running on it. It is + possible to be Draining and have no open pinned Workflows for a short time, since the drainage status is updated + periodically. +4. **Drained**: The version was draining and now all the pinned Workflows that were running on it are closed. + +You can see these statuses when you describe a Worker Deployment in the `WorkerDeploymentVersionStatus` of each +`VersionSummary`, or by describing the version directly. When a version is Draining or Drained, that is displayed in a +value called `DrainageStatus`. Periodically, the Temporal Service will refresh this status by counting any open pinned +Workflows using that version. + +On each refresh, `DrainageInfo.last_checked_time` is updated. Eventually, `DrainageInfo` will report that the version is +fully drained. At this point, no Workflows are still running on that version and no more will be automatically routed to +it, so you can consider shutting down the running Workers. + +You can monitor this by checking `WorkerDeploymentInfo.VersionSummaries` or with +`temporal worker deployment describe-version`: + +```bash +temporal worker deployment describe-version \ + --deployment-name "YourDeploymentName" \ + --build-id "YourBuildID" +``` + +``` +Worker Deployment Version: + Version llm_srv.1.0 + CreateTime 5 hours ago + RoutingChangedTime 32 seconds ago + RampPercentage 0 + DrainageStatus draining + DrainageLastChangedTime 31 seconds ago + DrainageLastCheckedTime 31 seconds ago + +Task Queues: + Name Type + hello-world activity + hello-world workflow +``` + +If you have implemented [Queries](/sending-messages#sending-queries) on closed pinned Workflows, you may need to keep +some Workers running to handle them. + +### Adding a pre-deployment test + +Before deploying a new Workflow revision, you can test it with synthetic traffic. + +To do this, use pinning in your tests, following the examples below + + + +```go +workflowOptions := client.StartWorkflowOptions{ + ID: "MyWorkflowId", + TaskQueue: "MyTaskQueue", + VersioningOverride: &client.PinnedVersioningOverride{ + Version: worker.WorkerDeploymentVersion{ + DeploymentName: "DeployName", + BuildID: "1.0", + }, + }, +} +// c is an initialized Client +we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, HelloWorld, "Hello") +``` + + +```java +MyWorkflow handle = client.newWorkflowStub( + MyWorkflow.class, + WorkflowOptions.newBuilder() + .setWorkflowId("MyWorkflowId") + .setTaskQueue("MyTaskQueue") + .setVersioningOverride(new VersioningOverride.PinnedVersioningOverride( + new WorkerDeploymentVersion("DeployName", "1.0"))) + .build() +); +WorkflowExecution we = WorkflowClient.start(handle::execute, "Hello"); +``` + + +```python +handle = client.start_workflow( + MyWorkflow.run, + "Hello", + id="MyWorkflowId", + task_queue="MyTaskQueue", + versioning_override=PinnedVersioningOverride( + WorkerDeploymentVersion("DeployName", "1.0") + ), +) +``` + + +```ts +const handle = await client.workflow.start('helloWorld', { + taskQueue: 'MyTaskQueue', + workflowId: 'MyWorkflowId', + versioningOverride: { + pinnedTo: { buildId: '1.0', deploymentName: 'deploy-name' }, + }, +}); +``` + + +```csharp +var workerV1 = new WorkerDeploymentVersion("deploy-name", "1.0"); +var handle = await Client.StartWorkflowAsync( + (HelloWorld wf) => wf.RunAsync(), + new(id: "MyWorkflowId", taskQueue: "MyTaskQueue") + { + VersioningOverride = new VersioningOverride.Pinned(workerV1), + } +); +``` + + +```ruby +worker_v1 = Temporalio::WorkerDeploymentVersion.new( + deployment_name: 'deploy-name', + build_id: '1.0' +) +handle = env.client.start_workflow( + HelloWorld, + id: 'MyWorkflowId', + task_queue: 'MyTaskQueue', + versioning_override: Temporalio::VersioningOverride.pinned(worker_v1) +) +``` + + + +## Garbage collection + +Worker Deployments are never garbage collected, but _Worker Deployment Versions_ (often referred to as Versions, Worker +Versions, Deployment Versions) are. + +Versions are deleted to keep the total number of versions in one Worker Deployment less than or equal to +[`matching.maxVersionsInDeployment`](https://github.com/temporalio/temporal/blob/48dc5a95949ea0e555ce0f48e0031b54633fc703/common/dynamicconfig/constants.go#L1501-L1505), +which is currently set to 100 in Temporal Cloud, but that's a conservative number and it could be increased if needed. + +For example, when you deploy your 101st Worker Version in a Worker Deployment, the server looks at the oldest drained +version in the Worker deployment. If it has had no pollers in the last 5 minutes, the server deletes it. If that version +still has pollers, the server will try the next oldest version. If none of the 100 versions are eligible for deletion +(ie. none of them are drained with no pollers), then no version will be deleted and the poll from the 101st version +would fail. + +At that point, to successfully deploy your 101st version, you would need to increase `matching.maxVersionsInDeployment` +or stop polling from one of the old drained versions to make it eligible for clean up. + +If you want to re-deploy a previously deleted version, start polling with a Worker that has the same build ID and +Deployment Name as the deleted version and the server will recreate it. + +This covers the complete lifecycle of working with Worker Versioning. We are continuing to improve this feature, and we +welcome any feedback or feature requests using the sidebar link! diff --git a/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx b/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx new file mode 100644 index 0000000000..cfdfb1e85f --- /dev/null +++ b/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx @@ -0,0 +1,266 @@ +--- +id: upgrade-on-continue-as-new +title: Upgrade on Continue-as-New +sidebar_label: Upgrade on Continue-as-New +description: Upgrade pinned Workflows to a new Worker Deployment Version by Continue-as-New when a Target Version becomes available. +toc_max_heading_level: 4 +keywords: + - workers + - versioning + - continue-as-new +tags: + - Temporal Service + - Durable Execution +--- + +import { SdkTabs, AnnotatedCode } from '@site/src/components'; + +## Upgrading on Continue-as-New {/* #upgrade-on-continue-as-new */} + +Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker +Deployment Versions at Continue-as-New boundaries without requiring patching. + +This pattern is ideal for: + +- **Entity Workflows** that run for months or years +- **Batch processing** Workflows that checkpoint with Continue-as-New +- **AI agent Workflows** with long sleeps waiting for user input + +:::note Public Preview + +This feature is in Public Preview as an experimental SDK-level option. + +::: + +### How it works {/* #upgrade-on-can-how-it-works */} + +By default, Pinned Workflows stay on their original Worker Deployment Version even when they Continue-as-New. With the +upgrade option enabled: + +1. Each Workflow run remains pinned to its version (no patching needed during a run) +2. The Temporal Server tells the workflow when a new [Target Version](/worker-versioning#versioning-definitions) becomes available +3. When the Workflow performs Continue-as-New with the upgrade option, the new run starts on the [Target Version](/worker-versioning#versioning-definitions) + +### Checking for new versions {/* #checking-for-new-versions */} + +When a new Worker Deployment Version becomes Current or Ramping, active Workflows can detect this through +`target_worker_deployment_version_changed`: + + + + +```go +func (w *Workflows) ContinueAsNewWithVersionUpgradeV1( + ctx workflow.Context, + attempt int, +) (string, error) { + if attempt > 0 { + return "v1.0", nil + } + + // Check GetTargetWorkerDeploymentVersionChanged periodically. + // GetTargetWorkerDeploymentVersionChanged is refreshed after each WFT completes. + for { + // Trigger a WFT when timer expires, thereby refreshing the GetTargetWorkerDeploymentVersionChanged flag. + // Since this is just a test workflow, we aren't doing any real work. In a real workflow regularly + // doing non-sleep workflow tasks, you would not need to artificially trigger a WFT to refresh the + // GetTargetWorkerDeploymentVersionChanged flag. You could choose to check the field periodically, or you + // might want to check before accepting updates, starting activities, or starting child workflows. + err := workflow.Sleep(ctx, 10*time.Millisecond) + if err != nil { + return "", err + } + info := workflow.GetInfo(ctx) + if info.GetTargetWorkerDeploymentVersionChanged() { + return "", workflow.NewContinueAsNewErrorWithOptions( + ctx, + workflow.ContinueAsNewErrorOptions{ + // Pass InitialVersioningBehavior=workflow.ContinueAsNewVersioningBehaviorAutoUpgrade + // to make the new run start with AutoUpgrade behavior and use the Target Version of + // its Worker Deployment. + InitialVersioningBehavior: workflow.ContinueAsNewVersioningBehaviorAutoUpgrade, + }, + "ContinueAsNewWithVersionUpgrade", + attempt+1, + ) + } + } +} + +func (w *Workflows) ContinueAsNewWithVersionUpgradeV2( + ctx workflow.Context, + attempt int, +) (string, error) { + return "v2.0", nil +} +``` + + + + +```java +public class ContinueAsNewWithVersionUpgradeImpl implements ContinueAsNewWithVersionUpgrade { + @Override + public String run(int attempt) { + if (attempt > 0) { + return "v1.0"; + } + + // isTargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. + // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial + // timer; you could check the flag periodically, or before accepting Updates, starting + // Activities, or starting child Workflows. + while (true) { + Workflow.sleep(Duration.ofMillis(10)); + if (Workflow.getInfo().isTargetWorkerDeploymentVersionChanged()) { + // Set InitialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade + // behavior and uses the Target Version of its Worker Deployment. + Workflow.continueAsNew( + ContinueAsNewOptions.newBuilder() + .setInitialVersioningBehavior(InitialVersioningBehavior.AUTO_UPGRADE) + .build(), + attempt + 1); + } + } + } +} +``` + + + + +```python +@workflow.defn +class ContinueAsNewWithVersionUpgrade: + @workflow.run + async def run(self, attempt: int) -> str: + if attempt > 0: + return "v1.0" + + # is_target_worker_deployment_version_changed() is refreshed after each Workflow Task + # completes. In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need + # an artificial timer; you could check the flag periodically, or before accepting Updates, + # starting Activities, or starting child Workflows. + while True: + await workflow.sleep(timedelta(milliseconds=10)) + if workflow.info().is_target_worker_deployment_version_changed(): + # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with + # AutoUpgrade behavior and uses the Target Version of its Worker Deployment. + workflow.continue_as_new( + attempt + 1, + initial_versioning_behavior=ContinueAsNewVersioningBehavior.AUTO_UPGRADE, + ) +``` + + + + +```ts +import * as wf from '@temporalio/workflow'; +import { InitialVersioningBehavior } from '@temporalio/common'; + +export async function continueAsNewWithVersionUpgrade(attempt: number): Promise { + if (attempt > 0) { + return 'v1.0'; + } + + // targetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. + // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial + // timer; you could check the flag periodically, or before accepting Updates, starting + // Activities, or starting child Workflows. + for (;;) { + await wf.sleep('10ms'); + if (wf.workflowInfo().targetWorkerDeploymentVersionChanged) { + // Set initialVersioningBehavior to AUTO_UPGRADE so the new run starts with AutoUpgrade + // behavior and uses the Target Version of its Worker Deployment. + return await wf.makeContinueAsNewFunc({ + initialVersioningBehavior: InitialVersioningBehavior.AUTO_UPGRADE, + })(attempt + 1); + } + } +} +``` + + + + +```csharp +[Workflow] +public class ContinueAsNewWithVersionUpgrade +{ + [WorkflowRun] + public async Task RunAsync(int attempt) + { + if (attempt > 0) + { + return "v1.0"; + } + + // TargetWorkerDeploymentVersionChanged is refreshed after each Workflow Task completes. + // In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an + // artificial timer; you could check the flag periodically, or before accepting Updates, + // starting Activities, or starting child Workflows. + while (true) + { + await Workflow.DelayAsync(TimeSpan.FromMilliseconds(10)); + if (Workflow.TargetWorkerDeploymentVersionChanged) + { + // Set InitialVersioningBehavior to AutoUpgrade so the new run starts with + // AutoUpgrade behavior and uses the Target Version of its Worker Deployment. + throw Workflow.CreateContinueAsNewException( + (ContinueAsNewWithVersionUpgrade wf) => wf.RunAsync(attempt + 1), + new ContinueAsNewOptions + { + InitialVersioningBehavior = InitialVersioningBehavior.AutoUpgrade, + }); + } + } + } +} +``` + + + + +```ruby +class ContinueAsNewWithVersionUpgrade < Temporalio::Workflow::Definition + workflow_versioning_behavior Temporalio::VersioningBehavior::PINNED + + def execute(attempt) + return 'v1.0' if attempt.positive? + + # target_worker_deployment_version_changed? is refreshed after each Workflow Task completes. + # In a Workflow that regularly does non-sleep Workflow Tasks you wouldn't need an artificial + # timer; you could check the flag periodically, or before accepting Updates, starting + # Activities, or starting child Workflows. + loop do + Temporalio::Workflow.sleep(0.01) + next unless Temporalio::Workflow.target_worker_deployment_version_changed? + + # Set initial_versioning_behavior to AUTO_UPGRADE so the new run starts with AutoUpgrade + # behavior and uses the Target Version of its Worker Deployment. + raise Temporalio::Workflow::ContinueAsNewError.new( + attempt + 1, + initial_versioning_behavior: Temporalio::ContinueAsNewVersioningBehavior::AUTO_UPGRADE + ) + end + end +end +``` + + + + +### Limitations {/* #upgrade-on-can-limitations */} + +:::caution Current Limitations + +- **Lazy moving only:** Workflows must execute a step to receive the target-version-changed information. Sleeping + Workflows won't proactively get it. If you have idle Workflows that you want to wake up so they can check the + target-version-changed flag, you can send them a Signal. +- **Interface compatibility:** When continuing as new to a different version, ensure your Workflow input provided by the + previous version's workflow definition is compatible with the new version's workflow definition. If incompatible, the + new run may fail on its first Workflow Task. + +::: + diff --git a/sidebars.js b/sidebars.js index 7f751809b1..a8ed82240b 100644 --- a/sidebars.js +++ b/sidebars.js @@ -1442,11 +1442,25 @@ module.exports = { id: 'production-deployment/worker-deployments/index', }, items: [ - 'production-deployment/worker-deployments/worker-versioning', - 'production-deployment/worker-deployments/recover-pinned-workflows', + { + type: 'category', + label: 'Worker Versioning', + collapsed: true, + link: { + type: 'doc', + id: 'production-deployment/worker-deployments/worker-versioning/index', + }, + items: [ + 'production-deployment/worker-deployments/worker-versioning/configure-worker', + 'production-deployment/worker-deployments/worker-versioning/roll-out-and-pin', + 'production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new', + 'production-deployment/worker-deployments/worker-versioning/sunset-and-gc', + 'production-deployment/worker-deployments/recover-pinned-workflows', + 'production-deployment/worker-deployments/unversioned-to-versioned-migration', + ], + }, 'production-deployment/worker-deployments/kubernetes-controller', 'production-deployment/worker-deployments/deploy-workers-to-aws-eks', - 'production-deployment/worker-deployments/unversioned-to-versioned-migration', { type: 'category', label: 'Serverless Workers', From 6b35cde4aa238b0a793f43807b3f8962890c2138 Mon Sep 17 00:00:00 2001 From: Jwahir Sundai Date: Fri, 7 Aug 2026 11:09:34 -0500 Subject: [PATCH 2/3] index, transition --- .../worker-versioning/index.mdx | 20 +++++++++---------- .../worker-versioning/roll-out-and-pin.mdx | 2 ++ .../worker-versioning/sunset-and-gc.mdx | 2 ++ .../upgrade-on-continue-as-new.mdx | 2 ++ 4 files changed, 15 insertions(+), 11 deletions(-) diff --git a/docs/production-deployment/worker-deployments/worker-versioning/index.mdx b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx index 5551265175..f8936d1349 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning/index.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx @@ -5,7 +5,6 @@ sidebar_label: Worker Versioning description: Use Worker Versioning to pin Workflow revisions to individual Worker Deployment Versions, avoiding the need for patching to support multiple code paths. -slug: /production-deployment/worker-deployments/worker-versioning toc_max_heading_level: 4 keywords: - scaling @@ -37,6 +36,15 @@ pinned Workflows. To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems). +## In this guide + +- [Configure a Worker for Versioning](/production-deployment/worker-deployments/worker-versioning/configure-worker) +- [Roll out and pin Workflows](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin) +- [Upgrade on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) +- [Sunset and garbage collection](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc) +- [Recover pinned Workflows](/production-deployment/worker-deployments/recover-pinned-workflows) +- [Unversioned to versioned migration](/production-deployment/worker-deployments/unversioned-to-versioned-migration) + :::tip Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo. @@ -200,13 +208,3 @@ specific Lambda function. In production, configure it with a qualified versioned pinned to the code they started against, regardless of whether your default behavior is Pinned or Auto-Upgrade. See [Worker Versioning with Serverless Workers](/serverless-workers#worker-versioning-with-serverless-workers). - -## In this guide - -- [Configure a Worker for Versioning](/production-deployment/worker-deployments/worker-versioning/configure-worker) -- [Roll out and pin Workflows](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin) -- [Upgrade on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) -- [Sunset and garbage collection](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc) -- [Recover pinned Workflows](/production-deployment/worker-deployments/recover-pinned-workflows) -- [Unversioned to versioned migration](/production-deployment/worker-deployments/unversioned-to-versioned-migration) - diff --git a/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx b/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx index 678f76215b..37b6e5017d 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin.mdx @@ -15,6 +15,8 @@ tags: import { SdkTabs, AnnotatedCode } from '@site/src/components'; +This page covers Rolling out changes with the CLI, Marking a Workflow Type as Pinned, Moving a pinned Workflow, and Migrating a Workflow from Pinned to Auto-Upgrade. + ## Rolling out changes with the CLI {/* #rolling-out-changes-with-the-cli */} Next, deploy your Worker with the additional configuration parameters. Before making any Workflow revisions, you can use diff --git a/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx b/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx index 9c9a4c316b..5d29ca5d5d 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning/sunset-and-gc.mdx @@ -15,6 +15,8 @@ tags: import { SdkTabs } from '@site/src/components'; +This page covers Sunsetting an old Deployment Version and Garbage collection. + ## Sunsetting an old Deployment Version {/* #sunsetting-an-old-deployment-version */} A Worker Deployment Version moves through the following states: diff --git a/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx b/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx index cfdfb1e85f..c21ab717fd 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new.mdx @@ -15,6 +15,8 @@ tags: import { SdkTabs, AnnotatedCode } from '@site/src/components'; +This page covers how to upgrade pinned Workflows to a new Worker Deployment Version with Continue-as-New. + ## Upgrading on Continue-as-New {/* #upgrade-on-continue-as-new */} Long-running Workflows that use [Continue-as-New](/workflow-execution/continue-as-new) can upgrade to newer Worker From d4eafddb2df52adb8a261723772390008177ea49 Mon Sep 17 00:00:00 2001 From: Jwahir Sundai Date: Fri, 7 Aug 2026 13:55:10 -0500 Subject: [PATCH 3/3] drop stacking of admonitions --- .../worker-versioning/index.mdx | 19 +++---------------- 1 file changed, 3 insertions(+), 16 deletions(-) diff --git a/docs/production-deployment/worker-deployments/worker-versioning/index.mdx b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx index f8936d1349..01288f3b8d 100644 --- a/docs/production-deployment/worker-deployments/worker-versioning/index.mdx +++ b/docs/production-deployment/worker-deployments/worker-versioning/index.mdx @@ -36,15 +36,6 @@ pinned Workflows. To use Workflow Pinning, we recommend using [rainbow deployments](#deployment-systems). -## In this guide - -- [Configure a Worker for Versioning](/production-deployment/worker-deployments/worker-versioning/configure-worker) -- [Roll out and pin Workflows](/production-deployment/worker-deployments/worker-versioning/roll-out-and-pin) -- [Upgrade on Continue-as-New](/production-deployment/worker-deployments/worker-versioning/upgrade-on-continue-as-new) -- [Sunset and garbage collection](/production-deployment/worker-deployments/worker-versioning/sunset-and-gc) -- [Recover pinned Workflows](/production-deployment/worker-deployments/recover-pinned-workflows) -- [Unversioned to versioned migration](/production-deployment/worker-deployments/unversioned-to-versioned-migration) - :::tip Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and see a demo. @@ -64,14 +55,12 @@ Watch this Temporal Replay 2025 talk to learn more about Worker Versioning and s ::: -:::note - -Minimum versions: +## Minimum versions - Go SDK version [v1.35.0](https://github.com/temporalio/sdk-go/releases/tag/v1.35.0) - Python [v1.11](https://github.com/temporalio/sdk-python/releases/tag/1.11.0) - Java [v1.29](https://github.com/temporalio/sdk-java/releases/tag/v1.29.0) -- Typescript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0) +- TypeScript [v1.12](https://github.com/temporalio/sdk-typescript/releases/tag/v1.12.0) - .NET [v1.7.0](https://github.com/temporalio/sdk-dotnet/releases/tag/1.7.0) - Ruby [v0.5.0](https://github.com/temporalio/sdk-ruby/releases/tag/v0.5.0) @@ -79,9 +68,7 @@ Self-hosted users: - Minimum Temporal CLI version [v1.4.1](https://github.com/temporalio/cli/releases/tag/v1.4.1) - Minimum Temporal Server version: [v1.29.1](https://github.com/temporalio/temporal/releases/tag/v1.29.1) -- Minimum Temporal UI Version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) - -::: +- Minimum Temporal UI version [v2.38.0](https://github.com/temporalio/ui/releases/tag/v2.38.0) ## Getting Started with Worker Versioning {/* #definition */}