Add guide for migrating Celery task queues to Standalone Activities - #5135
Draft
brianmacdonald-temporal wants to merge 1 commit into
Draft
Add guide for migrating Celery task queues to Standalone Activities#5135brianmacdonald-temporal wants to merge 1 commit into
brianmacdonald-temporal wants to merge 1 commit into
Conversation
Add a Python guide that walks through converting a Celery task into a Temporal Standalone Activity: running an Activity-only Worker, replacing .delay() and .get() calls, migrating max_retries to a Retry Policy, and using list_activities() in place of Flower. Link it from the Guides sidebar and add a card to the Guides landing page grid under a new Migration tag. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
📖 Docs PR preview links
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
|
|
||
| In this guide, you will migrate a Celery task to a Temporal Standalone Activity. You will convert the task into an Activity, run a Worker to process it, execute it both synchronously and fire-and-forget in place of your `.get()` and `.delay()` calls, migrate its retries to a Retry Policy, and inspect running Activities in place of Flower. By the end, you will have a working Temporal Application that reproduces the behavior of your Celery app with no Workflow code. | ||
|
|
||
| ## How Celery Concepts Map to Standalone Activities |
Contributor
There was a problem hiding this comment.
📝 [vale] reported by reviewdog 🐶
[Temporal.Headings] 'How Celery Concepts Map to Standalone Activities' should use sentence-style capitalization.
| brew install temporal | ||
| ``` | ||
|
|
||
| If you aren't using Homebrew, download the binary for your platform from the [Temporal CLI install guide](https://docs.temporal.io/cli/setup-cli) and add it to your `PATH`. |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/cli/setup-cli'.
|
|
||
| Your application code will connect to `localhost:7233`. The Web UI at `http://localhost:8233` lets you inspect Activities and their results; Standalone Activities appear under their own item in the UI's navigation. Leave this process running and open a new terminal for the remaining steps. | ||
|
|
||
| ## Step 4 — Converting a Celery task into an Activity |
Contributor
There was a problem hiding this comment.
📝 [vale] reported by reviewdog 🐶
[Temporal.Headings] 'Step 4 — Converting a Celery task into an Activity' should use sentence-style capitalization.
| Total activities: 1 | ||
| ``` | ||
|
|
||
| The `query` uses the same [List Filter](https://docs.temporal.io/list-filter) syntax as Workflow visibility, so you can filter by attributes such as `ActivityType` and `Status` — for example, `"ActivityType = 'send_welcome_email' AND Status = 'Running'"`. These calls return only Standalone Activities; Activities running inside Workflows are excluded. The Temporal CLI offers the same views with `temporal activity list` and `temporal activity count`. |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/list-filter'.
|
|
||
| Standalone Activities replace the common case: a Celery task that does one independent thing. They deliberately have no orchestration, so there is one situation they do not cover — multi-step pipelines. | ||
|
|
||
| If your Celery app uses [Canvas](https://docs.celeryq.dev/en/stable/userguide/canvas.html) primitives — chaining tasks so one result feeds the next (`chain`), fanning work out in parallel (`group`), or running a callback after a group finishes (`chord`) — that coordination logic needs somewhere to live durably. A Standalone Activity cannot call another Activity or guarantee progress across several steps. For those pipelines, wrap your Activities in a Temporal **Workflow**, where sequencing is ordinary `await` statements and parallelism is `asyncio.gather`. See the [Temporal Python documentation](https://docs.temporal.io/develop/python) for building Workflows. |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/develop/python'.
|
|
||
| In this tutorial, you migrated a Celery task to a Temporal Standalone Activity. You converted the task into an Activity, ran a Worker to execute it, invoked it both synchronously and fire-and-forget in place of your `.get()` and `.delay()` calls, replaced hand-written retries with a retry policy, and inspected your Activities in place of Flower — all without writing a single Workflow. Your jobs now survive Worker crashes, retry on well-defined policies, and remain queryable through the client and Web UI. | ||
|
|
||
| Because Standalone Activities are in Public Preview, review the [Standalone Activities feature guide](https://docs.temporal.io/develop/python/activities/standalone-activities) for the latest API details before relying on them in production. Useful next topics include: |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/develop/python/activities/standalone-activities'.
|
|
||
| Because Standalone Activities are in Public Preview, review the [Standalone Activities feature guide](https://docs.temporal.io/develop/python/activities/standalone-activities) for the latest API details before relying on them in production. Useful next topics include: | ||
|
|
||
| - The [Standalone Activities Quickstart](https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart) for the runnable reference sample. |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart'.
| Because Standalone Activities are in Public Preview, review the [Standalone Activities feature guide](https://docs.temporal.io/develop/python/activities/standalone-activities) for the latest API details before relying on them in production. Useful next topics include: | ||
|
|
||
| - The [Standalone Activities Quickstart](https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart) for the runnable reference sample. | ||
| - [Activity timeouts](https://docs.temporal.io/develop/python/activities/timeouts) for tuning `start_to_close` and related limits. |
Contributor
There was a problem hiding this comment.
[Temporal.RelativeLinks] Use a relative path instead of the absolute docs.temporal.io URL 'https://docs.temporal.io/develop/python/activities/timeouts'.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Python migration guide for converting Celery tasks to Temporal Standalone Activities.
Changes:
- Documents task conversion, invocation, retries, and monitoring.
- Adds the guide to sidebar navigation.
- Adds a migration card and filter tag to the Guides grid.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 10 comments.
| File | Description |
|---|---|
docs/guides/celery-to-standalone-activity.mdx |
Adds the migration guide and Python examples. |
sidebars.js |
Adds the guide to navigation. |
src/components/GuidesGrid/guides-data.json |
Adds the guide’s landing-page card. |
Suppressed comments (7)
docs/guides/celery-to-standalone-activity.mdx:369
maximum_attempts=5does not mirror Celery'smax_retries=5: Temporal includes the initial attempt, whereas Celery counts five retries after it. Update this explanation and the policy above to six total attempts; also account for the Celery task's fixed 10-second retry delay.
Here, `maximum_attempts=5` mirrors Celery's `max_retries`, and `maximum_interval` caps the backoff between attempts. The `non_retryable_error_types` list names errors that should fail immediately without retrying — the equivalent of *not* calling `self.retry()` for a permanent failure. To raise such an error from the Activity, use `ApplicationError` with `non_retryable=True` in `my_activity.py`:
docs/guides/celery-to-standalone-activity.mdx:103
- Both occurrences must use the proper product name “Temporal Service” (
readme/STYLE.md:24-31), not “Temporal service.” The table at line 37 needs the same correction.
Confirm the printed version is at least 1.7.0. With the tools installed, you can start a local Temporal service.
## Step 3 — Starting the Temporal Development Server
In Celery, work flows through a broker such as Redis. In Temporal, work flows through the Temporal service, which also stores each Activity's durable state. In this step, you will start a local development server that stands in for that service.
docs/guides/celery-to-standalone-activity.mdx:91
- Use the relative destination
/cli/setup-clifor this internal link. Absolutedocs.temporal.iolinks fail the CI-scopedTemporal.RelativeLinksrule; other occurrences remain at lines 25, 431, 437, 445, 447, and 448.
If you aren't using Homebrew, download the binary for your platform from the [Temporal CLI install guide](https://docs.temporal.io/cli/setup-cli) and add it to your `PATH`.
docs/guides/celery-to-standalone-activity.mdx:431
- Use the relative destination
/list-filterfor this internal link. Absolutedocs.temporal.iolinks fail the CI-scopedTemporal.RelativeLinksrule; other occurrences remain at lines 25, 91, 437, 445, 447, and 448.
The `query` uses the same [List Filter](https://docs.temporal.io/list-filter) syntax as Workflow visibility, so you can filter by attributes such as `ActivityType` and `Status` — for example, `"ActivityType = 'send_welcome_email' AND Status = 'Running'"`. These calls return only Standalone Activities; Activities running inside Workflows are excluded. The Temporal CLI offers the same views with `temporal activity list` and `temporal activity count`.
docs/guides/celery-to-standalone-activity.mdx:437
- Use the relative destination
/develop/pythonfor this internal link. Absolutedocs.temporal.iolinks fail the CI-scopedTemporal.RelativeLinksrule; other occurrences remain at lines 25, 91, 431, 445, 447, and 448.
If your Celery app uses [Canvas](https://docs.celeryq.dev/en/stable/userguide/canvas.html) primitives — chaining tasks so one result feeds the next (`chain`), fanning work out in parallel (`group`), or running a callback after a group finishes (`chord`) — that coordination logic needs somewhere to live durably. A Standalone Activity cannot call another Activity or guarantee progress across several steps. For those pipelines, wrap your Activities in a Temporal **Workflow**, where sequencing is ordinary `await` statements and parallelism is `asyncio.gather`. See the [Temporal Python documentation](https://docs.temporal.io/develop/python) for building Workflows.
docs/guides/celery-to-standalone-activity.mdx:448
- These three internal links must use relative destinations (
/develop/python/activities/standalone-activities,/develop/python/activities/standalone-activities-quickstart, and/develop/python/activities/timeouts). Absolutedocs.temporal.iolinks fail the CI-scopedTemporal.RelativeLinksrule; lines 25, 91, 431, and 437 have the same issue.
Because Standalone Activities are in Public Preview, review the [Standalone Activities feature guide](https://docs.temporal.io/develop/python/activities/standalone-activities) for the latest API details before relying on them in production. Useful next topics include:
- The [Standalone Activities Quickstart](https://docs.temporal.io/develop/python/activities/standalone-activities-quickstart) for the runnable reference sample.
- [Activity timeouts](https://docs.temporal.io/develop/python/activities/timeouts) for tuning `start_to_close` and related limits.
docs/guides/celery-to-standalone-activity.mdx:321
- Temporal core identifiers use
Id, so change these references to “Activity Id” and “Run Id” (readme/STYLE.md:50-58). The generic “user ID” at line 276 must likewise be written as “user identifier.”
The pattern mirrors Celery precisely. `start_activity` corresponds to `.delay()` and returns a handle immediately, the way Celery returns an `AsyncResult`. Calling `handle.result()` corresponds to `AsyncResult.get()`. If you need to reconnect to an Activity from a different process — for example, a web request started it and a later request checks on it — recreate the handle from the Activity's ID and run ID (the run ID is available on the handle returned by `start_activity`):
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| Before you begin, you will need the following: | ||
|
|
||
| - Python 3.14 or higher installed on your machine. |
| Result: sent to user42@example.com | ||
| ``` | ||
|
|
||
| A few details map directly from Celery. You pass the Activity's arguments through `args=[...]`, so the single `WelcomeEmailInput` goes in a list. The `id` you provide is a business identifier you choose (an order number, a user ID). Temporal uses it to guarantee the same Activity is not started twice, which is a built-in form of deduplication. Every Activity requires a timeout — `start_to_close_timeout` caps how long one attempt may run, replacing Celery's `task_time_limit`. |
Comment on lines
+356
to
+359
| retry_policy=RetryPolicy( | ||
| maximum_attempts=5, | ||
| maximum_interval=timedelta(minutes=1), | ||
| non_retryable_error_types=["InvalidUserError"], |
| - Migration | ||
| - Celery | ||
| - Standalone Activities | ||
| - Replay |
|
|
||
| In this guide, you will migrate a Celery task to a Temporal Standalone Activity. You will convert the task into an Activity, run a Worker to process it, execute it both synchronously and fire-and-forget in place of your `.get()` and `.delay()` calls, migrate its retries to a Retry Policy, and inspect running Activities in place of Flower. By the end, you will have a working Temporal Application that reproduces the behavior of your Celery app with no Workflow code. | ||
|
|
||
| ## How Celery Concepts Map to Standalone Activities |
| |---|---|---| | ||
| | Task (`@app.task`) | Activity (`@activity.defn`) | A single unit of work (I/O, API calls) | | ||
| | Worker (`celery worker`) | Worker (activities only, no Workflows) | Process that executes your code | | ||
| | Broker + result backend | Temporal service | Durably stores queue state and results | |
|
|
||
| [Celery](https://docs.celeryq.dev/en/stable/) is a distributed task queue that runs background jobs by pushing messages through a broker (such as Redis or RabbitMQ) to a pool of worker processes. Most Celery tasks are self-contained: send one email, resize one image, call one API. For that kind of single-step job, you want Durable Execution and automatic retries without having to stand up an orchestration layer around each task. | ||
|
|
||
| [Temporal](https://docs.temporal.io/) **Standalone Activities** fit that need. A Standalone Activity is an Activity you start directly from a Temporal Client, without wrapping it in a Workflow. You get Temporal's durability, retries, timeouts, and visibility for an individual unit of work, which maps almost one-to-one onto a Celery task. Because there is no Workflow to run a single Activity, Standalone Activities also use fewer resources than orchestrating one Activity through a Workflow. |
| description: Migrate Celery tasks to a Temporal Standalone Activity. | ||
| sidebar_label: Migrate from Celery | ||
| toc_max_heading_level: 3 | ||
| author: n/a |
| Result: sent to user42@example.com | ||
| ``` | ||
|
|
||
| A few details map directly from Celery. You pass the Activity's arguments through `args=[...]`, so the single `WelcomeEmailInput` goes in a list. The `id` you provide is a business identifier you choose (an order number, a user ID). Temporal uses it to guarantee the same Activity is not started twice, which is a built-in form of deduplication. Every Activity requires a timeout — `start_to_close_timeout` caps how long one attempt may run, replacing Celery's `task_time_limit`. |
| - The Temporal CLI, version 1.7.0 or higher (installed in Step 2). | ||
| - An existing Celery task you want to migrate, or the sample task shown in Step 4 if you are following along from scratch. | ||
|
|
||
| ## Step 1 — Setting up your project directory |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Adds a Python guide, Migrate a Celery task queue to a Temporal Standalone Activity, that walks through converting a Celery task into a Standalone Activity:
delay(),AsyncResult.get(),max_retries, Flower) onto their Temporal equivalents.delay()and.get()withclient.start_activity()andclient.execute_activity()max_retries/self.retry()to a Retry Policyclient.list_activities()andclient.count_activities()in place of FlowerAlso links the page from the Guides sidebar and adds a card to the Guides landing page grid under a new
Migrationtag.Notes to reviewers
Opened as a draft: the guide documents Standalone Activities, which are
publicPreviewinsrc/constants/featureReleaseTypes.js. Happy to mark it ready once the docs team is clear to merge preview coverage for this feature.yarn buildpasses; the page and its landing page card both appear in the build output.Two follow-ups intentionally left out of this PR:
docs/guides/loyalty-points-revised.mdxis a separate untracked draft in my working tree and is not included here. It's currently an unlinked orphan and needs its own PR.author: n/ain its frontmatter, which no other guide uses. Let me know if you'd like it dropped.🤖 Generated with Claude Code
┆Attachments: EDU-6982 Add guide for migrating Celery task queues to Standalone Activities