Conversation
A pod reclaimed between the trigger firing and the worker resuming the task takes the XCom sidecar with it. Returning normally in that case reports the task as successful while pushing no return_value, so downstream tasks that template on it fail with an error that points nowhere near the cause and no retry of the downstream can ever recover. Restore the loud failure for tasks that are contracted to produce a value: only a task with do_xcom_push unset can still succeed on a vanished pod, where the pod logs are the only thing lost. closes: apache#73117
|
Hello @rjgoyln - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
Summary
A deferrable
KubernetesPodOperatorwithdo_xcom_push=Truecan be incorrectly marked asSUCCESSwhen its pod is reclaimed after the trigger fires but before the worker resumes the task.When the pod is reclaimed, its XCom sidecar is removed with it, so the XCom result never reaches Airflow. The 404 handling introduced in #66716 currently treats the missing pod as a successful completion and returns
None. Sincetrigger_reentry's return value becomes the task'sreturn_value, this leaves the task successful but without the result it was expected to produce.This can cause downstream tasks to receive an unexpected
Noneand fail during templating. Retrying the downstream task cannot recover the missing XCom; the upstream task must be cleared and rerun.Changes
do_xcom_pushis unset, since only the pod logs are lost in this case.PodNotFoundExceptionwhendo_xcom_pushis enabled, so the task can be retried and the pod — along with its XCom result — can be recreated.PodNotFoundExceptioninstead of allowing the rawApiExceptionto escape, preserving the fix from KubernetesPodOperator deferrable: trigger_reentry crashes when pod is GC'd before re-entry #66715.No opt-in flag is added; this follows the first approach proposed in the issue.
Tests
do_xcom_push=True. The test fails onmain.do_xcom_push=False, making it clear that the test covers the existing silent-success behavior.closes: #73117
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines