Fail deferrable KPO instead of silent SUCCESS when a GC'd pod takes its XCom with it - #73749
Open
developer-rpai wants to merge 1 commit into
Open
developer-rpai wants to merge 1 commit into
developer-rpai wants to merge 1 commit into
Conversation
A deferrable KubernetesPodOperator with do_xcom_push=True was silently marked SUCCESS when its pod was garbage-collected between the trigger firing and task re-entry: the 404 handler in trigger_reentry() returned None for any "success" event, so no return_value XCom was ever pushed and downstream tasks failed unrecoverably at templating time. Keep the existing silent-success behavior when do_xcom_push is off (only pod logs are lost there). When XCom was expected, raise PodNotFoundException so the task fails/retries and recreates the pod along with its XCom sidecar. Fixes: apache#73117 Signed-off-by: Rakesh Ramakrishna Pai <developer-rpai@users.noreply.github.com>
developer-rpai
requested review from
hussein-awala,
jedcunningham and
jscheffl
as code owners
September 26, 2026 06:02
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A deferrable
KubernetesPodOperatorwithdo_xcom_push=Truecan be marked SUCCESS without pushing anyreturn_valueXCom when the pod is garbage-collected between the trigger firing and the worker resuming the task. Downstream tasks then fail unrecoverably at templating time (TypeError: the JSON object must be str, bytes or bytearray, not NoneType); retrying the downstream task can never recover because the upstream XCom does not exist.Root cause
In
trigger_reentry()(providers/cncf/kubernetes/operators/pod.py), the 404 handler added by #66716 returns silently whenever the trigger event status is"success"— regardless ofdo_xcom_push. The trigger observed the main container finish, but the XCom sidecar died with the GC'd pod, so the returnedNonebecomes a successful task with no XCom.Fix
Keep the silent-success path only when
do_xcom_pushis off (there, only pod logs are lost). When XCom was expected, raisePodNotFoundExceptioninstead, so the task fails (and retries, when configured) and recreates the pod along with its XCom sidecar — mirroring the existing behavior for non-success events.Note: a prior fix attempt (#73131) was closed unmerged during today's enforcement of the new 5-open-PR limit for non-committers, before any maintainer review. This PR re-implements the same approach (fail when
do_xcom_pushis set) with its own regression test.Tests
test_async_trigger_reentry_raises_pod_not_found_on_success_when_xcom_push: GC'd pod + success event +do_xcom_push=True->PodNotFoundException. Fails onmainbefore this change.test_async_trigger_reentry_returns_when_pod_gcd_on_successto pindo_xcom_push=False, making the preserved silent-success contract explicit.if event["status"] == "success": returnpattern before the fix and thedo_xcom_pushguard after.py_compilepasses on both touched files.Impact
Correctness: prevents silent data loss (missing XCom) and a wrong task status (SUCCESS for a task whose result was destroyed). No behavior change when
do_xcom_pushisFalse.Fixes: #73117
Was generative AI tooling used to co-author this PR?
Generated-by: Muse (Meta) following the guidelines