Skip to content

Fix Dataflow sensors losing their XCom value when not deferred - #71086

Merged
shahar1 merged 2 commits into
apache:mainfrom
FrankYang0529:airflow-dataflow-sensor-return-value
Sep 17, 2026
Merged

shahar1 merged 2 commits into
apache:mainfrom
FrankYang0529:airflow-dataflow-sensor-return-value

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Why

  • DataflowJobMessagesSensor, DataflowJobAutoScalingEventsSensor and DataflowJobMetricsSensor return their payload to XCom when deferred, but drop it in poke mode. deferrable defaults to False, so the default path is the broken one.

  • The payload travels through a chain, and these sensors break it at the last step:

    poke()                          →  PokeReturnValue(is_done=True, xcom_value=X)
    BaseSensorOperator.execute()    →  picks X out and returns it
    subclass execute()              →  must return it upwards        ← dropped here
    task runner                     →  pushes it as the return_value XCom
    

How


Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Aug 4, 2026
@FrankYang0529
FrankYang0529 marked this pull request as ready for review August 4, 2026 14:01
@FrankYang0529
FrankYang0529 requested a review from shahar1 as a code owner August 4, 2026 14:01
@FrankYang0529
FrankYang0529 force-pushed the airflow-dataflow-sensor-return-value branch 4 times, most recently from 7a15528 to 79e0ff2 Compare August 6, 2026 00:29
@potiuk potiuk added the ready for maintainer review Set after triaging when all criteria pass. label Aug 13, 2026
@molcay

molcay commented Sep 13, 2026 •

Copy link
Copy Markdown
Contributor

Hi @FrankYang0529,

Thank you for your change. It is very much appreciated!

From my side, it seems a good change. However we want to see the system tests run results before merging a change.

Did you run the system tests? Can you share the results of the system tests?

Signed-off-by: PoAn Yang <payang@apache.org>
Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 force-pushed the airflow-dataflow-sensor-return-value branch 2 times, most recently from 79e0ff2 to fb97f4b Compare September 14, 2026 13:08
@FrankYang0529

Copy link
Copy Markdown
Member Author

Did you run the system tests? Can you share the results of the system tests?

@molcay Thanks for the suggestion. I add a xcom assertion to system test and make sure it passes on this branch, but fails on main branch.

Screenshot 2026-09-14 at 9 57 04 PM

@shahar1
shahar1 merged commit 11a3892 into apache:main Sep 17, 2026
86 checks passed
@FrankYang0529
FrankYang0529 deleted the airflow-dataflow-sensor-return-value branch September 17, 2026 11:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues ready for maintainer review Set after triaging when all criteria pass.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants