Stop failing reschedule-mode sensors on a stale executor success

[rjgoyln]

Summary

A sensor running with mode="reschedule" is intermittently marked failed when the executor success for one poke reaches the scheduler after the task instance has already gone back to scheduled (or queued) for the next poke. The scheduler reads the late event as a task that finished without recording its own state, and fails it:

Executor CeleryExecutor(parallelism=512) reported that the task instance
<TaskInstance: <dag_id>.<sensor_task_id> ... [scheduled] ti_id=...>
finished with state success, but the task instance's state attribute is scheduled.

The existing guards for this race all assume the defer path. A reschedule exit leaves next_method unset and keeps the same try_number, so the executor key still matches and none of them apply. A reschedule row for the current try is the signal that separates the two cases: it says the worker exited up_for_reschedule rather than being killed. Anything with no such row still takes the externally-killed branch.

It is resolved once per event batch alongside the bulk task-instance fetch, so the per-event path gains no query.

closes: #71172

---

Was generative AI tooling used to co-author this PR?

• Yes — Claude Code (Opus 5)

Generated-by: Claude Code (Opus 5) following the guidelines

[steveahnahn]

Verified on Postgres with the source reverted to main and the new test kept, both params reproduce the bug (sensor marked failed) and with the fix both pass.

idx_task_reschedule_ti_id_id_desc covers the batched ti_id lookup, so it stays index-only as reschedule volume grows LGTM.

[rjgoyln]

Verified on Postgres with the source reverted to main and the new test kept, both params reproduce the bug (sensor marked failed) and with the fix both pass.

idx_task_reschedule_ti_id_id_desc covers the batched ti_id lookup, so it stays index-only as reschedule volume grows LGTM.

Thanks for review! Really appreciate it.

[potiuk]

Hello @rjgoyln - thank you for your contributions to Apache Airflow!

The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:

• Support OpenTelemetry auto-instrumentation and vendor distros #70758 - Support OpenTelemetry auto-instrumentation and vendor distros
• Honor OTEL_PROPAGATORS when propagating trace context across Airflow #70840 - Honor OTEL_PROPAGATORS when propagating trace context across Airflow
• Fix API client errors losing detail behind a reverse proxy #70936 - Fix API client errors losing detail behind a reverse proxy
• Fix airflowctl config lint filters matching partial config names #70940 - Fix airflowctl config lint filters matching partial config names
• Stop failing reschedule-mode sensors on a stale executor success #71194 - Stop failing reschedule-mode sensors on a stale executor success
• Drop the cncf.kubernetes provider dependency from airflow-core tests #71868 - Drop the cncf.kubernetes provider dependency from airflow-core tests
• Drop the git provider dependency from airflow-core tests #72041 - Drop the git provider dependency from airflow-core tests
• Fail deferred sensors on non-timeout errors when soft_fail is set #72053 - Fail deferred sensors on non-timeout errors when soft_fail is set
• Drop the celery provider dependency from airflow-core tests #72249 - Drop the celery provider dependency from airflow-core tests
• Honor config_file in deferrable Kubernetes tasks #72301 - Honor config_file in deferrable Kubernetes tasks
• Fix Dataproc job not cancelled when killing a deferred task #72591 - Fix Dataproc job not cancelled when killing a deferred task
• Stop the scheduler deadlocking with the triggerer on deferred-task timeouts #72595 - Stop the scheduler deadlocking with the triggerer on deferred-task timeouts
• Make the scheduler's HITL and orphan sweeps retry transient DB errors #72596 - Make the scheduler's HITL and orphan sweeps retry transient DB errors
• Keep the triggerer running when a trigger constructor raises #72758 - Keep the triggerer running when a trigger constructor raises
• Stop failing scheduled tasks when a serialized Dag is briefly missing #72814 - Stop failing scheduled tasks when a serialized Dag is briefly missing
• Stop fetching the run conf once per task instance in the scheduler #73022 - Stop fetching the run conf once per task instance in the scheduler
• Catch format-string arguments passed to exception constructors #73076 - Catch format-string arguments passed to exception constructors
• Fail deferrable KPO when a GC'd pod takes its XCom with it #73131 - Fail deferrable KPO when a GC'd pod takes its XCom with it
• Block new EM102 f-string exception messages in prek #73166 - Block new EM102 f-string exception messages in prek
• Fail the local yml mounts prek hook when synchronization fails #73170 - Fail the local yml mounts prek hook when synchronization fails
• Fix provider.yaml doc URL check with several suspended providers #73204 - Fix provider.yaml doc URL check with several suspended providers

These pull requests stay open because maintainers are already engaged in them - they count towards your limit:

• Batch upstream map-length lookups when resolving mapped kwargs #70620 - Batch upstream map-length lookups when resolving mapped kwargs
• Re-anchor custom queued deadline references when a Dag run is cleared #71850 - Re-anchor custom queued deadline references when a Dag run is cleared
• Make the default task failure email say what failed #73221 - Make the default task failure email say what failed

This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are.

What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or gh pr reopen <PR_NUMBER> --repo apache/airflow. Reopen the ones you are ready to follow through - keep them rebased, respond to review comments and fix failing checks.

While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack.

Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst.

---

Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting