Fix DeadlockImminentError when a connection is resolved inside an async task - #71890
Merged
Merged
Conversation
2 tasks
2 tasks
dabla
force-pushed
the
fix/deadlock-mask-secret-get-uri
branch
from
August 20, 2026 15:12
99c8738 to
db8a1c7
Compare
…rror in async tasks get_uri() accesses extra_dejson, which calls the synchronous mask_secret() → comms.send() from within the event-loop thread. Any async hook or task that calls aget_hook() / aget_connection() triggers this path, and Airflow 3.3.1's DeadlockImminentError detection surfaces the bug. Add aextra_dejson() — an async method that awaits amask_secret() instead of the blocking mask_secret() — and aget_uri(), which delegates URI assembly to a new shared _build_uri() helper and calls await self.aextra_dejson(). This keeps the entire connection-serialisation path safely on the async stack. The sync get_uri() / extra_dejson are unchanged; _build_uri() is the single source of truth for the URI format, shared by both paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
dabla
force-pushed
the
fix/deadlock-mask-secret-get-uri
branch
from
August 20, 2026 19:29
db8a1c7 to
9663c6a
Compare
1 task
amoghrajesh
reviewed
Sep 10, 2026
amoghrajesh
left a comment
Contributor
There was a problem hiding this comment.
Directionally fine, I have a few qns / todos here.
- Can we also update the doc: https://airflow.apache.org/docs/task-sdk/stable/deferred-vs-async-operators.html? The example mentions
hook = KiotaRequestAdapterHook.get_hook(conn_id="msgraph_default") - Should we be also covering the other methods like
to_dict,get_extra_dejsonhere?
kaxil
reviewed
Sep 10, 2026
kaxil
approved these changes
Sep 10, 2026
Co-authored-by: Amogh Desai <amoghrajesh1999@gmail.com>
Contributor
Author
Don't know if we still want to implement an async version of |
1 task
vatsrahul1001
added a commit
that referenced
this pull request
Sep 10, 2026
…nc task (#71890) (#72895) * Add aget_uri and aextra_dejson to Connection to fix DeadlockImminentError in async tasks get_uri() accesses extra_dejson, which calls the synchronous mask_secret() → comms.send() from within the event-loop thread. Any async hook or task that calls aget_hook() / aget_connection() triggers this path, and Airflow 3.3.1's DeadlockImminentError detection surfaces the bug. Add aextra_dejson() — an async method that awaits amask_secret() instead of the blocking mask_secret() — and aget_uri(), which delegates URI assembly to a new shared _build_uri() helper and calls await self.aextra_dejson(). This keeps the entire connection-serialisation path safely on the async stack. The sync get_uri() / extra_dejson are unchanged; _build_uri() is the single source of truth for the URI format, shared by both paths. (cherry picked from commit d057784) Co-authored-by: David Blain <info@dabla.be> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
imrichardwu
pushed a commit
to imrichardwu/airflow
that referenced
this pull request
Sep 11, 2026
…nc task (apache#71890) * Add aget_uri and aextra_dejson to Connection to fix DeadlockImminentError in async tasks get_uri() accesses extra_dejson, which calls the synchronous mask_secret() → comms.send() from within the event-loop thread. Any async hook or task that calls aget_hook() / aget_connection() triggers this path, and Airflow 3.3.1's DeadlockImminentError detection surfaces the bug. Add aextra_dejson() — an async method that awaits amask_secret() instead of the blocking mask_secret() — and aget_uri(), which delegates URI assembly to a new shared _build_uri() helper and calls await self.aextra_dejson(). This keeps the entire connection-serialisation path safely on the async stack. The sync get_uri() / extra_dejson are unchanged; _build_uri() is the single source of truth for the URI format, shared by both paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
xvega
pushed a commit
to xvega/airflow
that referenced
this pull request
Sep 13, 2026
…nc task (apache#71890) * Add aget_uri and aextra_dejson to Connection to fix DeadlockImminentError in async tasks get_uri() accesses extra_dejson, which calls the synchronous mask_secret() → comms.send() from within the event-loop thread. Any async hook or task that calls aget_hook() / aget_connection() triggers this path, and Airflow 3.3.1's DeadlockImminentError detection surfaces the bug. Add aextra_dejson() — an async method that awaits amask_secret() instead of the blocking mask_secret() — and aget_uri(), which delegates URI assembly to a new shared _build_uri() helper and calls await self.aextra_dejson(). This keeps the entire connection-serialisation path safely on the async stack. The sync get_uri() / extra_dejson are unchanged; _build_uri() is the single source of truth for the URI format, shared by both paths. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a
DeadlockImminentErrorraised in Airflow 3.3.1 when a connection is resolved inside an async task viaawait BaseHook.aget_hook(...)orawait BaseHook.aget_connection(...).Problem
The call chain when fetching a connection inside an async task is:
The error message says "use the async equivalents" — but the caller already is.
The bug is inside
Connectionitself:get_uri()andextra_dejsonhave no async counterparts, so secret masking always goes through the blockingcomms.send().amask_secret()(which usesasend()) already existed inairflow.sdk.logbut nothing in
Connectioncalled it.Changes
task-sdk/src/airflow/sdk/definitions/connection.py_build_uri(self, extra_dejson: dict)helper so sync and async paths share a single implementation.get_uri()toreturn self._build_uri(self.extra_dejson)— behaviour is identical to before.aextra_dejson()— async method that mirrorsextra_dejsonbut awaitsamask_secret()instead of calling the blockingmask_secret().aget_uri()— async method that callsawait self.aextra_dejson()and delegates to_build_uri().task-sdk/tests/task_sdk/definitions/test_connection.pytest_aget_uri—aget_uri()produces the same URI asget_uri()and delegates toaextra_dejson.test_aextra_dejson_calls_amask_secret—aextra_dejson()usesamask_secret(async) and never calls the syncmask_secret.test_aextra_dejson_no_extra—aextra_dejson()returns{}without callingamask_secretwhenextraisNone.Usage
Was generative AI tooling used to co-author this PR?
Claude Sonnet 4.6 (GitHub Copilot)
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.