Skip to content

Add Azure Analysis Services model refresh support - #71350

Merged
dabla merged 8 commits into
apache:mainfrom
aaron-y-chen:issue-51377-aas-refresh
Sep 7, 2026
Merged

dabla merged 8 commits into
apache:mainfrom
aaron-y-chen:issue-51377-aas-refresh

Conversation

@aaron-y-chen

@aaron-y-chen aaron-y-chen commented Aug 10, 2026

Copy link
Copy Markdown
Contributor

closes: #51377

Summary

  • Add an azure_analysis_services connection type using Microsoft Entra service-principal client-secret authentication.
  • Add a Hook, Operator, Sensor, and Trigger for starting and monitoring model refreshes in synchronous, fire-and-forget, and deferrable modes.
  • Add request and polling timeouts, response validation, diagnostic HTTP errors, provider metadata, documentation, and a system-test Dag.

Testing

Run the system-test Dag in Breeze against a live Azure Analysis Services instance:

  • Azure Analysis Services D1 Developer tier in West US
  • Compatibility-level-1200 model
  • Dedicated Microsoft Entra service principal and client-secret Airflow connection
  • Two calculate refreshes

The Dag runs a fire-and-forget Operator, a deferrable-enabled Sensor, and a deferrable-enabled Operator. All three task instances complete successfully on their first try.

51377_1

Verify both refresh IDs returned by Airflow through the Azure Analysis Services REST API:

Refresh ID Type Status Duration
437aa86c-…-8169e calculate succeeded 0.206 s
d9f4974e-…-9020 calculate succeeded 0.179 s

The Azure screenshot confirms the D1 resource, West US location, and compatibility-level-1200 model.

51377_2

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: [GPT 5.6-sol] following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@aaron-y-chen
aaron-y-chen marked this pull request as ready for review August 10, 2026 04:31

@dabla dabla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR adds a well-structured set of components (Hook, Operator, Sensor, Trigger) for Azure Analysis Services model refreshes. The architecture is sound: the trigger correctly offloads synchronous HTTP calls to asyncio.to_thread, deferrable mode is properly implemented, and the test suite is thorough.

Two items must be fixed before merge:

  1. _get_base_url allows userinfo and port injection (CWE-918). The host validation explicitly guards against path/query/fragment but silently accepts user@host (userinfo) and host:port inputs, which can redirect the AAS bearer token to an attacker-controlled server. Adding or parsed_host.userinfo or parsed_host.port to the guard closes the gap. A test case should also be added.

  2. requests must not be added as a dependency. httpx is already provided by the provider's transitive dependency on kiota-http and is the established HTTP client in this provider. The two requests.get / requests.post calls are straightforward to replace with the httpx sync API — see the inline comment for the exact diff.


Drafted-by: Claude Sonnet 4.6 (claude-sonnet-4.6); reviewed by @dabla before posting

Comment thread providers/microsoft/azure/pyproject.toml Outdated
@aaron-y-chen

Copy link
Copy Markdown
Contributor Author

I learned a lot from this round of reviews, many thanks 😄

@aaron-y-chen
aaron-y-chen force-pushed the issue-51377-aas-refresh branch from 3e858bd to c52856f Compare August 26, 2026 07:18

@dabla dabla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The design is solid: a purely async, deferred-only Hook/Operator/Sensor/Trigger set for Azure Analysis Services refreshes, correctly avoiding worker-slot blocking, with httpx.AsyncClient used natively (no asyncio.to_thread workaround needed), proper aclose() resource cleanup of both the HTTP client and the credential, and thorough test coverage including full deferred-lifecycle execution via execute_operator. The two concerns raised in earlier review rounds (_get_base_url userinfo/port injection, and adding requests as a redundant dependency alongside httpx) have both been correctly addressed in later commits (_assert_host now rejects credentials/ports in the netloc, and the hook is httpx-only). _get_headers correctly builds the Authorization header as an OAuth bearer credential using the fetched token's .token attribute.

One design suggestion (see comment [1]): move the trigger's polling loop into an async hook method, mirroring the split used in #72336 for SFTPTransferTrigger/SFTPHookAsync.transfer(). This keeps run() a thin dispatcher and makes the polling logic independently testable. Not a blocker - a worthwhile follow-up polish either in this PR or a fast-follow.

@dabla dabla left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well done @aaron-y-chen !

@dabla
dabla merged commit 328730c into apache:main Sep 7, 2026
84 checks passed
pierrejeambrun pushed a commit to astronomer/airflow that referenced this pull request Sep 8, 2026
* Add Azure Analysis Services model refresh support
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
* Add Azure Analysis Services model refresh support
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure Virtual Analysis Services Operator

2 participants