Skip to content

Keep MSGraph pagination offset across paginated pages - #71986

Merged
dabla merged 1 commit into
apache:mainfrom
FrankYang0529:msgraph-pagination-keyerror
Sep 8, 2026
Merged

dabla merged 1 commit into
apache:mainfrom
FrankYang0529:msgraph-pagination-keyerror

Conversation

@FrankYang0529

@FrankYang0529 FrankYang0529 commented Aug 23, 2026 •

Copy link
Copy Markdown
Member

Why

  • default_pagination() recovered the $skip offset with sum(len(result["value"]) for result in responses()).
  • A user can define result_processor as lambda result, **context: result.get("value"). append_result() then extends the accumulator with the page's items, so XCom holds bare messages and those have no "value" key. On the deferrable path responses() reads exactly that XCom, so the run dies with KeyError: 'value' once a second full page arrives.
  • Every page runs in a new worker process that rebuilds the operator from the serialized Dag, so self.query_parameters has only what the Dag file declares, not the $skip computed for the previous page. XCom was the only place the operator could read that offset back from.
  • paginated_run() also recovered $skip by summing the page lengths in responses. The responses already held the current page when pagination_function ran, so that sum was too large. The correct offset for the request after page 1 was $skip=12, but the operator sent $skip=24, so page 2's 12 records were never fetched.

How

  • Derive $skip from the query parameters the page was requested with, so the offset no longer depends on what a user callback left in XCom. The same expression fixes the off-by-one page in paginated_run() and keeps a user-supplied starting offset.
  • Carry those query parameters across the defer boundary with defer(kwargs=...) and restore them in execute_complete().

Verification

  • uv run --project providers/microsoft/azure --frozen pytest providers/microsoft/azure/tests/unit -q

Was generative AI tooling used to co-author this PR?
  • Yes - Claude Code

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@FrankYang0529
FrankYang0529 marked this pull request as ready for review August 24, 2026 10:44
@FrankYang0529
FrankYang0529 requested a review from dabla as a code owner August 24, 2026 10:44
Signed-off-by: PoAn Yang <payang@apache.org>
@FrankYang0529
FrankYang0529 force-pushed the msgraph-pagination-keyerror branch from e75c5ae to 0daece3 Compare August 31, 2026 05:53
@dabla
dabla merged commit 77cf38b into apache:main Sep 8, 2026
88 checks passed
@dabla

dabla commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Nice one thx @FrankYang0529 !

@FrankYang0529
FrankYang0529 deleted the msgraph-pagination-keyerror branch September 8, 2026 23:35
imrichardwu pushed a commit to imrichardwu/airflow that referenced this pull request Sep 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants