Skip to content

Fix GCS operators emitting invalid OpenLineage events with no dataset name - #73246

Merged
mobuchowski merged 2 commits into
apache:mainfrom
kacpermuda:ol-fix-gcs
Sep 16, 2026
Merged

mobuchowski merged 2 commits into
apache:mainfrom
kacpermuda:ol-fix-gcs

Conversation

@kacpermuda

Copy link
Copy Markdown
Collaborator

GCSDeleteObjectsOperator and LocalFilesystemToGCSOperator build one OpenLineage Dataset per object/file without checking for falsy values. When objects/src resolves at runtime to a list containing None or "" (e.g. via XCom), or bucket_name/bucket resolves to a falsy value, the emitted event has a Dataset with no name, which violates the OpenLineage spec and gets rejected by consumers, silently dropping lineage for the whole task.

This fixes both operators to skip falsy object/file names and to return an empty OperatorLineage when the bucket is falsy, instead of emitting a malformed event.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5)

Generated-by: Claude Code (Sonnet 5) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@kacpermuda
kacpermuda requested a review from shahar1 as a code owner September 16, 2026 14:38
@boring-cyborg boring-cyborg Bot added area:providers provider:google Google (including GCP) related issues labels Sep 16, 2026
kacpermuda added a commit to kacpermuda/airflow that referenced this pull request Sep 16, 2026
Addresses review feedback on apache#73246: a None destination
path crashed the OpenLineage hook before building any dataset, instead
of the empty OperatorLineage the bucket check already returns.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
kacpermuda and others added 2 commits September 16, 2026 17:30
Addresses review feedback on apache#73246: a None destination
path crashed the OpenLineage hook before building any dataset, instead
of the empty OperatorLineage the bucket check already returns.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@mobuchowski
mobuchowski merged commit 53862c8 into apache:main Sep 16, 2026
86 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants