Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
71c4728 to
c1d5460
Compare
match_glob is a template field, so its value is not available until Jinja rendering happens on the worker, well after __init__ has run. The guard that rejects match_glob on an unsupported google provider tested it for truthiness, which the validate-operators-init prek hook flags. The check only asks whether the argument was supplied; it never inspects the value. Per the false-positive guidance on apache#70296 that makes it a provision check, which is fixed in place rather than moved to execute(). Moving it would break the check under render_template_as_native_obj=True, where a supplied field can render to None and so becomes indistinguishable from an omitted one, and would defer a static authoring mistake from Dag parse time to every task instance and retry. Switching to an explicit is not None comparison changes one case: match_glob="" is now rejected on a google provider older than 10.3.0, where truthiness previously read the supplied empty string as absent. An empty glob is not a valid pattern and was never honoured on those versions. Narrow the raised exception from AirflowException to ValueError, which is what an invalid argument warrants, and drop the file from generated/known_airflow_exceptions.txt accordingly. This follows the merged apache#70359 precedent for S3DeleteObjectsOperator, which made the same one-for-one change to the same four kinds of file. Also remove the class from the prek exemption list, since the hook fails on stale entries, and add tests covering supplied, empty-string and omitted match_glob. Related: apache#70296
c1d5460 to
0d4fcac
Compare
is not None for GCSToS3Operator match_glob provision check
is not None for GCSToS3Operator match_glob provision check|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Backport failed to create: v3-3-test. View the failure log Run detailsNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
You can attempt to backport this manually by running: cherry_picker e8ae850 v3-3-testThis should apply the commit to the v3-3-test branch and leave the commit in conflict state marking After you have resolved the conflicts, you can continue the backport process by running: cherry_picker --continueIf you don't have cherry-picker installed, see the installation guide. |
…ache#71591) match_glob is a template field, so its value is not available until Jinja rendering happens on the worker, well after __init__ has run. The guard that rejects match_glob on an unsupported google provider tested it for truthiness, which the validate-operators-init prek hook flags. The check only asks whether the argument was supplied; it never inspects the value. Per the false-positive guidance on apache#70296 that makes it a provision check, which is fixed in place rather than moved to execute(). Moving it would break the check under render_template_as_native_obj=True, where a supplied field can render to None and so becomes indistinguishable from an omitted one, and would defer a static authoring mistake from Dag parse time to every task instance and retry. Switching to an explicit is not None comparison changes one case: match_glob="" is now rejected on a google provider older than 10.3.0, where truthiness previously read the supplied empty string as absent. An empty glob is not a valid pattern and was never honoured on those versions. Narrow the raised exception from AirflowException to ValueError, which is what an invalid argument warrants, and drop the file from generated/known_airflow_exceptions.txt accordingly. This follows the merged apache#70359 precedent for S3DeleteObjectsOperator, which made the same one-for-one change to the same four kinds of file. Also remove the class from the prek exemption list, since the hook fails on stale entries, and add tests covering supplied, empty-string and omitted match_glob. Related: apache#70296
match_globis inGCSToS3Operator.template_fields, so the constructor only ever sees the un-rendered Jinja expression, never the value the DAG author meant. The guard that rejectsmatch_globon an unsupported google provider tested it for truthiness:which is what
validate-operators-initflags, and why the class is on the exemption list.The check stays in
__init__. It is a provision check, not a value check: it asks whethermatch_globwas supplied, combined with an environment capability the constructor can already answer for itself. It never inspects the value. Per the false-positives section of #70296 those belong in the constructor and get rewritten in place rather than moved — withrender_template_as_native_obj=Truea provided field can render toNone, so the same check inexecute()would report a supplied argument as missing; and raising at construction surfaces a static authoring mistake as a Dag import error rather than once per task instance and per retry.Since #70505 narrowed the hook to sanction
is None/is not Nonereads, the in-place rewrite passes and the exemption entry still goes.What this changes
match_glob is not NoneGCSToS3Operatorfromscripts/ci/prek/validate_operators_init_exemptions.txt— the hook fails on stale exemptions, so this has to land in the same commit as the fixBehavioural impact
One cell changes, on google providers older than 10.3.0:
match_globNone)"**/*.csv"""match_glob=""was supplied by the user and truthiness read it as absent. An empty glob is not a valid pattern, and on a provider below 10.3.0 it was never going to be honoured anyway — it would have been passed to aGCSHook.list()call that cannot accept it. This is the stricter direction that05_pull_requests.rstsanctions for provision checks.Relationship to #70723
Flagging this up front: #70723 is open against the same entry and reaches the same one-line fix. I worked this independently and only found that PR afterwards, so this is not a deliberate competing implementation — I'm raising it because the two diffs are close enough that a reviewer deserves to know rather than discover it.
Where they differ:
AirflowExceptiontoValueErrorand drops the now-stalegcs_to_s3.py::1entry fromgenerated/known_airflow_exceptions.txt, per the Keep S3DeleteObjectsOperator validation in __init__, narrow to ValueError #70359 precedent. The production change is now equivalent to Check GCSToS3Operator match_glob support after template rendering #70723's.If maintainers prefer #70723 as the base — which is reasonable, it is older and more complete on the exception-type question — I am happy for this to be closed and will offer the missing test case there instead. I would rather that than have two near-identical PRs consuming review time.
Deliberately not changed
except ImportError.gcs_to_s3.pyalready does a module-levelfrom airflow.providers.google.cloud.hooks.gcs import GCSHook, so theexcept ImportErroraround the function-levelfrom airflow.providers.google import __version__in__init__cannot fire — the module import would have failed first. Real, but a separate concern.flatten_structure/keep_directory_structurewarning. Reads two fields that are not template fields, so the rule does not apply and the hook does not flag it.Verification
Run in a WSL checkout, from
providers/amazon:No newsfragment:
providers/amazonhas nonewsfragments/directory, andproviders/AGENTS.mdsays never to use them for providers.related: #70296
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines