Reject a DataSourceConfig with no uri or format in DataFusionEngine - #73287
Conversation
d73ec5f to
793bcac
Compare
LLMSQLOperator, DataFusionToolset, and AnalyticsOperator hand DataFusionEngine.register_datasource whatever DataSourceConfig the caller supplies, unlike LLMSchemaCompareOperator which checks the connection kind first. A plain-database-table config reaching any of them failed deep inside credential resolution with a confusing "Unknown connection type" error instead of a clear message. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
793bcac to
8f3a1c3
Compare
potiuk
left a comment
There was a problem hiding this comment.
The guard is correct and lands exactly where it should: storage_type is None on a non-table-provider config is, by DataSourceConfig.__post_init__'s invariants, exactly the "neither uri nor format" shape, so the check turns the misleading Unknown connection type failure from _get_credentials into a clear error before any connection lookup, and the new test fails without it.
I traced the three pass-through callers (LLMSQLOperator, DataFusionToolset, AnalyticsOperator) and every DataSourceConfig(...) in the common-ai / common-sql tests; none hands a plain-table config to a real engine, and the existing docs already describe uri/format as required for those consumers, so nothing else changes.
Smaller observations
- The commit carries a
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>trailer.AGENTS.md("Commits and PRs") says "Never list an agent as a commit co-author"; theGenerated-by:line in the description is the sanctioned disclosure. I'll drop the trailer from the squash message at merge time. - For
LLMSchemaCompareOperator, a plain-table config whoseconn_idis mistyped now reports "has no uri or format" instead of "conn_id isn't defined" (because_is_dbapi_connectionswallows the not-found error and falls through to DataFusion). Probably the right layering; just noting the message no longer points at the real cause in that one case.
This review was drafted by an AI-assisted tool and
confirmed by an Apache Airflow maintainer. The maintainer
approving this PR has read the findings and signed off. If
something feels off, please reply on the PR and a maintainer
will follow up.More on how Apache Airflow handles maintainer review:
contributing-docs/05_pull_requests.rst.
Follow-up per review feedback on #73273.
LLMSQLOperator,DataFusionToolset, andAnalyticsOperatorhandDataFusionEngine.register_datasourcewhateverDataSourceConfigthe caller supplies, unlikeLLMSchemaCompareOperatorwhich checks the connection kind first. A plain-database-table config reaching any of them failed deep inside credential resolution with a confusingUnknown connection typeerror instead of a clear message.register_datasourcenow rejects that shape immediately.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5) following the guidelines