Allow DataSourceConfig to represent a plain database table - #73273
Conversation
|
Ran this end to end against a real Postgres backend in breeze: two real tables, a real Airflow connection, and each consumer of a plain-database The feature itself works. That path was unreachable before, since the config could not be constructed, so the Where it needs another look is
Row three is a behaviour change against current main, which raises there today. Row two costs a worse error at task runtime: With the check hoisted, the full On docs: Separately, and for a follow-up rather than this PR: three of the four |
03b26fa to
6dd30df
Compare
|
Pushed an update: hoisted the Drafted-by: Claude Code (Sonnet 5); reviewed by @pankajastro before posting |
6dd30df to
c2a634a
Compare
kaxil
left a comment
There was a problem hiding this comment.
Verified the last push independently rather than off the diff alone. Extracted config.py at apache/main and at this head by SHA and exec'd both side by side: the storage_type=LOCAL/S3-with-no-uri case that regressed in the previous push now matches main again (ok, not a raise), and the format-with-no-uri case gets the new, reachable message instead of the old Unsupported storage type for URI: on an empty string. Every other delta from main is an intended part of this PR: the bug it fixes, and the justified iceberg blank-table_name tightening backed by its own test.
Full common.sql + common.ai unit suites pass (2086 passed, 7 skipped, mysql-only), and mypy is clean on config.py and the example DAG. I could not get a clean build-docs common.ai in my session, but that's an environment artifact, not this PR's: this branch's merge-base predates main's DuckDB provider, so the container's provider-discovery metadata references a module this checkout doesn't have. Round 2's docs build was clean on nearly the same RST content, and this round's diff to it is wording only.
Also checked #73287 directly: it guards DataFusionEngine.register_datasource itself, which covers the three previously-ungated call sites at their one choke point rather than patching each individually, with a test for the plain-DB case.
LLMSchemaCompareOperator accepts a DataSourceConfig with only conn_id and table_name, introspected via DbApiHook instead of DataFusion when the connection resolves to one, but construction failed for that shape. Also hoist the table_name check so it can no longer be skipped by an explicit storage_type, and require a uri whenever format is set, since format alone can't be registered without one. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
c2a634a to
5faa8f6
Compare
…73287) Follow-up per [review feedback](#73273 (review)) on #73273. `LLMSQLOperator`, `DataFusionToolset`, and `AnalyticsOperator` hand `DataFusionEngine.register_datasource` whatever `DataSourceConfig` the caller supplies, unlike `LLMSchemaCompareOperator` which checks the connection kind first. A plain-database-table config reaching any of them failed deep inside credential resolution with a confusing `Unknown connection type` error instead of a clear message. `register_datasource` now rejects that shape immediately.
DataSourceConfigalways inferredstorage_typefromuri, which defaults to empty.LLMSchemaCompareOperatorbuilds aDataSourceConfigwith onlyconn_idandtable_name(nouri, noformat) to introspect a plain database connection viaDbApiHook, and that construction failed.Reproduction (local
airflow dags testrun):plain_db_tablewould failstorage_typestaysNone.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5) following the guidelines