Data Lineage Not Added for New Table Creation in Spark

[dntjr8096]

Description:
When creating a new Impala table using Spark with existing Impala tables, the data lineage is not being added. It seems that the toEntity in the code doesn’t retrieve information for the new table.

Expected Behavior:
in general, if it's a new table, I thought it would be appropriate to add a table to the data service (impala here) inside open metadata and data lineage should be automatically added for the new Impala table created in Spark, ensuring complete metadata lineage.

Actual Behavior:
No data lineage is added for the new table, resulting in incomplete metadata lineage.

Slack Link

• https://openmetadata.slack.com/archives/C02B6955S4S/p1714439693952469?thread_ts=1714439693.952469&cid=C02B6955S4S

Additional Information:

• Environment: YARN
• Openmetadata Configs: 1.3x
• OpenMetadata-spark-agent : 1.0.0-beta
• Spark Version: 3.2.3
• Impala Version: 4.1.1