dbt-athena: Rework iceberg full refresh for high availability

[alex-antonison]

resolves #458
docs dbt-labs/docs.getdbt.com/#

Problem

When dbt-athena does a full refresh on an Iceberg table, it deletes the table and then re-creates the table. This results in the Iceberg table being unavailable for however long it takes to full refresh the table. For Iceberg tables that take longer to produce, this is very problematic as it causes significant amount of downtime.

Solution

Instead of dropping the Iceberg at the beginning of the full refresh, instead first check if the Iceberg table already exists and if so, create the new full refresh table as a tmp relation and then upon completion, swap out the full refreshed Iceberg table for the existing Iceberg table. This approach adds in high availability support when doing a full refresh on an Iceberg table.

For this to work, it requires using s3_data_naming set to schema_table_unique as in order to prevent the s3 locations being inconsistent, it will create the new tmp_relation within the same s3 path using a UUID to keep them separate. Once the new fully refreshed tmp_relation is completed, can then simply use renaming to change the tmp_relation to the target_relation.

Checklist

• I have read the contributing guide and understand what's expected of me
• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

[github-actions]

Thank you for your pull request! We could not find a changelog entry for this change in the dbt-athena package. For details on how to document a change, see the Contributing Guide.

[parsable-alex-antonison]

Realized there is an issue with my renaming approach where the underlying s3 location is not being updated. Need to revisit my approach.

[alex-antonison]

With the help of @nicor88 I was able to come up with a solution to the s3 paths being inconsistent.

The solution does require that s3_data_naming is schema_table_unique because it relies on the table Iceberg s3 data being stored within a unique UUID location within the same s3 location as the existing table. With this approach, I can fully materialize the existing incremental iceberg table, then do a table name switch, and then drop the existing table relation.

[nicor88]

you might want to do a similar check to what was done here

('unique' in s3_data_naming and external_location is none)

pretty much we allow this feature by default for all unique location, and location that are not explicitly set via external_location.

[nicor88]

The approach that you proposed looks good.

It will be good to add functional test to this feature to verify systematically that all runs fine.