Dbt 1.8 migration (#244)

### Summary dbt-core migration to 1.8.* version ### Description Summarisation of 2 PRs to migrate to dbt-core 1.8.* version ### Test Results I'm expecting it as a baseline for future improvements. Tests statistics after migration to dbt 1.8 (with CE Dremio on-prem version): tests: - caching: passing - component: 2 out of 4 are passing (these tests rely on both DC and Software dremio versions) - functional: - basic: passing with ignored snapshot tests. - dbt_clone: failing - dremio_specific: passing - empty: failing - grants: failing - query_comment: passing - relation: 1 out of 2 are passing - unit_testing: failing - utils: passing - hooks: 7 out of 12 are passing - simple_copy: passing - unit: passing ### Changelog - [x] migration to dbt-core 1.8 is finished - [x] some tests are failing still --------- Co-authored-by: Ravjot Brar <[email protected]>
dremio · Nov 21, 2024 · 6b65f44 · 6b65f44
1 parent 424cdd0
commit 6b65f44
Show file tree

Hide file tree

Showing 49 changed files with 414 additions and 138 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,4 +1,12 @@
-# dbt-dremio MAIN
+# dbt-dremio v1.8.0
+
+## Changes
+
+## Dependency
+
+-   [#222](https://github.com/dremio/dbt-dremio/issues/222) Upgrade dbt-core to 1.8.8 and dbt-tests-adapter to 1.8.0
+
+## Features
 
 -   [#223](https://github.com/dremio/dbt-dremio/issues/224) Implement merge strategy for incremental materializations
 -   [#229](https://github.com/dremio/dbt-dremio/issues/229) Add max operator to get_relation_last_modified macro 

diff --git a/README.md b/README.md
@@ -6,20 +6,20 @@ dbt is the T in ELT. Organize, cleanse, denormalize, filter, rename, and pre-agg
 
 [Dremio docs for our dbt adapter ](https://github.com/dremio/dbt-dremio/wiki/Using-Materializations-with-Dremio)
 
-## dbt-dremio version 1.7.0
-
 The `dbt-dremio` package contains all of the code enabling dbt to work with [Dremio](https://www.dremio.com/). For more information on using dbt with Dremio, consult [the docs](https://docs.getdbt.com/reference/warehouse-profiles/dremio-profile).
 
 The dbt-dremio package supports both Dremio Cloud and Dremio Software (versions 22.0 and later).
 
-Version 1.7.0 of the dbt-dremio adapter is compatible with dbt-core versions 1.2.0 to 1.7.*.
+## dbt-dremio version 1.8.0
+
+Version 1.8.0 of the dbt-dremio adapter is compatible with dbt-core versions 1.8.*.
 
 > Prior to version 1.1.0b, dbt-dremio was created and maintained by [Fabrice Etanchaud](https://github.com/fabrice-etanchaud) on [their GitHub repo](https://github.com/fabrice-etanchaud/dbt-dremio). Code for using Dremio REST APIs was originally authored by [Ryan Murray](https://github.com/rymurr). Contributors in this repo are credited for laying the groundwork and maintaining the adapter till version 1.0.6.5. The dbt-dremio adapter is maintained and distributed by Dremio starting with version 1.1.0b.
 
 ## Getting started
 
 -   [Install dbt-dremio](https://docs.getdbt.com/reference/warehouse-setups/dremio-setup)
-    -   Version 1.7.0 of dbt-dremio requires dbt-core >= 1.2.0 and <= 1.7.*. Installing dbt-dremio will automatically upgrade existing dbt-core versions earlier than 1.2.0 to 1.7.*, or install dbt-core v1.7.0 if no version of dbt-core is found.
+    -   Version 1.8.0 of dbt-dremio requires dbt-core >= 1.8.*.
 -   Read the [introduction](https://docs.getdbt.com/docs/introduction/) and [viewpoint](https://docs.getdbt.com/docs/about/viewpoint/)
 
 ## Join the dbt Community

diff --git a/THIRD_PARTY_LICENSES.txt b/THIRD_PARTY_LICENSES.txt
@@ -13,10 +13,10 @@
  charset-normalizer        3.1.0      MIT License
  click                     8.1.3      BSD License
  colorama                  0.4.6      BSD License
- dbt-core                  1.7.0      Apache Software License
- dbt-dremio                1.7.0      Apache Software License
+ dbt-core                  1.8.8      Apache Software License
+ dbt-dremio                1.8.0      Apache Software License
  dbt-extractor             0.4.1      Apache Software License
- dbt-tests-adapter         1.7.0      Apache Software License
+ dbt-tests-adapter         1.8.0      Apache Software License
  exceptiongroup            1.1.1      MIT License
  future                    0.18.3     MIT License
  grpclib                   0.4.3      BSD License
@@ -42,6 +42,7 @@
  pyrsistent                0.19.3     MIT License
  pytest                    7.2.2      MIT License
  python-dateutil           2.8.2      Apache Software License; BSD License
+ python-dotenv             1.0.1      BSD License
  python-slugify            8.0.1      MIT License
  pytimeparse               1.1.8      MIT License
  pytz                      2022.7.1   MIT License

diff --git a/dbt/adapters/dremio/__version__.py b/dbt/adapters/dremio/__version__.py
@@ -9,5 +9,4 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-version = "1.7.0"
-
+version = "1.8.0"
diff --git a/dbt/adapters/dremio/api/authentication.py b/dbt/adapters/dremio/api/authentication.py
@@ -17,7 +17,7 @@
 from typing import Optional
 from abc import abstractmethod
 
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 

diff --git a/dbt/adapters/dremio/api/cursor.py b/dbt/adapters/dremio/api/cursor.py
@@ -25,7 +25,7 @@
 )
 from dbt.adapters.dremio.api.parameters import Parameters
 
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 
@@ -39,11 +39,20 @@ def __init__(self, api_parameters: Parameters):
         self._rowcount = -1
         self._job_results = None
         self._table_results: agate.Table = None
+        self._description = None
 
     @property
     def parameters(self):
         return self._parameters
 
+    @property
+    def description(self):
+        return self._description
+
+    @description.setter
+    def description(self, value):
+        self._description = value
+
     @property
     def closed(self):
         return self._closed
@@ -126,7 +135,6 @@ def _populate_rowcount(self):
 
         while True:
             time.sleep(0.2)
-
             if job_status_state != last_job_state:
                 logger.debug(f"Job State = {job_status_state}")
 

diff --git a/dbt/adapters/dremio/api/handle.py b/dbt/adapters/dremio/api/handle.py
@@ -16,7 +16,7 @@
 from dbt.adapters.dremio.api.parameters import Parameters
 from dbt.adapters.dremio.api.rest.endpoints import login
 
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 

diff --git a/dbt/adapters/dremio/api/rest/endpoints.py b/dbt/adapters/dremio/api/rest/endpoints.py
@@ -35,7 +35,7 @@
 from dbt.adapters.dremio.api.parameters import Parameters
 from dbt.adapters.dremio.api.rest.url_builder import UrlBuilder
 
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 

diff --git a/dbt/adapters/dremio/connections.py b/dbt/adapters/dremio/connections.py
@@ -20,12 +20,14 @@
 from dbt.adapters.dremio.api.handle import DremioHandle
 from dbt.adapters.dremio.api.parameters import ParametersBuilder
 
+from dbt_common.clients import agate_helper
+
 import time
 import json
 
-import dbt.exceptions
+import dbt_common.exceptions
 from dbt.adapters.sql import SQLConnectionManager
-from dbt.contracts.connection import AdapterResponse
+from dbt.adapters.contracts.connection import AdapterResponse
 
 from dbt.adapters.dremio.api.rest.endpoints import (
     delete_catalog,
@@ -43,7 +45,7 @@
     DremioBadRequestException,
 )
 
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 
@@ -61,13 +63,13 @@ def exception_handler(self, sql):
         except Exception as e:
             logger.debug(f"Error running SQL: {sql}")
             self.release()
-            if isinstance(e, dbt.exceptions.DbtRuntimeError):
+            if isinstance(e, dbt_common.exceptions.DbtRuntimeError):
                 # during a sql query, an internal to dbt exception was raised.
                 # this sounds a lot like a signal handler and probably has
                 # useful information, so raise it without modification.
                 raise
 
-            raise dbt.exceptions.DbtRuntimeError(e)
+            raise dbt_common.exceptions.DbtRuntimeError(e)
 
     @classmethod
     def open(cls, connection):
@@ -184,7 +186,7 @@ def execute(
         if fetch:
             table = cursor.table
         else:
-            table = dbt.clients.agate_helper.empty_table()
+            table = agate_helper.empty_table()
 
         return response, table
 
@@ -254,10 +256,10 @@ def _create_folders(self, database, schema, api_parameters):
             except DremioAlreadyExistsException:
                 logger.debug(f"Folder {folder} already exists.")
             except DremioBadRequestException as e:
-               if "Can not create a folder inside a [SOURCE]" in e.message:
-                   logger.debug(f"Ignoring {e}")
-               else:
-                   raise e
+                if "Can not create a folder inside a [SOURCE]" in e.message:
+                    logger.debug(f"Ignoring {e}")
+                else:
+                    raise e
 
     def _create_path_list(self, database, schema):
         path = [database]

diff --git a/dbt/adapters/dremio/credentials.py b/dbt/adapters/dremio/credentials.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-from dbt.adapters.base import Credentials
+from dbt.adapters.contracts.connection import Credentials
 from dataclasses import dataclass
 from typing import Optional
 from dbt.adapters.dremio.relation import DremioRelation

diff --git a/dbt/adapters/dremio/impl.py b/dbt/adapters/dremio/impl.py
@@ -29,7 +29,7 @@
     Capability,
 )
 from dbt.adapters.sql.impl import DROP_RELATION_MACRO_NAME
-from dbt.events import AdapterLogger
+from dbt.adapters.events.logging import AdapterLogger
 
 logger = AdapterLogger("dremio")
 

diff --git a/dbt/include/dremio/macros/adapters/snapshot.sql b/dbt/include/dremio/macros/adapters/snapshot.sql
@@ -26,6 +26,10 @@ limitations under the License.*/
 
     when not matched
         then insert ({{ insert_cols_csv }})
-        values ({{ insert_cols_csv }})
+        values
+            ({% for column_name in insert_cols -%}
+                DBT_INTERNAL_SOURCE.{{ column_name }}
+                {%- if not loop.last %}, {%- endif %}
+            {%- endfor %})
 
 {% endmacro %}
diff --git a/dbt/include/dremio/macros/adapters/validate_sql.sql b/dbt/include/dremio/macros/adapters/validate_sql.sql
@@ -1,3 +1,17 @@
+/*Copyright (C) 2022 Dremio Corporation
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.*/
+
 {% macro default__validate_sql(sql) -%}
   {% call statement('validate_sql') -%}
     explain plan for {{ sql }}

diff --git a/dbt/include/dremio/macros/materializations/incremental/strategies.sql b/dbt/include/dremio/macros/materializations/incremental/strategies.sql
@@ -60,23 +60,13 @@ limitations under the License.*/
         {%- endfor %}
     {% endif %}
 
-
-    {%- set dest_col_names = dest_columns | map(attribute="name") -%}
-    {%- set dest_cols_prefixed = [] -%}
-
-    {% for dc in dest_col_names -%}
-        {% set this_key_match %}
-                    DBT_INTERNAL_SOURCE."{{ dc }}"
-        {% endset %}
-        {% do dest_cols_prefixed.append(this_key_match) %}
-    {%- endfor %}
-
-
     when not matched then insert
         ({{ dest_cols_csv }})
     values
-        ({{ dest_cols_prefixed | join(',') }})
-
+        ({% for column_name in dest_columns | map(attribute="name") -%}
+            DBT_INTERNAL_SOURCE.{{ column_name }}
+            {%- if not loop.last %}, {%- endif %}
+        {%- endfor %})
 
 {% endmacro %}
 

diff --git a/dbt/include/dremio/macros/materializations/seed/seed.sql b/dbt/include/dremio/macros/materializations/seed/seed.sql
@@ -17,10 +17,7 @@ limitations under the License.*/
   {%- set identifier = model['alias'] -%}
   {%- set format = config.get('format', validator=validation.any[basestring]) or 'iceberg' -%}
   {%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
-  {%- set target_relation = api.Relation.create(identifier=identifier,
-                                                schema=schema,
-                                                database=database,
-                                                type='table') -%}
+  {%- set target_relation = this.incorporate(type='table')-%}
   {% set grant_config = config.get('grants') %}
 
   {{ run_hooks(pre_hooks) }}

diff --git a/dbt/include/dremio/macros/materializations/table/table.sql b/dbt/include/dremio/macros/materializations/table/table.sql
@@ -17,10 +17,7 @@ limitations under the License.*/
   {%- set identifier = model['alias'] -%}
   {%- set format = config.get('format', validator=validation.any[basestring]) or 'iceberg' -%}
   {%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}
-  {%- set target_relation = api.Relation.create(identifier=identifier,
-                                                schema=schema,
-                                                database=database,
-                                                type='table') -%}
+  {%- set target_relation = this.incorporate(type='table') -%}
   {% set grant_config = config.get('grants') %}
   {{ run_hooks(pre_hooks) }}
 

diff --git a/dbt/include/dremio/macros/materializations/view/create_or_replace_view.sql b/dbt/include/dremio/macros/materializations/view/create_or_replace_view.sql
diff --git a/dbt/include/dremio/macros/materializations/view/view.sql b/dbt/include/dremio/macros/materializations/view/view.sql
@@ -19,8 +19,7 @@ limitations under the License.*/
 
   {%- set exists_as_view = (old_relation is not none and old_relation.is_view) -%}
 
-  {%- set target_relation = api.Relation.create(
-      identifier=identifier, schema=schema, database=database, type='view') -%}
+  {%- set target_relation = this.incorporate(type='view')-%}
 
   {% set grant_config = config.get('grants') %}
 

diff --git a/dbt/include/dremio/macros/utils/data_types.sql b/dbt/include/dremio/macros/utils/data_types.sql
@@ -1,3 +1,17 @@
+/*Copyright (C) 2022 Dremio Corporation
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.*/
+
 {% macro dremio__type_string() %}
     {{ return(api.Column.translate_type("VARCHAR")) }}
 {% endmacro %}
diff --git a/dev_requirements.txt b/dev_requirements.txt
@@ -3,7 +3,9 @@ Babel==2.12.1
 betterproto==1.2.5
 certifi==2023.7.22
 charset-normalizer==3.1.0
-dbt-tests-adapter==1.7.0
+dbt-core==1.8.8
+dbt-tests-adapter==1.8.0
+python-dotenv==1.0.1
 exceptiongroup==1.1.1
 future==0.18.3
 grpclib==0.4.3

diff --git a/setup.py b/setup.py
@@ -22,7 +22,7 @@
 
 package_name = "dbt-dremio"
 
-package_version = "1.7.0"
+package_version = "1.8.0"
 
 description = """The Dremio adapter plugin for dbt"""
 
@@ -37,7 +37,8 @@
     packages=find_namespace_packages(include=["dbt", "dbt.*"]),
     include_package_data=True,
     install_requires=[
-        "dbt-core>=1.2, <=1.7.13",
+        "dbt-core>=1.8",
+        "dbt-adapters>=1.0.0, <2.0.0",
         "requests>=2.31.0",
     ],
     classifiers=[