[Bug] dbtCore - after upgrading, can't run DBT

[moseleyi]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

I installed dbt-core, bigquery, and postgres, and then when typing dbt --version I get this error:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\___\AppData\Roaming\Python\Python312\Scripts\dbt.exe\__main__.py", line 4, in <module>
  File "C:\Users\___t\AppData\Roaming\Python\Python312\site-packages\dbt\cli\__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "C:\Users\___\AppData\Roaming\Python\Python312\site-packages\dbt\cli\requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

This is after installing bigquery, postgres,

Expected Behavior

Should show the version of dbt packages

Steps To Reproduce

1. Install dbt-core
2. Run dbt --version

Relevant log output

No response

Environment

- OS: Windows
- Python: 3.12
- dbt: 1.8

Which database adapter are you using with dbt?

No response

Additional Context

No response

[jtcohen6]

@moseleyi Could you run pip freeze | grep 'dbt' and report back with the exact versions of Python packages that you have installed?

Also - you might try installing all of these by name, along with the --upgrade flag, and see if that resolves the issue for you:

pip install --upgrade dbt-core dbt-bigquery dbt-postgres

[rubenhelsloot]

@jtcohen6 I have the same issue, this is the output you asked for:

❯ pip freeze | grep 'dbt'
dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1
sqlfluff-templater-dbt==3.0.6

The issue persists when I uninstall and then reinstall dbt-core and dbt-postgres. FWIW I'm using poetry.

[jtcohen6]

Thanks @rubenhelsloot! Are you also using Python 3.12?

[moseleyi]

My output:

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Some people in my team got it working but maybe because it was clean install, they never had to use python packages before. Perhaps some dependencies of dbt are not updated with the latest update?

[moseleyi]

Surprisingly dbt 1.7.13 works fine with Python 3.12.3 - I checked on my other laptop

[dbeatty10]

@rubenhelsloot! Are you also using Windows?

@moseleyi Which commands are you using to installing dbt? Are you using a virtual environment with venv or Poetry?

The following files and commands worked for me:

• on macOS
• using venv
• with Python 3.12.3 (managed by pyenv)

Create this file:

requirements.txt

dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-postgres==1.8.0
dbt-semantic-interfaces==0.5.1

Run these commands:

python3 -m venv env
source env/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
source env/bin/activate
python --version
dbt --version
deactivate

Output:

Python 3.12.3
Core:
  - installed: 1.8.0
  - latest:    1.8.0 - Up to date!

Plugins:
  - postgres: 1.8.0 - Up to date!

[leoebfolsom]

In case helpful, when I upgraded to dbt-core 1.8, I had the same issue on Python 3.9.6 (OS: Mac Ventura 13.4.1), and it was resolved on upgrade to Python 3.12.3.

[moseleyi]

I don't use venv nor poetry. At first I just upgraded pip install dbt-core --upgrade and the adapters. That didn't work so I tried uninstalling everything and installing again using different commands (core + adapters in one command, adapter only etc.)

All of them produce this error

[troyharvey]

Reproduced with Python 3.10.

＄ python --version
Python 3.10.14

＄ dbt deps
Traceback (most recent call last):
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/bin/dbt", line 5, in <module>
    from dbt.cli.main import cli
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/__init__.py", line 1, in <module>
    from .main import cli as dbt_cli  # noqa
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/main.py", line 14, in <module>
    from dbt.cli import requires, params as p
  File "/opt/homebrew/Caskroom/miniconda/base/envs/ds-redshift/lib/python3.10/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

[paulbakkerbloom]

Had the same issue with Python 3.9.18 -> solved when I upgraded to Python 3.11.6

I use poetry and upgraded through poetry update

dbt-adapters                          1.1.1             The set of adapter ...
dbt-bigquery                          1.8.0             The Bigquery adapte...
dbt-common                            1.0.4             The shared common u...
dbt-core                              1.8.0             With dbt, data anal...
dbt-extractor                         0.5.1             A tool to analyze a...
dbt-semantic-interfaces               0.5.1             The shared semantic...
dbt-tests-adapter                     1.8.0             The set of reusable...

[moseleyi]

I was hoping it was about python issue but seems like it's not. It could be a dependency that should be updated with dbt core update but isn't being forced to update?

[moseleyi]

I also found this: AltimateAI/vscode-dbt-power-user#1029 but we all have dbt-adapters installed

[brabster]

A bit more info: I had the same problem, I can reproduce at will, Python 3.11.8:

Install dbt-bigquery==1.7.7 into a clean venv, working fine (via pip install -U -r requirements.txt)

Bump to dbt-bigquery>=1.7.7 and same pip command, broken

 File "/workspaces/pypi_vulnerabilities/venv/lib/python3.11/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

Delete venv, same requirements.txt, pip install - get 1.8, works fine.

In the broken venv, the factory.py (and other) files in dbt/adapters is missing.

You should be able to reproduce these steps exactly with a codespace off https://github.com/brabster/pypi_vulnerabilities

[dbeatty10]

Thanks for this info @brabster 🏅

Could you share the contents of your requirements.txt file also?

[brabster]

@dbeatty10
It's just the dbt-bigquery dependency. Originally as per repo. https://github.com/brabster/pypi_vulnerabilities/blob/main/requirements.txt

Clean venv, set to dbt-bigquery==1.7.7 to force pre-1.8.0 and install. Works.
Then set to dbt-bigquery>=1.7.7 to upgrade to latest. Broken
Clean venv, install same requirements. Works

[jtcohen6]

@brabster If in step (2), you run pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, do you still see the error?

(I'm hoping this might be because the underlying dependencies are not being upgraded, when they should be, but I'm not 100% positive: #10123)

[dbeatty10]

Thank you @brabster ! 🏅 I was able to reproduce the error using your description. See "reprex" below for full details.

@jtcohen6 when I tried running pip install --upgrade dbt-core dbt-common dbt-adapters dbt-bigquery, it still gave the same error.

Reprex

Create requirements files for both 1.7 and 1.8:

cat << EOF > requirements_1.7.txt
dbt-bigquery==1.7.7
EOF
cat << EOF > requirements_1.8.txt
dbt-adapters==1.1.1
dbt-common==1.0.4
dbt-core==1.8.0
dbt-extractor==0.5.1
dbt-bigquery==1.8.0
dbt-semantic-interfaces==0.5.1
EOF

✅ Create a fresh virtual environment and install the requirements for 1.7:

rm -rf env
python3 -m venv env
source env/bin/activate
python -m pip install -r requirements_1.7.txt
source env/bin/activate
dbt --version

❌ Try upgrading by installing the requirements for 1.8 in the same virtual environment:

python -m pip install -r requirements_1.8.txt
source env/bin/activate
dbt --version

Get an error like the following:

    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

✅ Try upgrading by installing the requirements for 1.8 in a fresh virtual environment:

deactivate
rm -rf env
python3 -m venv env
source env/bin/activate
python -m pip install -r requirements_1.8.txt
source env/bin/activate
dbt --version

[dbeatty10]

When I got the same error as the reported above, the following workaround resolved it for me:

python -m pip uninstall -y dbt-adapters
python -m pip install --upgrade dbt-core dbt-common dbt-adapters
dbt --version

The pip uninstall -y dbt-adapters is the important part.

The 2nd command can be replaced with whichever command you use to install the requirements for dbt 1.8.

[brabster]

Can you run 1.7.x adapter implementations with dbt 1.8.x core/adapters? If not then that might explain @ann8ty problem, looks like they have pinned exact 1.7.17 Pg/Redshift adapter versions.

More general question maybe aimed at @dbeatty10 - If you folks are decoupling versions of adapter implementations from core and adapters packages, how can a user tell whether different versions are compatible? Probably some docs somewhere I haven't found?

[smittal086]

pip install --upgrade dbt-adapters

run this command, hopefully it will solve the issue

[jessanmen1]

pip install -r requirements.txt --no-cache-dir --ignore-installed worked for me on Python 3.10.

[mroy-seedbox]

Simply upgrading dbt-adapters solved the issue! 👌

[paulav6]

Re-installing dbt-adapters did not work for me. However cleaning up virtual environment rm -rf .ve and reinstalling pip install --upgrade -r requirements.txt that had the dbt versions worked.

[nguyentoanit]

After trying all above solutions, Only upgrading python to v3.12 worked for me. 🤷

- OS: Linux
- Python: 3.12.4
- dbt: 1.8.4

[ozkary]

After upgrading to dbt 1.8.4, I also got the error:

File ".local/lib/python3.8/site-packages/dbt/cli/requires.py", line 10, in <module>
    from dbt.adapters.factory import adapter_management, register_adapter, get_adapter
ModuleNotFoundError: No module named 'dbt.adapters.factory'

I was able to correct this by removing the current adapters and installing it again:

pip uninstall -y dbt-adapters

pip install --upgrade dbt-adapters

All looks fine now. I am using python 3.8.10

Core:
  - installed: 1.8.4
  - latest:    1.8.4 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

Thanks for helping us resolve this.

[moltar]

I have some observations to throw in.

It appears that the problem is intermittent. I've had the same CI run fail, and then after retrying it had passed.

I used a pip freeze solution with no force upgrades.

On the same GitHub Actions run, I had three workflows that used dbt install identical steps. One out of three runs had failed with the No module named 'dbt.adapters.factory' error.

Then, I clicked "Re-run failed," and it passed.

$> grep dbt "$DBT_PROJECT_DIR/requirements.txt"

dbt-adapters==1.4.0
dbt-common==1.7.0
dbt-core==1.8.4
dbt-extractor==0.5.1
dbt-postgres==1.8.2
dbt-semantic-interfaces==0.5.1
sqlfluff-templater-dbt==3.1.0

[brabster]

@moltar are you caching the dependencies in the GH Action? I would suspect that for causing intermittent problems as the state of what's ending up on in the filesystem (and then being cached) may be quite unpredictable. I've seen no issues in my GH actions, but I don't cache. If you are caching, maybe try disabling that to get a clean install from scratch?

[moltar]

@brabster Yup:

      - name: Install Python
        uses: actions/setup-python@v4
        with:
          cache: pip
          python-version: '3.12'

[moltar]

@brabster Removed caching, but still get random failures.

[brabster]

@moltar strange - if the root cause I identified earlier is true, clean install should not have a problem (unless something else is going on as well to confound the problem). Curious, so happy to take a look if you can share the details of the workflow between the setup python task the step that fails like what's the specific install command, and any logs you can share about the install/update process?

[alxsbn]

Same issue after upgrading dbt within our Airflow stack. We have Python 3.8.18. We try to solve this by using pip install --upgrade dbt-adapters without success as-well-as doing a --force-recreate of our stack (so it's not a cache issue)

airflow-dbt==0.4.0
dbt-adapters==1.4.1
dbt-common==1.7.0
dbt-core==1.8.6
dbt-databricks==1.8.5
dbt-extractor==0.5.1
dbt-semantic-interfaces==0.5.1
dbt-spark==1.8.0

[saeedkhattak]

Had this issue with my dbt with oracle adapter. The following code fixed it:
pip install --upgrade dbt-core dbt-oracle
Make sure you upgrade both your dbt core as well as your adapter

[hanslemm]

In my case, after many trials and errors, I discovered that if you reinstall dbt-core after dbt-adapters is already installed, this error arises. So you actually need to reinstall both with --force-install.

[jaceksan]

Hit this issue too when bumping dbt from 1.7.x to 1.9.x because I had to bump dbt-duckdb (MotherDuck deprecates 1.0).
I used Python 3.10 due to historical reasons.
When I bumped Python to 3.12, the problem disappeared.

[petersjanjaap]

pip install dbt-fabric fixed it for me, it adds the adapter for fabric. It's unfortunate it is imported into the cli functionality, because it requires you to install dbt-fabric, even though you are using a different db to interact with (say BigQuery).

[tgmof]

In my case, after many trials and errors, I discovered that if you reinstall dbt-core after dbt-adapters is already installed, this error arises. So you actually need to reinstall both with --force-install.

I can confirm what @hanslemm wrote. We have a CICD pipeline that tries to build / install dependencies using Poetry and it only fails when our pipeline installs dbt-adapters and then tries to upgrade dbt-core.

Indeed, the pipelines that use a Docker image that already contains dbt-core (but not yet dbt-adapters) do the following:

Installing dbt-adapters (1.7.0)
Updating dbt-core (1.7.0 -> 1.8.7)

The pipelines that use an 'empty' Docker image do the following:

Installing dbt-adapters (1.7.0)
Installing dbt-core (1.8.7)

I can't confirm 100% that this "installing/updating" order is the root cause of it but at least I observe the same thing as what @hanslemm describes.
I doubt that upgrading to python3.12 solves the issue since this whole issue thread was created by someone who raised the issue for Python3.12 (so my guess is that the people that claim that "upgrading to python 3.12 solved the issue" actually re-installed all dependencies and this re-installation had the consequence of solving the issue, not the Python Version itself).

Detail of the poetry traces

The CICD run that works:

  • Installing pytz (2024.2)
  • Installing requests (2.32.3)
  • Installing tzdata (2024.2)
  • Installing zipp (3.21.0)
  • Installing click (8.1.7)
  • Installing dbt-common (1.10.0)
  • Installing google-api-core (2.23.0)
  • Installing google-crc32c (1.6.0)
  • Installing grpcio-status (1.62.3)
  • Installing importlib-metadata (6.11.0)
  • Installing more-itertools (10.5.0)
  • Installing packaging (24.2)
  • Installing pandas (2.2.3)
  • Installing pyarrow (18.1.0)
  • Installing pydantic (2.10.2)
  • Installing pyyaml (6.0.2)
  • Installing daff (1.3.46)
  • Installing db-dtypes (1.3.1)
  • Installing dbt-adapters (1.7.0)
  • Installing dbt-extractor (0.5.1)
  • Installing dbt-semantic-interfaces (0.5.1)
  • Installing google-cloud-core (2.4.1)
  • Installing google-resumable-media (2.7.2)
  • Installing grpc-google-iam-v1 (0.13.1)
  • Installing logbook (1.5.3)
  • Installing minimal-snowplow-tracker (0.0.2)
  • Installing networkx (3.4.2)
  • Installing sqlparse (0.5.2)
  • Installing dbt-core (1.8.7)
  • Installing google-cloud-bigquery (3.27.0)
  • Installing google-cloud-dataproc (5.15.1)
  • Installing google-cloud-storage (2.18.2)
  • Installing dbt-bigquery (1.8.2)
  • Downgrading setuptools (69.5.1 -> 68.2.2)

The CICD run that fails:

  • Updating pytz (2024.1 -> 2024.2)
  • Installing tzdata (2024.2)
  • Updating zipp (3.19.2 -> 3.21.0)
  • Installing dbt-common (1.10.0)
  • Updating google-api-core (2.19.1 -> 2.23.0)
  • Updating google-crc32c (1.5.0 -> 1.6.0)
  • Updating grpcio-status (1.62.2 -> 1.62.3)
  • Updating more-itertools (10.3.0 -> 10.5.0)
  • Updating packaging (24.1 -> 24.2)
  • Installing pandas (2.2.3)
  • Installing pyarrow (18.1.0)
  • Updating pydantic (2.8.2 -> 2.10.2)
  • Updating pyyaml (6.0.1 -> 6.0.2)
  • Installing daff (1.3.46)
  • Installing db-dtypes (1.3.1)
  • Installing dbt-adapters (1.7.0)
  • Updating dbt-semantic-interfaces (0.4.4 -> 0.5.1)
  • Updating google-resumable-media (2.7.1 -> 2.7.2)
  • Updating networkx (3.3 -> 3.4.2)
  • Updating sqlparse (0.4.4 -> 0.5.2)
  • Updating dbt-core (1.7.0 -> 1.8.7)
  • Updating google-cloud-bigquery (3.25.0 -> 3.27.0)
  • Updating google-cloud-dataproc (5.10.1 -> 5.15.1)
  • Updating google-cloud-storage (2.18.0 -> 2.18.2)
  • Updating dbt-bigquery (1.7.6 -> 1.8.2)

[jaceksan]

I can't confirm 100% that this "installing/updating" order is the root cause of it but at least I observe the same thing as what @hanslemm describes. I doubt that upgrading to python3.12 solves the issue since this whole issue thread was created by someone who raised the issue for Python3.12 (so my guess is that the people that claim that "upgrading to python 3.12 solved the issue" actually re-installed all dependencies and this re-installation had the consequence of solving the issue, not the Python Version itself).

Works for me with this setup.

requirements-dbt.txt:

dbt-postgres~=1.9.0
dbt-snowflake~=1.9.0
# Vertica has not been updated to 1.9 yet. The current pipeline does not run on Vertica.
# dbt-vertica==1.7.3
dbt-duckdb~=1.9.1
duckdb~=1.1.3
setuptools==69.5.1

packages.yml:

packages:
  - package: Snowflake-Labs/dbt_constraints
    version: 0.6.1
  - package: dbt-labs/codegen
    version: 0.10.0
  - package: calogica/dbt_date
    version: 0.9.1

Dockefile:

ARG PYTHON_VERSION=3.12
ARG DBT_VERSION=1.9.1
ARG DBT_IMAGE=ghcr.io/dbt-labs/dbt-core:$DBT_VERSION
ARG SRC_DATA_PIPELINE=data_pipeline
FROM $DBT_IMAGE

ARG IMAGES_WORKDIR=/project
WORKDIR ${IMAGES_WORKDIR}

ARG SRC_DATA_PIPELINE
COPY ${SRC_DATA_PIPELINE}/requirements-dbt.txt requirements-dbt.txt
RUN pip3 install -r requirements-dbt.txt

COPY ${SRC_DATA_PIPELINE}/packages.yml packages.yml
COPY ${SRC_DATA_PIPELINE}/dbt_project.yml dbt_project.yml

# dbt deps needs profiles.yml, don't know why. TODO: investigate
COPY ${SRC_DATA_PIPELINE}/profile/profiles.yml /root/.dbt/profiles.yml
RUN dbt deps

ENTRYPOINT ["dbt"]

[tgmof]

I can't confirm 100% that this "installing/updating" order is the root cause of it but at least I observe the same thing as what @hanslemm describes. I doubt that upgrading to python3.12 solves the issue since this whole issue thread was created by someone who raised the issue for Python3.12 (so my guess is that the people that claim that "upgrading to python 3.12 solved the issue" actually re-installed all dependencies and this re-installation had the consequence of solving the issue, not the Python Version itself).

Works for me with this setup.

Yes, what I wrote above and tried to explain (but maybe too fuzzily and not clearly enough?) is:

• It works on all fresh install (which looks to be what your Docker file does, a fresh install that includes a req.txt that references dependencies that themselves depend on dbt-adapters).
• It does NOT work for Docker images where dbt-core was pre-installed and we do poetry install (at runtime, not build time) which updates dbt-core after installing dbt-adapters (which is the issue I tried to explain / document in my message above and matches with what @hanslemm mentioned - even though I'm unsure his findings are just correlated to the root cause or ARE the root cause).

[jaceksan]

Yes, what I wrote above and tried to explain (but maybe too fuzzily and not clearly enough?) is:

* It works on all fresh install (which looks to be what your Docker file does, a fresh install that includes a req.txt that references dependencies that themselves depend on dbt-adapters).

* It does NOT work for Docker images where dbt-core was pre-installed and we do poetry install (at runtime, not build time) which updates dbt-core after installing dbt-adapters (which is the issue I tried to explain / document in my message above and matches with what [@hanslemm](https://github.com/hanslemm) mentioned - even though I'm unsure his findings are just correlated to the root cause or ARE the root cause).

Eh, sorry. I overlooked the context (expand all comments).
Another reason why the immutable approach is so beneficial ;-)

[pablo-espeso]

These lines worked for me:

pip3 install --force-reinstall dbt-duckdb
pip3 install --force-reinstall dbt-core

So it seems forcing the reinstalling of these packages fixes the issue.

[brabster]

Still seems like the explanation I gave in this May 17, 2024 comment explains all the issues folks have seen. In a nutshell:

• Prior to the adapters refactor (moving it out of dbt-core) in dbt 1.8, dbt-core "owned" the installed dbt/adapters dir in site_packages, as far as pip et al. are concerned.
• The refactor to extract dbt-adapters made dbt-adapters "own" that directory instead.
• When pip does an in-place upgrade, it iterates over packages in alphabetical order by package name, copying new files into site_packages and deleting files that are no longer needed.
• So.... when you try and upgrade from pre-1.8 to post-1.8, pip FIRST installs dbt-adapters files into site_packages/dbt/adapters AND THEN deals with dbt-core DELETING files that dbt-adapters put there that have the same names - like factory.py

Why do the fixes work?

• When you then run install dbt-adapters it's not fixed because pip thinks there's nothing to do.
• --force-reinstall makes pip install all packages again, regardless of the package you specify (docs), fixing the problem because we no longer have a version currently installed that "owns" files that are now "owned" by dbt-adapters - so it's same as a fresh install.
• reinstalling from scratch works for the same reason - no version of dbt-core that owned those files currently installed so they don't get deleted.

Maybe I missed something - any behaviour seen since not explained by that? (cc @dbeatty10?)

It seems odd and non-default behaviour that these packages files end up in the same site_packages dir. Presumably that's a choice dbt Labs made as part of the refactor - and it appears to have this unfortunate side effect (darned alphabet!). Perhaps if a future version moved the directory that dbt-adapters lands its files in site_packages, the problem could actually be fixed. If I get a minute I might try a PR to see.

[fabian-bode]

--force-install

it should be --force-reinstall though :)