[Bug] dbt retry does not respect --threads

[barberscott]

Is this a new bug in dbt-core?

• I believe this is a new bug in dbt-core
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Sometimes concurrency can be the cause of failures (e.g. database OOM, connection timeouts) and I would want to retry with a lower --threads value. dbt does not respect a new --threads value when passed in on a retry.

Expected Behavior

Expect that --threads is respected on retry, otherwise if not passed, re-use same threads from the previous invocation of dbt.

Steps To Reproduce

dbt build --threads 10
dbt retry --threads 1

Relevant log output

No response

Environment

- OS: all
- Python: 3.11.9
- dbt: 1.8.5

Which database adapter are you using with dbt?

other (mention it in "Additional Context")

Additional Context

Using clickhouse adapter currently

[dbeatty10]

Thanks for reaching out about this @barberscott !

Was able to reproduce what you described. Agreed that dbt retry should respect any CLI flags (including --threads!).

Reprex

models/my_model.sql

{% if execute %}
  {{ exceptions.raise_compiler_error("Forced error") }}
{% endif %}

select 1 as id

Run these commands:

dbt build -s my_model --threads 3
dbt retry --threads 2

See that the output for retry says "Concurrency: 3 threads" instead of "Concurrency: 1 threads":

(dbt_1.8) $ dbt build -s my_model --threads 3

14:39:12  Running with dbt=1.8.3
14:39:15  Registered adapter: duckdb=1.8.1
14:39:15  Found 2 models, 406 macros
14:39:15  
14:39:15  Concurrency: 3 threads (target='dev')
14:39:15  
14:39:15  1 of 1 START sql view model dev.my_model ....................................... [RUN]
14:39:15  1 of 1 ERROR creating sql view model dev.my_model .............................. [ERROR in 0.01s]
14:39:15  
14:39:15  Finished running 1 view model in 0 hours 0 minutes and 0.19 seconds (0.19s).
14:39:15  
14:39:15  Completed with 1 error and 0 warnings:
14:39:15  
14:39:15    Compilation Error in model my_model (models/my_model.sql)
  Forced error
14:39:16  
14:39:16  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

(dbt_1.8) $ dbt retry --threads 2

14:39:17  Running with dbt=1.8.3
14:39:17  Registered adapter: duckdb=1.8.1
14:39:17  Found 2 models, 406 macros
14:39:17  
14:39:18  Concurrency: 3 threads (target='dev')
14:39:18  
14:39:18  1 of 1 START sql view model dev.my_model ....................................... [RUN]
14:39:18  1 of 1 ERROR creating sql view model dev.my_model .............................. [ERROR in 0.01s]
14:39:18  
14:39:18  Finished running 1 view model in 0 hours 0 minutes and 0.17 seconds (0.17s).
14:39:18  
14:39:18  Completed with 1 error and 0 warnings:
14:39:18  
14:39:18    Compilation Error in model my_model (models/my_model.sql)
  Forced error
14:39:18  
14:39:18  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

[donjin-master]

Hey @dbeatty10,
I have raised the PR for this fix. May I know who will be reviewing the code.

[dbeatty10]

Thanks for raising a PR @donjin-master !

We haven't assigned an engineer to review yet, but when we do, it will show up in the PR in this section: