[ADAP-369] [Feature] Include all bytes processed in the adapter response

[abuckenheimer]

Is this a regression in a recent version of dbt-bigquery?

• I believe this is a regression in dbt-bigquery functionality
• I have searched the existing issues, and I could not find an existing issue for this regression

Current Behavior

Hey folks this is a bit of a convoluted "regression" so I'm happy to mark this as something else but following the merge of dbt-labs/dbt-bigquery#77 I noticed that the bytes_processed returned in the adapter response for models with copy_partitions: true is now 0. This kind of makes sense, bigquery copies are free so 0 bytes is kind of correct but its ultimately unexpected because the construction of the temporary table is ultimately not free.

This is indicative of a larger problem with the adapter response in that it only returns the bytes processed for the last step, it does not consider every part of running the model. You can see this when copy_partitions: false, dbt is under reporting the cost by up to half of running incremental models because we big-query will charge you for both the creation of a temporary table and then again to read the temporary table for the merge query into the destination.

Expected/Previous Behavior

given a similar config to:

model_name:
    +materialized: incremental
    +partition_by:
              copy_partitions: 
              field: timestamp
              data_type: timestamp
              granularity: day

On a dbt run I'd expect to see the bytes processed be equal to the amount of bytes that would be processed by running the target/compiled/.../model_name.sql file in the bigquery console, instead I see:

19:20:57  1 of 1 OK created sql incremental model dataset.model_name................ [None (0 processed) in 166.72s]

Steps To Reproduce

see above

Relevant log output

No response

Environment

- OS:Ubuntu 18.04.6 LTS (Bionic Beaver)
- Python: 3.11.2
- dbt-core (working version): N/A
- dbt-bigquery (working version): N/A 
- dbt-core (regression version): 1.5.0-b3
- dbt-bigquery (regression version): 1.5.0b2

Additional Context

No response

[dbeatty10]

Thanks for opening this @abuckenheimer !

Agree with your assessment that this is because the adapter response only returns the bytes processed for the last step, and it does not consider each query that executed as a result of running the model. This is problematic when it appears to the user that it is counting everything.

Although this may be surprising to many dbt-bigquery users, the current behavior is what we expect, so I relabeled this as a feature request / enhancement. I also re-titled accordingly.

We could imagine some scenario where dbt was able to accumulate the bytes processed over multiple statements and then display them. This could be a useful proxy for how much it cost to run a particular model.

There's a few main things that give me pause though:

1. The output emitted by dbt doesn't feel like the best place for comprehensive cost tracking and management
2. Not all cost-generating activity is likely to be easily captured and assigned to an individual dbt model -- think API calls or necessary metadata queries.
3. We typically prefer to expose features that are supported on multiple data platforms (e.g., not just a single platform like BigQuery)

bytes_processed in dbt-bigquery is already an exception to the norm of only exposing features supported by multiple warehouses, and we could push further down that path by trying to do an accumulation / roll-up of the reported value.

But my dominating thought is that most cost is going to be incurred once a project is in production, and it seems crucial that teams have a method to monitor their costs outside of the CLI output of dbt. I'd rather analytics engineers be comfortable seeing their all-in costs (including consumption by end users like BI) than relying on the reported bytes during model development.

Interested to hear your feedback -- I am overestimating the role of comprehensive cost observability (outside of the dbt output)?

[abuckenheimer]

I agree I don't expect dbt console reporting to be comprehensive but its the most practical feedback loop you can have while developing. We ask our developers to come up with an estimate of cost in PR's when introducing new models or making changes and seeing the gb processed in the console is a really great starting point telling us is this going up if so by how much. We know this is just an estimate and misses the nuance you might get in running a model over a period of time and seeing client read patterns but at least it tells you within an order of magnitude what something might cost which is most of what matters in evaluating changes. Now that this just reports 0 you can't even start a discussion without bringing in other tools.

[dbeatty10]

I hear you about supporting a practical feedback loop and how the optimization in dbt-labs/dbt-bigquery#77 removed some information that was there previously (however imperfect it was).

Although we aren't able to prioritize it ourselves at this time, we'd welcome a PR from a community member that provides a targeted, best-effort roll-up of bytes processed on a per-model basis. Labeling this accordingly as help_wanted.

[dbeatty10]

These two issues also asked for all bytes processed to be included in the adapter response and run_results.json:

• [ADAP-624] [CT-2694] [Bug] First SQL Query to Create Incremental Model Processing Does not Show Up in dbt Logs dbt-bigquery#775
• [Bug] incremental run with __dbt_tmp table does not log the real bytes_billed in run_results.json dbt-bigquery#1230

I closed them each in favor of this issue.

[richardbiddle]

This is an earlier issue report which is related to this request:
CT-1107 Log output for materialization incremental incorrectly reports bytes processed

[codigo-ergo-sum]

I don't have the opportunity to submit a PR right now for this, but just chiming in as I am the original poster/opener for dbt-labs/dbt-bigquery#775. Understanding the practical challenges of doing so, I am still very interested in seeing this functionality included.

[Thrasi]

I believe this affects packages like dbt elementary
which use the run_results.json to keep track of model performance.