[CT-1033] [Enhancement] Apply all dataset access grants for authorized views at run end, in main thread

[jtcohen6]

Describe the feature

"Batch up" all dataset access grants to the end of an invocation, similar to the discussion in #87 (comment). This is much faster / more efficient, and avoids thread-safety issues — but at the cost of significant potential delay between when each view is (re)created, and when their access is authorized (all at once, after every model has run).

Big idea:

• At run end, identify all authorized views that have been created during the run
• Batch up into a set of mappings between {dataset_to_grant_access_on: [views_needing_access]}
• Make API call to apply grants, once per dataset, for all datasets in sequence

There's some pseudo-code outlining this in #87 (comment). Of course, we'd want to write that logic in Python. We might need a new task/run-level "hook," similar to adapter.post_model_hook.

Describe alternatives you've considered

Provide thread-safety by other means, e.g. moving the logic into adapter.post_model_hook (instead of view materialization) and holding a lock.

Update: This is what's proposed in #265

Additional context

• https://cloud.google.com/bigquery/docs/share-access-views
• https://docs.getdbt.com/reference/resource-configs/bigquery-configs#authorized-views

Not to be confused with standard grants!

Who will this benefit?

Users of multiple authorized views on dbt-bigquery

Are you interested in contributing this feature?

Definitely open to a community contribution!

[judahrand]

Even as a temporary step is there something wrong with the approach in #265? It should prevent the access entry updates from colliding at the expense of slowing things down with locking. What is the additional advantage of moving it into post_model_hook?

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue, or it will be closed in 7 days.