[EPIC] Sample Mode

[QMalcolm]

Sample mode in dbt will be a "special" running mode to based on some criteria build models with a subset of the input data (i.e. a "sample"). This is important because a lot of us deal with incredibly large data sets on a frequent basis. During development, when we want to be iterating quickly, these large models take a while to run and slow us down. In its simplest form, at the end of this epic, one will be able to do the following dbt run --sample 3 days. The resulting run execution will create "sample" models for any model with an event_time with at least one input that also has an event_time defined.

P0

Preview Give feedback

1. Gate using sample mode with dbt_experimental_sample_mode environment variable #11227
   enhancement sample_mode +2
   
2. Add --sample flag to CLI invocation for run command #11230
   enhancement sample_mode +2
   
3. Allow for specification of a "relative" time window for sample mode #11231
   enhancement sample_mode +2
   
4. Allow for specifying sample time window in project configs #11232
   enhancement sample_mode +2
   
5. allow for the building of "normal" models as "samples" #11248
   enhancement sample_mode +2
   
6. allow for the building of "incremental" models as "samples" #11254
   enhancement sample_mode +2
   
7. allow for the building of "microbatch" models as "samples" #11252
   enhancement sample_mode +2
   

P1

Preview Give feedback

1. Remove dbt_experimental_sample_mode environment variable gate #11228
   enhancement sample_mode +2
2. allow for specification of sample time window in targeted manner #11258
   +0
3. allow for opting out specific models from sample mode
4. allow for sample mode during build command

P2

Preview Give feedback

1. allow for specifying sample time window at model level

bugs 👀

Preview Give feedback

No tasks being tracked yet.