improve flattening performance for dbt_columns #681
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
👋 @ofek1weiss |
ofek1weiss
commented
Mar 19, 2024
| @@ -35,8 +35,8 @@ | |||
|
|
|||
| {% set flattened_columns = [] %} | |||
| {% for column_node in column_nodes.values() %} | |||
| {% set flat_column = elementary.flatten_column(table_node, column_node) %} | |||
| {% if not elementary.get_config_var('upload_only_columns_with_descriptions') or flat_column['description'] %} | |||
| {% if not elementary.get_config_var('upload_only_columns_with_descriptions') or column_node.get('description') %} | |||
There was a problem hiding this comment.
changed filter to happen before flattening, to improve performance
haritamar
approved these changes
Mar 19, 2024
konstantin-baidin-y42
added a commit
to goes-funky/dbt-data-reliability
that referenced
this pull request
Apr 9, 2024
* support count_true and count_false for boolean columns in BigQuery * support count_true and count_false for boolean columns in BigQuery * fix: change generate profile args macro name for Athena * Add unique_combination_of_columns to common_tests_configs_mapping * missing comma * fix drop_failure_percent_threshold failing a non anomalous test * Support all anomaly vars on all configuration levels * ELE-2470 temp tables are not being deleted * Add empty line at the end of a filre * fix typo * Removed default detection/training_period * Make sure we delete temp tables last * removed unused import * Not collecting metrics by default. * release 0.14.0 * fix bug when no temp tables xist * Update macros/edr/tests/test_utils/clean_elementary_test_tables.sql Co-authored-by: IDoneShaveIt <[email protected]> * 1. add tags to all elementary monitors 2. only run tests if elementary is enabled * rename tag * Create clean_dbt_columns_temp_tables macro * Add empty line at the end of a file * clean logs * add arg chunk_size for all insert_rows() (elementary-data#669) * release 0.14.1 * override primary_test_model_id (elementary-data#671) * added ignore_small_changes to freshness and event_freshness * improvement: bigquery specific for query_table_metrics (elementary-data#674) * improvement: bigquery specific for query_table_metrics Using information schema to get row count is much more performant than doing a full table scan * use TABLE_STORAGE and add database & schema * add empty case * add set * Add index on created_at test_result_rows and remove backfill post hook * Ele 2606 package version with caching and extra logs (elementary-data#673) * artifacts: use cache also for model post-hook * add performance logs to artifacts logic * duration monitoring - bugfix - handle the case the duration stack is not initialized * Change the aggregate of failed_row_count_calc to count(*) * Readme updates (elementary-data#684) * changes to readme * changes to readme * changes * changes * image url * image url * changes * formating * formating * changes * link * pre commit * improve flattening performance for dbt_columns (elementary-data#681) * improve flattening performance for dbt_columns * removed unused const * black * Add get_requires_permissions and validate_required_permissions macros * Improved messages * Fixed default__get_required_permissions + add target.database to get_relevant_databases --------- Co-authored-by: suelai <[email protected]> Co-authored-by: Roman Korsun <[email protected]> Co-authored-by: Yasuhisa Yoshida <[email protected]> Co-authored-by: Ofek Weiss <[email protected]> Co-authored-by: Ofek Weiss <[email protected]> Co-authored-by: IDoneShaveIt <[email protected]> Co-authored-by: IDoneShaveIt <[email protected]> Co-authored-by: Elon Gliksberg <[email protected]> Co-authored-by: GitHub Actions <[email protected]> Co-authored-by: Ella Katz <[email protected]> Co-authored-by: J.C <[email protected]> Co-authored-by: Noy Arie <[email protected]> Co-authored-by: Chris Dong <[email protected]> Co-authored-by: noakurman <[email protected]> Co-authored-by: Itamar Hartstein <[email protected]> Co-authored-by: Maayan Salom <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.