snowplow-utils v0.16.0

Summary

This release contains a new array size macro as well as the optional ability for the unnest macros to also surface the index of the original array element for preserving the order. It also contains some fixes for redshift. Please note that from this version onwards this package is under the SPAL license.

Features

• Add get_array_size macro
• Add optional index to unnest

Fixes

• Fix existing tests for redshift
• Fix issue with multiple end hooks failing (Close #152)

Under the hood

• Update license to SPAL

Upgrading

To upgrade, bump the package version in your packages.yml file.

snowplow-utils v0.15.2

Summary

This release contains a bug fix in our new base macros, which did not affect default behaviour. This also does not impact behavior in any of the Snowplow packages that rely on the utils package.

Fixes

• Fix an incorrect default value in the base_create_snowplow_events_this_run macro.

Upgrading

To upgrade, bump the package version in your packages.yml file.

snowplow-utils v0.15.1

Summary

This release combines a few bug fixes in our new base macros, including an issue with redshift query size limits.

Fixes

• Fix issue with 8001 redshift error
• Add condition for late loading events to fix issue where session events before session start timestamp were processed
• Fix the manifest table not using the specified session timestamp field
• Provide correct error message when no user identifier is present

Under the hood

• Fix some issues in our integration tests

Upgrading

To upgrade, bump the package version in your packages.yml file.

snowplow-utils v0.15.0

Summary

This full release introduces new "base" macros which are used to recreate the snowplow_base_events_this_run and related tables. It gives you the option to customise your session and user identifiers, your partitioning timestamps, and much more. Additionally, we remove the deprecated snowplow_incremental materialization that was removed in 0.14.0, and we add multi-version support for the get_field macro for BigQuery.

We have documentation explaining the outlined changes in detail here that you can use to explore the base macro changes in particular. For information about how to do this, see our Advanced Usage of the Utils package page and the utils configuration variables.

Finally, before we dig into the details, starting with this version all future releases of our dbt packages will be released under the Snowplow Community License instead of the Apache 2 license. For what this means for you, check out our FAQ, and our announcement blog.

Features

• Create the "base" macros to enable greater flexibility
• Update the package to be under the Snowplow Community License
• Released a fix where get_optional_fields returned field aliases even when include_field_alias was set to false (#132)
• Add multi-version support to the get_field macro for BigQuery
• Remove deprecated snowplow_incremental materialization

Under the hood

• Introduce integration tests to check customization

Upgrading

To upgrade, bump the package version in your packages.yml file.

snowplow-utils v0.15.0-rc1

Summary

This is a pre-release version of the package, we believe it to be in working condition but you may encounter bugs and some features may change before the final release.

This release introduces new "base" macros which are used to recreate the snowplow_base_events_this_run and related tables. It gives you the option to customise your session and user identifiers, your partitioning timestamps, and much more. We have temporary docs hosted here that you can use to explore docs related to these changes. Once this is fully released, these docs will migrate over to the offical docs site

For information about how to do this, see our Advanced Usage of the Utils package page and the utils configuration variables.

Finally, before we dig into the details, starting with this version all future releases of our dbt packages will be released under the Snowplow Community License instead of the Apache 2 license. For what this means for you, check out our FAQ, and our announcement blog.

Features

• Create the "base" macros to enable greater flexibility
• Update the package to be under the Snowplow Community License
• Released a fix where get_optional_fields returned field aliases even when include_field_alias was set to false (#132)

Under the hood

• Introduce integration tests to check customization

Upgrading

To upgrade, bump the package version in your packages.yml file.

snowplow-utils v0.14.3

Summary

This version adds the ability to specify which package is returning the run limits in the print_run_limits macro, and fixes an issue with using string agg in Databricks with many null values.

Features

• Add ability to tag package in limits (Close #133)

Fixes

• Fix issue with large volume of nulls in databricks string agg

Upgrading

To upgrade bump the package version in your packages.yml file.

snowplow-utils v0.14.2

Summary

This version adds the get_field macro to enable extracting fields from our self-describing-event and context columns easier and more consistent across warehouses.

Features

• Add new macro get_field to make it easier to extract fields from unstruct/context columns across warehouses

Upgrading

To upgrade bump the package version in your packages.yml file.

snowplow-utils v0.14.1

Summary

This version makes changes to the new get_sde_or_context macro to support returning multiple entities per context, and changes the join column names to reduce the risk of clashes.

🚨 Breaking Changes 🚨

• Column names output by the get_sde_or_context macro have been changed from <context_name>_id/tstamp to <context_name>__id/tstamp. There is no need for you to make any changes unless you have used this macro in a custom model.

Features

• Add support for contexts with multiple values per event to the get_sde_or_context macro

Upgrading

To upgrade bump the package version in your packages.yml file.

snowplow-utils v0.14.0

Summary

This version makes some big changes by deprecating our snowplow_incremental materialization and a few of our other macros, with the goal of providing a simpler usage experience and easier maintenance of the package going forward. We've also added a brand new macro, get_sde_or_context to aid working with Self Describing Events or Contexts for our Redshift/Postgres users.

🚨 Breaking Changes 🚨

Deprecated snowplow_incremental materialization

We have deprecated the snowplow_incremental materialization and will be removing it entirely in a future version. In place we are providing an optimization on top of the built-in incremental materialization. To use this optimization for incremental materialized models, each model config must have snowplow_optimize=true and the following must be added to the top level of your dbt_project.yml file:

# dbt_project.yml
...
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']

For more information see here.

Deprecated macros

The get_cluster_by and get_partition_by macros have also been deprecated and will be removed in a future version. These should be replaced by get_value_by_target_type which offers the same functionality but more generally.

type_string and type_max_string

type_string has been entirely removed from the package and should be replaced with calls direct to dbt.type_string() instead. In all cases except redshift this should be suitable, for redshift when you need a column of length greater than 256 we provide type_max_string instead. For all other warehouses this is just a wrapper to dbt.type_string().

Features

• Deprecate get_cluster_by and get_partition_by macro in favor of get_value_by_target_type
• Remove type_string() and rework type_max_string() to prioritize dbt logic where possible
• Deprecated old materialization
• Add new get_sde_or_context macro

Under the hood

• Remove all internal references to snowplow_incremental materialization
• Migrate tests to new materialization approach

Docs

• Update readme

Upgrading

To upgrade bump the package version in your packages.yml file, and follow our migration guide for the above breaking changes.

snowplow-utils v0.14.0-rc2

Summary

This is a pre-release version of the package, we believe it to be in working condition but you may encounter bugs and some features may change before the final release.

This version fixes a few issues from the first release candidate, including failing when you had no data in a scratch table, and deprecates some macros.

As a reminder Users will need to add the following to their dbt_project.yml to benefit from the enhancements:

# dbt_project.yml
...
dispatch:
  - macro_namespace: dbt
    search_order: ['snowplow_utils', 'dbt']

For custom models and more details, please refer to our temporary docs page: https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-advanced-usage/dbt-incremental-logic-pre-release/

Features

Deprecate get_cluster_by and get_partition_by macro in favor of get_value_by_target_type
Remove type_string() and rework type_max_string() to prioritize dbt logic where possible
Fix inability to progress when scratch table contained no data
Ensure type consistency for the upsert_date_key throughout query

Under the hood

Remove all internal references to snowplow_incremental materialization
Migrate tests to new materialization approach