You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
Pipelinewise is not respecting the settings in target yml file for add_metadata_columns= False
All fastsync supported taps may need to be adjusted, per Peter Kosztolanyi.
tap-mysql-fastsync adding metadata columns [(https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_mysql.py#L261)]
tap-postgres-fastsync adding metadata columns [https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_postgres.py#L407]
For a Snowflake target and Postgres tap running in Singer mode, the metadata columns are being back to the table, even when manually deleted from the Target table post initial sync.
To Reproduce
Steps to reproduce the behavior:
Target yml file includes lines:
# ------------------------------------------------------------------------------
# General Properties
# ------------------------------------------------------------------------------
id: "snowflake3" # Unique identifier of the target
name: "Snowflake" # Name of the target
type: "target-snowflake" # !! THIS SHOULD NOT CHANGE !!
add_metadata_columns: False
For an existing Postgres tap - Snowflake target pipeline, manually delete the three metadata columns from the target table
alter table xyz drop column _SDC_DELETED_AT,_sdc_extracted_at, _SDC_BATCHED_AT;
Run the pipeline.
Check Snowflake table or log and see that columns have been added back in.
Expected behavior
I would expect the metadata columns not to be added when add_metadata_columns= False.
Screenshots
If applicable, add screenshots to help explain your problem.
cap-itadmin
changed the title
add_metadata_columns and hard_delete flags not being recognized in either fastsync or singer sync
add_metadata_columns flag not being recognized in either fastsync or singer sync
Apr 17, 2021
Describe the bug
Pipelinewise is not respecting the settings in target yml file for add_metadata_columns= False
All fastsync supported taps may need to be adjusted, per Peter Kosztolanyi.
tap-mysql-fastsync adding metadata columns [(https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_mysql.py#L261)]
tap-postgres-fastsync adding metadata columns [https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_postgres.py#L407]
For a Snowflake target and Postgres tap running in Singer mode, the metadata columns are being back to the table, even when manually deleted from the Target table post initial sync.
To Reproduce
Steps to reproduce the behavior:
For an existing Postgres tap - Snowflake target pipeline, manually delete the three metadata columns from the target table
alter table xyz drop column _SDC_DELETED_AT,_sdc_extracted_at, _SDC_BATCHED_AT;
Run the pipeline.
Check Snowflake table or log and see that columns have been added back in.
Expected behavior
I would expect the metadata columns not to be added when add_metadata_columns= False.
Screenshots
If applicable, add screenshots to help explain your problem.
Your environment
Additional context
Link to discussion on Slack Singer Pipelinewise channel: https://singer-io.slack.com/archives/CNL7DL597/p1617988746067000
The text was updated successfully, but these errors were encountered: