Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add_metadata_columns flag not being recognized in either fastsync or singer sync #692

Open
cap-itadmin opened this issue Apr 14, 2021 · 1 comment
Labels
bug Something isn't working

Comments

@cap-itadmin
Copy link

cap-itadmin commented Apr 14, 2021

Describe the bug
Pipelinewise is not respecting the settings in target yml file for add_metadata_columns= False

  1. All fastsync supported taps may need to be adjusted, per Peter Kosztolanyi.
    tap-mysql-fastsync adding metadata columns [(https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_mysql.py#L261)]
    tap-postgres-fastsync adding metadata columns [https://github.com/transferwise/pipelinewise/blob/ac926d7fe0322e8382850f4889fac85cd316384f/pipelinewise/fastsync/commons/tap_postgres.py#L407]

  2. For a Snowflake target and Postgres tap running in Singer mode, the metadata columns are being back to the table, even when manually deleted from the Target table post initial sync.

To Reproduce
Steps to reproduce the behavior:

  1. Target yml file includes lines:
# ------------------------------------------------------------------------------
# General Properties
# ------------------------------------------------------------------------------
id: "snowflake3"                          # Unique identifier of the target
name: "Snowflake"                      # Name of the target
type: "target-snowflake"             # !! THIS SHOULD NOT CHANGE !!

add_metadata_columns: False
  1. For an existing Postgres tap - Snowflake target pipeline, manually delete the three metadata columns from the target table
    alter table xyz drop column _SDC_DELETED_AT,_sdc_extracted_at, _SDC_BATCHED_AT;

  2. Run the pipeline.

  3. Check Snowflake table or log and see that columns have been added back in.

Expected behavior
I would expect the metadata columns not to be added when add_metadata_columns= False.

Screenshots
If applicable, add screenshots to help explain your problem.

Your environment

  • Running in Docker
  • Source: Postgres
  • Target: Snowflake
  • Using Log Based Replication

Additional context
Link to discussion on Slack Singer Pipelinewise channel: https://singer-io.slack.com/archives/CNL7DL597/p1617988746067000

@cap-itadmin cap-itadmin added the bug Something isn't working label Apr 14, 2021
@cap-itadmin cap-itadmin changed the title add_metadata_columns and hard_delete flags not being recognized in either fastsync or singer sync add_metadata_columns flag not being recognized in either fastsync or singer sync Apr 17, 2021
@Saadmairaj
Copy link

Facing the same issue when syncing from s3 to Postgres as well. Found any solution or workaround to this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants