Allow for custom Singer catalogs

[judahrand]

I guess in the context of Pipelinewise a Singer 'stream' is a 'table'. It would be nice to be able to edit the Pipelinewise 'tables' to not pull all columns. Is this something which could be supported?

[koszti]

This is currently not supported and every column selected automatically. In case of schema changes newly added columns are picked up automatically as well.

Defining which columns to extract is basically possible but requires some development. A possible solution would be to define it in the tap YAML files in this format:

schemas:
  - source_schema: "my_db"        # Source schema (aka. database) in MySQL/ MariaDB with tables
    target_schema: "my_db"        # Target schema in the destination Data Warehouse
    tables:
      - table_name: "table_one"
        replication_method: "LOG_BASED"   # One of INCREMENTAL, LOG_BASED and FULL_TABLE
        columns:
          - column_one
          - column_two