Fastsync and ordinary sync result in different column types

[mhindery]

Describe the bug
When syncing from mysql to postgres, first it does a sync using fastsync. This creates columns with a certain type. Then when running the sync again, it used binlog sync, and it seems to decide it is now a different type. This results in a second column being added and the 'old one' (which was created seconds ago) being renamed. The source database table has had no changes in the meantime. This breaks a lot of things for us, as there are random additional columns being added and the data is not present in the columns they should be.

To Reproduce
I've seen this happen on integer-type columns. E.g. a column which is in mysql int(10) unsigned, fastsync creates a corresponding int4 postgres column for it, and then the ordinary sync makes a int8 additional column afterwards.

Expected behavior
Both to use the same types, so there are no additional columns generated when not necessary.

Screenshots
/

Your environment
PipelineWise 0.25.0 - Command Line Interface

MySQL to Postgres replication

[mhindery]

On a sidenote, it is incredibly frustratring that fastsync cannot be disabled without manually editing the source code of the package and installing a modified version. This syncing method is your custom implementation which we've had several problems with in the past. I'd rather have a working synchronization than a fast one.

[koszti]

hey @mhindery I'm sorry to hear your pain.

Making fastsync conditional is currently not on the roadmap but I'm sure if you can send a PR that keeps the fastsync to be enabled by default then we can review, give feedback and can merge it.

Regarding your original problem, seems like the fastsync mysql-to-postgres type mapping and the "ordinary" sync type mapping are not in sync. We don't actively use postgres target apart from some testing, so if you're confident how to fix it please go ahead, send PR and we'll review.

What do you think, would that work for you?

[mhindery]

I have made a PR to allow turning off fastsync here: #697