Redshift generates text fields that are too small

[JasonSanDiego]

When syncing from MySQL to Redshift, a table with varchar >256 leads to a varchar(256) in Redshift, leading to errors in Redshift like:

Value too long for type character varying(256)

For example, my MySQL table has a varchar(500) but pipelinewise (really the underlying Singer target) only generates a varchar(256).

I was able to workaround this by manually changing the size to varchar(500).

[JasonSanDiego]

I did some digging around and it looks like the Redshift target is fully custom to Pipelinewise. It looks totally different from the official Singer Redshift target. I also saw that the default length of characters is supposed to be 10K chars. So, I'm not sure why all of my text fields are 256 by default, and not expanding as needed.

Perhaps, the MySQL tap isn't generating the right properties file? How custom is that tap relative to the official Singer one?

[koszti]

You're right, target-redshift is written from scratch for PPW but as far as I know there is no official open source target-redshift connector by Singer. There is a target-redshift connector written by DataMill: https://github.com/datamill-co/target-redshift , but there are certain differences in the two implementations, for example denesting objects, or adding new columns.

Regarding your issue I think this seems to be a bug in the mysql-to-redshift fastsync. Fastsync and target-redshift column mappings should be in sync but seems like it's not the case. But fixing it looks relatively straightforward:

Colum mapping:

• mysql-to-redshift Fast Sync
• target-redshift Singer