-
Notifications
You must be signed in to change notification settings - Fork 126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Violates not-null constraint #648
Comments
Hey, I'm not sure I'm understanding your questions, but I'll try. re the columns re |
Dear @Samira-El , Follow your instruction, I keep going to run command $ pipelinewise sync_tables --target postgres_dwh --tap mysql_web_warehouse --tables pushsale.users And everything from our data is correct. ❤️ Keep thing about that, I updated the data source in the MySQL database and run the pipeline again and got an exception, and the data become the incorrect schema. $ pipelinewise run_tap --tap mysql_web_warehouse --target postgres_dwh
I always keep to trying run |
FastSync command:
|
My tap-mysql configure like this, please help some ideas to find out the problem @Samira-El ---
id: "mysql_web_warehouse" # Unique identifier of the tap
name: "web_warehouse MySQL Database" # Name of the tap
type: "tap-mysql" # !! THIS SHOULD NOT CHANGE !!
owner: "[email protected]" # Data owner to contact
db_conn:
host: "mysql" # MySQL/ MariaDB host
port: 3306 # MySQL/ MariaDB port
user: "root" # MySQL/ MariaDB user
password: "password" # Plain string or vault encrypted
dbname: "pushsale" # MySQL/ MariaDB database name
target: "postgres_dwh" # ID of the target connector where the data will be loaded
batch_size_rows: 20000 # Batch size for the stream to optimise load performance
stream_buffer_size: 0 # In-memory buffer size (MB) between taps and targets for asynchronous data pipes
schemas:
- source_schema: "pushsale" # Source schema (aka. database) in MySQL/ MariaDB with tables
target_schema: "public" # Target schema in the destination Data Warehouse
target_schema_select_permissions: # Optional: Grant SELECT on schema and tables that created
- developer
tables:
- table_name: "advertisers"
replication_method: "LOG_BASED"
- table_name: "offers"
replication_method: "LOG_BASED"
- table_name: "leads"
replication_method: "LOG_BASED"
- table_name: "leads_meta"
replication_method: "LOG_BASED"
- table_name: "users"
replication_method: "LOG_BASED"
- table_name: "opportunities"
replication_method: "LOG_BASED"
- table_name: "opportunity_attributes"
replication_method: "LOG_BASED"
- table_name: "orders"
replication_method: "LOG_BASED" |
what's the column type of |
Are you have any solution to debug this case? Or something wrong with my configure @Samira-El |
Can you help me to debug this issue? @Samira-El |
Hey, Can you also share the detailed table information in of this table in postgres? |
Dear @Samira-El, My user table in Postgres: And detailed value is Postgres: |
In the process Pipelinewise run_tap, I checked my destination database and the table columns inserted are correct. And after the pipeline is finished. I re-check and it's wrong like the problem I described above. Logging information from Pipelinewise:
In fact, for one pipelinewise run_tap. We have 2 log files:
With what I did, Do you think I am configuring something wrong? @Samira-El |
Singer always tries to create a table but it exists because fastsync created 😮
|
I have the same problem when sync data from MySQL to Postgres. Any solutions? |
Your command:
pipelinewise run_tap --tap mysql_web_warehouse --target postgres_dwh
Why tap-mysql create a new column to target-postgres and foreign key always Null value?
Your question

And after the data source is updated, I re-run the pipeline again, and then it throws an error: violates not-null constraint.
Please help me fix this issue.
Details exception:
The text was updated successfully, but these errors were encountered: