Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create (dataless) transaction to advance replication log for postgres RDS #58

Open
drdee opened this issue Jul 3, 2019 · 0 comments
Open

Comments

@drdee
Copy link
Contributor

drdee commented Jul 3, 2019

When you have a Postgres database in Amazon RDS that receives very little or no new data, the hard drive will fill-up with unreleased wal file segments. The reasons for that is (according to AWS support team):

In RDS, we write to a heartbeat table in our internal "rdsadmin" database every 5 minutes, so RDS
 instances are usually never "idle" for longer than 5 minutes, whereas PostgreSQL installed on EC2
 machine may appear "idle" indefinitely. On RDS hosts, we periodically write heartbeats to our admin 
DB on the host. As a result, even if there is no user activity, WAL files are generated. 

A similar issued affected Debezium and is discussed at https://issues.jboss.org/browse/DBZ-892?attachmentViewMode=gallery&_sscc=t

One way of fixing this is if tap-postgres could create a (dataless) transaction that advances the replication and thereby RDS releases the unused WAL files and preventing the hard drive from filling up. We have implemented our own _heartbeat table on the RDS databases and upsert the current timestamp in that table. Perhaps this is something that tap-postgres can do out-of-the-box.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant