Skip to content

Create (dataless) transaction to advance replication log for postgres RDS #58

Open
@drdee

Description

@drdee

When you have a Postgres database in Amazon RDS that receives very little or no new data, the hard drive will fill-up with unreleased wal file segments. The reasons for that is (according to AWS support team):

In RDS, we write to a heartbeat table in our internal "rdsadmin" database every 5 minutes, so RDS
 instances are usually never "idle" for longer than 5 minutes, whereas PostgreSQL installed on EC2
 machine may appear "idle" indefinitely. On RDS hosts, we periodically write heartbeats to our admin 
DB on the host. As a result, even if there is no user activity, WAL files are generated. 

A similar issued affected Debezium and is discussed at https://issues.jboss.org/browse/DBZ-892?attachmentViewMode=gallery&_sscc=t

One way of fixing this is if tap-postgres could create a (dataless) transaction that advances the replication and thereby RDS releases the unused WAL files and preventing the hard drive from filling up. We have implemented our own _heartbeat table on the RDS databases and upsert the current timestamp in that table. Perhaps this is something that tap-postgres can do out-of-the-box.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions