Stream Binlogs to S3

[tanmoysrt]

We have database server generating 100~500MB of binlog every minute. It's not possible to keep them stored in disk. We need to ship it to some cold storage. One way is to send them to S3.

Now the issue is, we need to search through files using mysqlbinlog cli. Now if we upload stuff to S3, we can;t do that. Because it's not possible to download 24hr of binlog and search through those.

Need some kind of pre-process and search index system.

So, the main challenge of this is Search Functionality.

Already there is an attempt for S3 upload > https://github.com/pboros/binlog2s3

Initial Idea -
We need to pre-process the binlogs and keep track of database events in binlog and there offsets for faster search.

Some Learning Material -

• https://medium.com/@mahadir.ahmad/reading-mysql-binlog-with-python-c85a8ece3b28
• https://mariadb.com/kb/en/replication-protocol/
• https://github.com/julien-duponchelle/python-mysql-replication