Stream MySQL binlogs to AWS S3

Help and Support
1

Hi,

Is there a way to use rclone to continuously stream MySQL binlogs to AWS S3 bucket?

Regards,

Kaushal

2

Hi,

I will appreciate if somebody can pitch in for help for my earlier post to this mailing list.

Regards,

Kaushal

3

rclone doesn’t stream so regular syncing of a huge changing file will caused it to be uploaded afresh each time.

Can you make lots of smaller binlogs? rclone would be good at syncing those to an s3 bucket.

4

@ncw Thanks for the reply. I have downloaded and configured rclone in CentOS Linux version 6.9
when i do rclone lsd remote: it lists out all the buckets in AWS S3 and when i do aws s3 ls s3://pg-backups/logs/pgs1/ it lists out all the files but when i run the command

rclone ls remote:pg-backups/logs/pgs1/ I face the below issue

017/07/27 17:43:00 ERROR : : error listing: BucketRegionError: incorrect region, the bucket is not in ‘ap-southeast-1’ region
status code: 301, request id: , host id:
2017/07/27 17:43:00 Failed to ls: BucketRegionError: incorrect region, the bucket is not in ‘ap-southeast-1’ region
status code: 301, request id: , host id:

rclone config
[remote]
type = s3
env_auth = true
access_key_id = xxxxxxxxxxxxxxxxxxx
secret_access_key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
region = ap-southeast-1
endpoint =
location_constraint = ap-southeast-1
acl = private
server_side_encryption =
storage_class =

Any help will be highly appreciable.

Regards,

Kaushal

5

You need to create the bucket in the same region that you configured rclone to access. This might mean reconfiguring rclone, or deleting the bucket and remaking it.

6

@ncw Thanks a lot and it worked perfectly. I have a followup question regarding running of rclone utility in continuous mode. How do i ensure it is always up and running and syncing any changed files in a particular directory which should get pushed to AWS S3 bucket.

Thanks again for all the help. Please suggest.

Regards,

Kaushal

7

Rclone doesn’t have a continuous running mode yet. Just run it off the crontab at regular intervals.

8

Hi Nick,

I am facing issues when i do a copy or a sync operation using rclone. The issue details are described in http://sprunge.us/EagU

The file -rw-rw----. 1 mysql mysql 290796327 Jul 31 13:05 mysqld-bin.000031 is written continuously and the file size is increasing. I am using rclone v1.37 and running CentOS release 6.8 (Final) in my setup.

Please let me know if i am missing anything.

Thanks in Advance.

Regards,

Kaushal

9

I’m afraid that is a problem with rclone - it isn’t good at uploading files which are changing.

Can you snapshot the file system or maybe rotate the binlogs?

10

Hi Nick,

Thanks for the reply. Please help me understand about snapshot the file system or maybe rotate the binlogs. Are you saying to take backup of the filesystem using standard backup tool and what should be the approximate filesize of MySQL binlog which has to be rotated which rclone tool can push it efficiently to AWS S3 bucket.

Please comment. Thanks in Advance.

Regards,

Kaushal

11

Depending on which filesystem you are using (eg ZFS or something on LVM), you could snapshot it. Then you’ll have a consistent backup.

rclone can’t sync stuff which is being written to. So how much of the binlog can you afford to lose? Say it is 5 minutes, then you rotate the binlogs every 5 minutes.

Another alternative might be to manually snapshot the binlog with cp to a different directory then sync that.