I am looking to sync a Amazon S3 bucket (source) exactly into Backblaze B2 cloud (destination) every week on Sunday. The source S3 bucket is around 5 TB so I'd like to utilize delta syncing. My understanding is that I cannot use sync directly with a source of S3 and B2. So then, what is the best way in terms of performance of doing this?
Must I copy from the S3 source locally to disk, then run sync from the local files to B2 Cloud? Or, can I use rclone mount to mount the S3 bucket locally and then use rclone sync ? Is there a better solution?
Can you provide example commands? Thanks so much.
What is your rclone version (output from rclone version)
to copy from s3 to b2, you need to create two remotes.
one remote for s3 and one remote for b2,
let's call them s3: and b2:
to simplify it a bit; the command would be something like. rclone sync s3: b2:
and for a file in s3: that needs to be copied to b2:,
rclone will read a chunk at a time from s3: to memory and copy that chunk from memory to b2:
rclone will not need to download the entire file to from s3: to local and then copy the entire file to b2:
rclone will not use the local disk storage.
if you do not want to use your local computer and bandwidth, then rent a virtual machine from aws, and run rclone on that.
I cannot for the life of me find my Backblaze Account ID or Application Key ID? I have a master application key keyID? Am I supposed to put that twice in rclone config?
Account ID or Application Key ID
Enter a string value. Press Enter for the default ("xxxxxxxxxxxxxx").
account> xxxxxxxxxxxxxx
Application Key
Enter a string value. Press Enter for the default ("xxxxxxxxxxxxxx").
key> xxxxxxxxxxxxxx
One more question. I would like to self encrypt files before syncing from s3 to Backblaze B2 cloud. Is there a way to do this natively with Rclone while maintaining sync (delta functionality)?
delta functionality, rclone does not support that.
rclone, unlike rsync, can only copy entire files.
if you have a source file of size 100GB, that has been copied from s3 to b2
and then you change one byte of the source file, rclone sync will copy 100GB, not one byte.
as for encryption, you can create a rclone crypt remote at b2 https://rclone.org/crypt/ and sync to that.
Thanks I got crypt remote working. Delta functionality in terms of sync is good enough, i.e. if a byte changes in a file sync the entire file again is fine.