Syncing large S3 bucket to B2 Cloud

What is the problem you are having with rclone?

I am looking to sync a Amazon S3 bucket (source) exactly into Backblaze B2 cloud (destination) every week on Sunday. The source S3 bucket is around 5 TB so I'd like to utilize delta syncing. My understanding is that I cannot use sync directly with a source of S3 and B2. So then, what is the best way in terms of performance of doing this?

Must I copy from the S3 source locally to disk, then run sync from the local files to B2 Cloud? Or, can I use rclone mount to mount the S3 bucket locally and then use rclone sync ? Is there a better solution?

Can you provide example commands? Thanks so much.

What is your rclone version (output from rclone version)

rclone v1.56.0
- os/version: darwin 11.5 (64 bit)
- os/kernel: 20.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.16.6
- go/linking: dynamic
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu Linux, 64 bit.

Which cloud storage system are you using? (eg Google Drive)

Amazon S3 and Backblaze B2 Cloud.

to copy from s3 to b2, you need to create two remotes.
one remote for s3 and one remote for b2,
let's call them s3: and b2:

to simplify it a bit; the command would be something like.
rclone sync s3: b2:

and for a file in s3: that needs to be copied to b2:,
rclone will read a chunk at a time from s3: to memory and copy that chunk from memory to b2:
rclone will not need to download the entire file to from s3: to local and then copy the entire file to b2:
rclone will not use the local disk storage.

if you do not want to use your local computer and bandwidth, then rent a virtual machine from aws, and run rclone on that.

Thanks for the reply @asdffdsa.

I cannot for the life of me find my Backblaze Account ID or Application Key ID? I have a master application key keyID? Am I supposed to put that twice in rclone config?

Account ID or Application Key ID
Enter a string value. Press Enter for the default ("xxxxxxxxxxxxxx").
account> xxxxxxxxxxxxxx
Application Key
Enter a string value. Press Enter for the default ("xxxxxxxxxxxxxx").
key> xxxxxxxxxxxxxx

that is documented at https://rclone.org/b2/#application-keys
"you can't use the master Account ID"
"Follow Backblaze's docs to create an Application Key"

i do not use B2, but a quick search....
https://www.backblaze.com/blog/b2-application-keys/

One more question. I would like to self encrypt files before syncing from s3 to Backblaze B2 cloud. Is there a way to do this natively with Rclone while maintaining sync (delta functionality)?

delta functionality, rclone does not support that.
rclone, unlike rsync, can only copy entire files.

if you have a source file of size 100GB, that has been copied from s3 to b2
and then you change one byte of the source file, rclone sync will copy 100GB, not one byte.

as for encryption, you can create a rclone crypt remote at b2
https://rclone.org/crypt/ and sync to that.

Thanks I got crypt remote working. Delta functionality in terms of sync is good enough, i.e. if a byte changes in a file sync the entire file again is fine.

rclone sync --fast-list --progress --transfers 32 s3:acme-playground encrypted:

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.