Nick giving a talk about Rclone and Backblaze

Annoucements
1

On Thursday 17th Sep (1800 UK time, 1000-1300 US time) I'm going to be giving a webinar with Backblaze, talking in general about rclone and in particular about the B2 integration.

I'd love you all to come and listen and ask questions.

Anything you'd like me to prepare in advance?

Please sign up here if you are interested :slight_smile:

10 Likes
3

Your thoughts on upcoming features. Will rclone become raid for the cloud?
https://forum.rclone.org/t/union-fail-over/18928/3

Rclone was rsync for cloud storage. When to use rsync, and when rclone?

What aspects of rclone made it so appealing, as a tool, to the ProLock hackers?
https://www.bleepingcomputer.com/news/security/fbi-issues-second-alert-about-prolock-ransomware-stealing-data/

2 Likes
4

Nice questions thank you @Edward_Barker

PS I had no idea rclone was being used by the ProLock hackers! I guess all tools can be used for good and evil!

5

Why b2 don't compute the checksum of files themselves? I always have to use the option to disable checksum or it would take forever to upload my stuff, while other clouds have no issues with this?

It makes me don't trust any data I store there for archiving,etc...

6

Are you talking about for large files uploaded in chunks?

Each chunk has a strong checksum, but they just don't compute the checksum of the whole file.

AWS S3 works in exactly the same way.

7

Yes...I'm talking about large files... I still don't see why I would trust b2 more than google drive, where i can check the hash with my local file

8

That is a good point and worth discussion if we have time.

9

I have actually put a lot of thought into this since I wrote a few sync tools that wrap rclone and/or rsync.

The thing to remember is that rsync is actually two things:

  1. A utility to mirror/one-way-sync one computer to another (but they must be local on one side and ssh (sftp) or local on the other)
  2. An algorithm to propagate small changes of a large file efficiently.

Number 2 is actually a HUGE benefit to rsync. If you have a 10gb file and insert a few bytes at an arbitrary location, rsync will very efficiently make that change. It does this based on rolling checksums and is really clever. It is worth noting that if two files have the same name but completely different content, the algorithm will end up with the correct transfer but slightly less efficient as it is trying to be efficient with the wrong data.

rclone is just number one. It is a tool to mirror (one way again) a set of files and directories from one side to the other. The big difference is that while rsync can only handle ssh (sftp), rclone handles basically everything under the sun. That's where the benefit comes from. And it is a HUGE benefit. As for things like number 2, rclone will repush the entire file.

rclone also has, what I would argue to be a more reasonable API. First, it always does the equivalent of the rsync -a flag. And it skips the nonsense about trailing / that rsync is crazy about (it's not bad once you understand it but it takes some time). However, in addition to the many storage options, it offers some functions rsync doesn't such as rename tracking.

rclone is also SO MUCH MORE than just sync. It has encryption, mounting, serving, interfacing, etc. Some of those are pretty major and useful features.

rsync can also do some tricks that rclone can't. Part of that is because it knows it is only working on real file systems (and often POSIX ones). So it can do things with hardlinks including hardlinking unchanged files and also recreating them on the destination.

So what is the answer to your question:

  • If you will only ever be syncing local to local or SSH and local and you want to efficiently transfer small changes to large files, use rsync.
  • If you want to sync with things other than SSH, you really have no choice but to use rclone. Also while rsync over SSH is encrypted in transit, rclone offers at-rest encryption.
    • Even if it's just SSH, there is benefit to using rclone as you learn the tool that can do it all.
    • (edited to add) rclone can also multi-thread transfers while rsync is serial. I’ve gotten better speeds on local <—> local transfers than rsync because it can do more than one at a time
4 Likes
10

From my perspective as a viewer that went well. Came over very genuine.

The most heartfelt @ncw quote might have been "rclone has a lot of flags".

11

Thanks! Glad you enjoyed it :slight_smile:

Ha ha!

That is my new career, flag hoarder :wink:

12

After watching the stream, had to listen to this :grinning:
Baha Men - Who Let The Dogs Out (Original version)

Also discover the rclone help flags, seriously...

Nice presentation, and thanks for the "team" slide, make me better understand it. :+1:

2 Likes
13

Very good presentation, thanks!

Got to echo about the team slide - gives some perspective!

And we got a handy certificate of attendance :stuck_out_tongue:

1 Like
14

He he! I'm sure that will be useful at your next job interview :wink: