Rclone *really* long-term run report, and thanks

Praise
1

Hello everyone,

I'm glad to report that a really long run for rclone has finished here a few days ago:

rclone -v --low-level-retries=1000 copy egd: egdl:
    [...]
    2018/05/29 17:21:27 INFO  :
    Transferred:   14886.291 GBytes (717.001 kBytes/s)
    Errors:                15
    Checks:          23811993
    Transferred:     11466709
    Elapsed time:  6047h20m2.4s
    2018/05/29 17:21:49 INFO  : Encrypted drive 'egdl:': Waiting for checks to finish
    2018/05/29 17:21:49 INFO  : Encrypted drive 'egdl:': Waiting for transfers to finish
    2018/05/29 17:21:49 ERROR : Attempt 3/3 failed with 15 errors and: failed to open source object: bad response: 500: 500 Internal Server Error
    2018/05/29 17:21:50 Failed to copy: failed to open source object: bad response: 500: 500 Internal Server Error

As you can see above, it has taken more than 6047/24= 251 days (ie, over 8 months) to run; it was a copy of one of my entire Google Drive remotes to a different one.

Just 15 errors on this entire set of over 34M files is what I' d call an accomplishment, given Google Drive vagaries. Worth of note is also the fact that I've monitored that rclone process during the entire time it ran, and it always kept a reasonable memory footprint (as reported by ps auwx VSIZ and RSS fields), so I'd bet rclone has no memory leaks.

Thanks again to @ncw for this amazing tool! rclone is certainly one of the programs I use most, right after the Linux kernel, screen, bash and vi!

Cheers,
-- Durval.

2 Likes
2

Very interesting!

A curiosity: did this job run from a local machine or from a virtual cloud machine?

3

Hi TowerBR,

I ran it from inside a Docker container on a Contabo VPS-M VPS which I installed from the ground up with Devuan Linux Jessie.

Speaking of curiosity, is the “BR” on your nickname meant as an abbreviation for Blue Ray, or the 2-letter country code for BRazil, or something else? :wink:

Cheers,
– Durval.

4

Interesting, I didn’t know, good prices …

BR really stands for Brazil, it’s because the nick “Tower” was already taken years ago when I created this :smile:

And by your surname, it’s your country too, right? :wink:

5

Yeah, I’m from Brazil too :wink:

6

Wow, that is amazing! Thanks for sharing that.

Thank you for your kind words, and also for your excellent bug reports :slight_smile:

7

I can say simply, that we run rclone on over 1200 clients PC’s to backup their inventory images and keep them synced to our GCS Bucket. Everyone uploads daily, sending a config and a key to the folder while it runs, and then deleting it afterwards.

rclone helped so much, I cant thank the developers everyone enough for this fine piece of code. It’s always in my toolbox when the time comes.

8

That is really good to hear :smile:

My ambition for rclone was it to be the tool everyone thinks of when transferring data to object storage in a script, like rsync is!