Deciding whether or not Rclone would be better for me

Hi,

I want use my Onedrive (1tb) account to backup some (encrypted) large files and Rclone seems to work very nicely. But unfortunately my upload speed is horrible (2.5 MB/s at max) and the internet sometimes is unstable (it may drop out). It seems Rclone can't resume the upload, so if I'm uploading a 3GB file, for example, and I loose my internet connection for a brief moment, the file will need to be uploaded from the beginning.

I'm also testing Cryptomator for the same purpose, and it seems to allow resuming the upload (or download). Unfortunately it works with local sync only.

So, considering the problems that I have with my upload speed and that I can't resume my uploads using Rclone, does it means Cryptomator is my only option or is there anything else that could mitigate my problem when using Rclone?

Thanks.

Rclone should be able to continue an upload if the internet drops provided you don't stop rclone.

Have you got a log of this?

Just to increase my own understanding of this - what is the threshold of when an upload can be resumed? Is it when the spesific connection is either closed or times out?

And does that continue at the end of the last chunk? .. or is it at the TCP packet level?

Sorry, I don't want to derail the thread, but it would be nice to have a better grasp of the underlying mechanics of resume :slight_smile:

No, unfortunately I don't have a log.

Today I finished transferring a folder to my Onedrive account, encrypted with Cryptomator, with more than 50 files (each one averaging about 3GB). It took me a few days to upload the whole thing, including some periods that I paused the synchronization multiple times to reboot my computer or to use my upload for others tasks, and one or two internet drop as well (that I needed to reboot my cable modem).

Then, I downloaded the whole thing from the cloud (my download speed is much better) just to do a checksum (comparing with the local source), and it was perfect.

I guess Rclone can't offer the same flexibility?

i thought that onedrive is not for backups, but for syncing lots of small files.
i have office365 with 1TB of cloud storage and i have found it is not suitable for backup of large files.
i use veeam agent and veeam backup and replication, and tho veeam agent can backup to onedrive, it is a joke.

i use wasabi s3 provider and verizon fios gigabit internet connection.
for uploads, i average 1.8GB/minute

please know that for uploads, rclone will do a md5 checksum for each local file before uploading to cloud.
that takes time and given the way rclone displays progress, it appears that actual upload throughput is much slower.
i am super paranoid, i want that rclone to do a md5 checksum for each upload.

you can do some testing such flag --size-only, which skips the md5 checksum for each upload file.

here is a log for uploading a file, including the md5 checksum.

2019/09/20 17:04:11 DEBUG : rclone: Version "v1.49.1" starting with parameters ["C:\data\rclone\scripts\arclone.exe" "sync" "v:\EN07\en07.veaamfull\EN07" "wasabiwest01:vserver03-en07.veaamfull\backup\" "--stats=0" "--progress" "--backup-dir=wasabiwest01:vserver03-en07.veaamfull\archive\20190920.170411\" "--log-level" "DEBUG" "--log-file=C:\data\rclone\logs\en07.veaamfull\20190920.170411\en07.veaamfull_20190920.170411_rclone.log"]
2019/09/20 17:28:01 DEBUG : EN072019-09-14T104415.vbk: MD5 = 892315144346cdfab149475df9e4aafb OK
2019/09/20 17:28:01 INFO : EN072019-09-14T104415.vbk: Copied (new)
2019/09/20 17:28:01 INFO : Waiting for deletions to finish

24 minutes to upload 40.2GB is 1.8GB/minute

I split files that are more than 15GB before uploading.

Since your upload speed is so much better than mine, it's a no issue. Since my upload is much slower, I need to deal with possible interruptions.

It wouldn't be nice to upload 90% of a 15GB file, having the uploaded interrupted, and loosing two hours because I need to upload the whole thing again.

If rclone is uploading using a chunked upload method then rclone can retry the individual chunks providing some protection to things going wrong.

There could be a bug in this which is why I want to see a log (with -vv) if possible!

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.