Delay before rclone uploads files, a newbie question

What is the problem you are having with rclone?

i am loving rlcone. very quickly i have become depdendent on it. good that i donated.
the problem is that there is a delay before uploading each file. the larger the size of the file, the larger the time delay before upload starts. i am guessing that rclone is doing a checksum of the local file before upload to wasabi. is this correct?
if so, is there a way to get rclone to calculated the checksum on the fly, as rlcone is uploading the file?
thanks much,
david

What is your rclone version (output from rclone version)

1.49

Which OS you are using and how many bits (eg Windows 7, 64 bit)

windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

2019/08/29 15:19:58 DEBUG : rclone: Version "v1.49.1" starting with parameters ["C:\data\rclone\scripts\rclone.exe" "sync" "b:\veeam-en07-west\EN07\veeam-en07\EN07" "wasabiwest01:westvserver03.veeam-en07-west\backup\" "--stats=0" "--progress" "--transfers" "2" "--backup-dir=wasabiwest01:westvserver03.veeam-en07-west\archive\20190829_151954\" "--log-level" "DEBUG" "--log-file=C:\data\rclone\logs\veeam-en07-west\20190829_151954\veeam-en07-west_20190829_151954_rclone.log"]

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

When it has to upload a file, it has to calculate the MD5SUM for the file so that is the delay you are seeing:

https://rclone.org/s3/#s3-disable-checksum

You can disable the checksum depending on how critical you feel the data is.

Animosity022,

i use rclone to upload large veeam backup files to wasabi.

i am very paranoid so the md5sum time delay is worth it.

i know that rclone 'propaganda' likes to compare itself to rsync, tho rclone seems nothing like rsync to me. rsync does rolling checksums.

i assume that since rclone is using aws s3 sdk, that rolling checksum is not possible.

thanks,
david

I personally wouldn't compare the two as I can't mount my cloud storage with rsync.

They have similar things but are really very different imo.

i just updated to v1.49.1
i see that v1.49.1 has changed the log files to make it clear that rclone is doing a checksum before upload.
2019/08/29 15:49:13 DEBUG : EN072019-05-09T073008.vbk: MD5 = 388b5464c416598fd3e55ddbdecfa258 OK
2019/08/29 15:49:13 INFO : EN072019-05-09T073008.vbk: Copied (new)

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.