this is my command:
rclone --retries-sleep=5m --retries 3 -vv sync \syn-ipc\backup-virtual#snapshot\GMT+02-2020.04.05-06.00.05\ ipc-veeam:veeam/GMT+02-2020.04.05-06.00.05/
version is v1.50.2
so i included 3 retries in my command and all of them failed same way.
ok updated rclone now.
its just first time i started this command, although i used rclone similary in past for other things.
when rclone is run, and I have alot of data to sync (around 3TB in 2 files which should be uploaded now, other files are already synced), those big files are read by rclone and this takes ages. why does it read the files from source on start (?) just to see that those files do not exist on destination and then fails copying to destination with error i pasted before.
rclone, by default, use checksums, when uploading files; so that delay on start is from that.
if you look in the logs, you will see
2020/04/11 14:38:40 DEBUG : EN07/EN072020-04-11T102901.vbk: MD5 = 0ada13e08229cc81dfd8a0bdc727424c OK
2020/04/11 14:38:40 INFO : EN07/EN072020-04-11T102901.vbk: Copied (new)
rclone saves that md5 checksum as meta data for the file transferred.
now, i could be wrong, but as i understand it, rclone has to set the meta data before upload.
so that forces rclone to calculate the checksum first, update the meta data and then upload the file.
this prevents rclone from doing a rolling checksum while uploading.
thus the delay
ok i figured that, is there a way for me to skip this - using checksum?
so only check size + modification date if they dont match on source and target - copy the files.
now, since i already waited a long time checksums should be calculated and ready but again it reads the big files from scratch which will take ages.
i believe that is the default behaviour, https://rclone.org/docs/#c-checksum Normally rclone will look at modification time and size of files to see if they are equal
i understand that, but i ment skipping calculating checksom for files on source.
thats whats causing the abnormal delay since to calculate checksum whole files must be read and that takes alot of time.
yes it is, how did you know?
i know it depends on those checksums, but they were already calculated once and that took maybe 15 hours of reading source files...so now i started command again its reading those big files again and calculating checksum (whereas it should have recorded checksum in metadata).
that large delay on start, is exacerbated by using your computer, accessing a networked synbox with raid subsystem.
perhaps run rclone on the synbox itself.
indeed that is cool way to compare checksums, but...
what you are saying to me is that it is normal for rclone, when syncing, always reads whole all of source files and there is no way around that (since it has to calculate checksum every time).
rclone, by default, when syncing, does not calculate checksum for files on the source.
rclone uses timestamp and file size to determine if a file needs to be uploaded.
it only calculates the checksum when uploading a file.
there might be a way to prevent rclone from check-summing an file before upload.
as i mentioned i use rclone to upload large veeam backup file and i need that check-summing.
a backup file that cannot be trusted, is not a backup
i suggest that you start a new post, use the question template and ask the following question such as
"how to force rclone to skip the initial checksum when uploading files large files"
else write your own script, with a simple csv database.
i have a 300+ python script that i use for rclone, fastcopy and rclone.
