I have ~250,000 files, but a rclone sync performs ~700,000 checks. It seems to me the crypt remote also doubles the amount of files in the remote. Not sure I understand enough yet about this.
Is this config ok, or can I improve it?
Background: I am giving Hetzner storage box a try (tried sftp and webdav), which has max 10 concurrent connections. Found that a sync of the above files takes 45min but that is only a test subset of my data. Files are small. I thought cutting the number of files in half (if crypt config could be improved) or otherwise cutting the number of checks would help. (I have seen the recommendations to update only recently changed files and then only sync fully less often, but would like to keep that as a fallback.)
Run the command 'rclone version' and share the full output of the command.
rclone v1.68.1
os/version: Microsoft Windows 10 Home 22H2 (64 bit)
os/kernel: 10.0.19045.5131 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.23.1
go/linking: static
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Hetzner Storage Box
The command you were trying to run (eg rclone copy /tmp remote:tmp)
with rclone sync, if there is an error, rclone re-checks all the file again, default is three times.
~250,000 X 3 = ~700,000
a debug log should show that.
that is the way to go.
i have been using sftp on storagebox for about five years.
great service but sftp is very slow with lots of small files and calculating the checksums takes additional time.
not sure about your setup, but sometimes i find this works well.
every day, run rclone copy --max-age=1d
once a week, run a full rclone sync
Thanks for the advice.
So I added --retries 1 because there were a few filenames that rclone didn't like (is that still report-worthy?). Obviously you knew. I didn't expect rclone to retry the entire sync three times. So lots of time saved.
Very surprisingly, though: rclone copy [] --max-age 1d was no faster than a rclone sync with otherwise identical parameters !
I must assume that it is my NAS that is actually limiting performance? It a quite old QNAP model.
the best way to compare copy --max-age=1d and rclone sync is using --dry-run.
that will list+check only, without taking the time to actually copy the files.
Thanks, @asdffdsa . That's what I did (--dry-run) and how I came to the conclusion that in my case it may be the NAS being a strong contributor to overall performance.
The longer I experiment, though, I feel, the less I know. So I have a related question. Wasn't sure it justifies a new topic. It relates to this copy vs sync with remotes that don't support modtime. I am using Hetzner now and have the option of using webdav (faster, but no modtime support apparently) vs sft (slower):
What are the implications of not having modtime? Appears to me, the problem is files being updated without changing filesize. Appears, rclone sync will not update them in the remote. Anything else?
And then I noticed: rclone check WILL in fact notice the outdated file in the remote. Now I am confused, because the "normal" sync will miss something. Are we supposed to run check routinely?
But then, even if check finds it, a mere copy still won't update the remote (because copy thinks the files are the same based on size). Is there a forceful copy?
So I don't know. I feel I have no idea anymore what the workflow should be
i run rclone on synology devices. have not used a qnap in a few years, pretty sure can run i did try rclone on qnap.
what is the output of uname -a ?
i have been using hetzner storagebox for 5+ years, such a versatile product.
i also use it has a cheap off-site backup repository for veeam backup and replication.
for me the big difference, on storagebox, between sftp and webdav, is file verification using checksums.
sftp supports both modtimes and checksums, that is why i choose to use it.
slow and steady wins the race.