How to avoid checksumming 'local' syncs?

[rcRsc]

What is the problem you are having with rclone?

Doc says "Normally rclone will look at modification time and size of files to see if they are equal." but doing a simple rclone sync Z:\ F:\ --progress -v leads to massive traffic as checksums are done.

I haven't found any switch to do so, so how to avoid/disable checksumming here.

P.S.
Q: "Why in the world would you want to?!"
A: Neither F:\ nor Z:\ are local drives, they're cloud storage mounts. I simply cannot download 1TB data each time I try to sync and using hasher was at best a workaround.

Run the command 'rclone version' and share the full output of the command.

rclone v1.68.0

• os/version: Microsoft Windows 10 Enterprise LTSC 2021 21H2 (64 bit)
• os/kernel: 10.0.19044.3324 (x86_64)
• os/type: windows
• os/arch: amd64
• go/version: go1.23.1
• go/linking: static
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

• Filen, mounted as drive F:\
• pCloud, mounted as drive Z:\

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync Z:\ F:\ --progress -v

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

'local' drives require no config, so no config at all

Thank you.

[kapitainsky]

I think it is post copy/sync check you can disable by using:

--ignore-checksum   Skip post copy check of checksums

[rcRsc]

Thank you, but no, that's not the needed trick as it's 8 files at once (AFAIK the default for checker), no matter if I set --transfers to 1or 16.

Also you see in the screenshot "Transferred: 2/2" but it's "checking" 8.

[asdffdsa (jojothehumanmonkey)]

@rcRsc

8 files at once (AFAIK the default for checker)

correct, can change that with --checkers

[rcRsc]

Wouldn't help me as I need zero checkers, bc. checkers download the whole file(s, ~1 TB in my case - for every.single.sync(!)) .

BTW: --checkers 0 still runs a single (1) checker @ncw

[asdffdsa (jojothehumanmonkey)]

just curious, why run three instances of rclone, instead of one?

rclone sync pcloud: filen: --progress -v

[rcRsc]

AFAIK I run a single instance for the sync job, Taskmanager also only shows a single one.
How do you get the idea I ran 3?

If it's about the mounts: Z:\ is not an rclone mount (F:\ is indeed an rclone* mount**, no other way to access Filen w/o using their [censored] client).
But the checkers are ran by my basic command, not by the rclone mount.

edit: %AppData%\Roaming\filen-virtual-drive\rclone.conf:

[Filen]
type = webdav
url = http://127.0.0.1:1025
vendor = other
user = [snipped]
pass = [snipped]

* apparently (and unfortunately) they did not contribute to rclone but fork it
** command line looks like
%AppData%\Roaming\filen-virtual-drive\filen_rclone_win32_x64_1670.exe mount Filen: F: --config "%AppData%\Roaming\filen-virtual-drive\rclone.conf" --vfs-cache-mode full --cache-dir "C:\FilenCache\filenCache" --vfs-cache-max-size 30Gi --vfs-cache-min-free-space 5Gi --vfs-cache-max-age 720h --vfs-cache-poll-interval 1m --dir-cache-time 1m --cache-info-age 1m --noappledouble --noapplexattr --no-gzip-encoding --use-mmap --disable-http2 --file-perms 0666 --dir-perms 0777 --use-server-modtime --vfs-read-chunk-size 128Mi --buffer-size 0 --vfs-read-ahead 1024Mi --vfs-read-chunk-size-limit 0 --no-checksum --vfs-fast-fingerprint --rc --rc-addr 127.0.0.1:1025 --log-file "%AppData%\Roaming\@filen\logs\rclone.log" --devname Filen --volname \\Filen\Filen -o FileSecurity="D:P(A;;FA;;;WD)" --network-mode --exclude ".DS_Store" --exclude "._*" --exclude ".Trashes/**" --exclude ".Spotlight-V100/**" --exclude ".TemporaryItems/**" --exclude "*.tmp" --exclude "~*" --exclude "Thumbs.db" --exclude "desktop.ini" --exclude "$RECYCLE.BIN/**" --exclude "System Volume Information/**" --exclude "Temp/**" --exclude "AppData/Local/Temp/**" --exclude ".Trash*" --exclude "*.swp" --exclude "*.temp" --exclude ".*.swx" --exclude "/tmp/**" --exclude "/var/tmp/**" --exclude "**/.cache/**" --exclude "**/Cache/**" --exclude "**/.npm/_cacache/**"

[asdffdsa (jojothehumanmonkey)]

@rcRsc

apparently (and unfortunately) they did not contribute to rclone but fork it

that is an alpha version of a custom compiled version of rclone using an unsupported and untested backend.
imo, might post at their forum, contact their tech support.

The package is still a work in progress. DO NOT USE IT IN PRODUCTION YET. Class names, function names, types, definitions, constants etc. are subject to change until we release a fully tested and stable version.
the seems valid as the lastest source code change was just 13 hours ago.

[Animosity022]

@rcRsc

Wouldn't help me as I need zero checkers, bc. checkers download the whole file(s, ~1 TB in my case - for every.single.sync(!)) .

BTW: --checkers 0 still runs a single (1) checker @ncw

Checkers are what 'checks' a file when you copy something.

If you ran a copy and had 0, rclone would not work which is why you can't set it to 0.

Checkers also do the checksums for file validation.

[rcRsc]

I'm sorry but the unwanted checkers are run by official rclone v. 1.68.0, the screen shot is the output of rclone(v.1.68.0) sync Z:\ F:\ ...

Dunno why you try to redirect me to a different vendor when it's pretty obviously an issue with official rclone.

[rcRsc]

Fine, but how do I achieve the documented "will look at modification time and size of files to see if they are equal" vs. the (undocumented) 8 checkers downloading every.single.file I experience?

[Animosity022]

@kapitainsky

--ignore-checksum

^^^^

I think there is some confusion.

If you run a copy, if the date and sizes are identical, no copy happens. No checksum happens.

If you run a copy and data/time are mismatched, you get a copy, you get a checksum unless cancel it like @kapitainsky told you a few posts up.

Checksum

texter@Earls-Mac-mini t % mkfile 1g testfile.txt
texter@Earls-Mac-mini t % rclone copy testfile.txt test2.txt -vv
2024/09/10 09:55:09 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "testfile.txt" "test2.txt" "-vv"]
2024/09/10 09:55:09 DEBUG : Creating backend with remote "testfile.txt"
2024/09/10 09:55:09 DEBUG : Using config file from "/Users/texter/.config/rclone/rclone.conf"
2024/09/10 09:55:09 DEBUG : fs cache: adding new entry for parent of "testfile.txt", "/Users/texter/Downloads/t"
2024/09/10 09:55:09 DEBUG : Creating backend with remote "test2.txt"
2024/09/10 09:55:09 DEBUG : fs cache: renaming cache item "test2.txt" to be canonical "/Users/texter/Downloads/t/test2.txt"
2024/09/10 09:55:09 DEBUG : testfile.txt: Need to transfer - File not found at Destination
2024/09/10 09:55:09 DEBUG : /Users/texter/Downloads/t/test2.txt/testfile.txt.91b00ea0.partial: isCloned: true, error: <nil>
2024/09/10 09:55:09 DEBUG : testfile.txt.91b00ea0.partial: server-side cloned!
2024/09/10 09:55:11 DEBUG : testfile.txt: md5 = cd573cfaace07e7949bc0c46028904ff OK
2024/09/10 09:55:11 DEBUG : testfile.txt.91b00ea0.partial: renamed to: testfile.txt
2024/09/10 09:55:11 INFO  : testfile.txt: Copied (server-side copy)
2024/09/10 09:55:11 INFO  :
Transferred:   	        1 GiB / 1 GiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 1 GiB
Elapsed time:         2.1s

No checksum

2024/09/10 09:55:11 DEBUG : 6 go routines active
texter@Earls-Mac-mini t % rclone copy testfile.txt test3.txt -vv --ignore-checksum
2024/09/10 09:57:48 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "testfile.txt" "test3.txt" "-vv" "--ignore-checksum"]
2024/09/10 09:57:48 DEBUG : Creating backend with remote "testfile.txt"
2024/09/10 09:57:48 DEBUG : Using config file from "/Users/texter/.config/rclone/rclone.conf"
2024/09/10 09:57:48 DEBUG : fs cache: adding new entry for parent of "testfile.txt", "/Users/texter/Downloads/t"
2024/09/10 09:57:48 DEBUG : Creating backend with remote "test3.txt"
2024/09/10 09:57:48 DEBUG : fs cache: renaming cache item "test3.txt" to be canonical "/Users/texter/Downloads/t/test3.txt"
2024/09/10 09:57:48 DEBUG : testfile.txt: Need to transfer - File not found at Destination
2024/09/10 09:57:48 DEBUG : /Users/texter/Downloads/t/test3.txt/testfile.txt.91b00ea0.partial: isCloned: true, error: <nil>
2024/09/10 09:57:48 DEBUG : testfile.txt.91b00ea0.partial: server-side cloned!
2024/09/10 09:57:48 DEBUG : testfile.txt.91b00ea0.partial: renamed to: testfile.txt
2024/09/10 09:57:48 INFO  : testfile.txt: Copied (server-side copy)
2024/09/10 09:57:48 INFO  :
Transferred:   	        1 GiB / 1 GiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Server Side Copies:     1 @ 1 GiB
Elapsed time:         0.0s

2024/09/10 09:57:48 DEBUG : 6 go routines active

You save yourself a lot of time and share a debug log as that's the exact reason it's asked for as it shows exactly what you ran and the output and folks can help you explain the output and offer help.

[rcRsc]

@Animosity022

You save yourself a lot of time and share a debug log

...tried to avoid it for privacy (file/folder names) but found a privacy workaround, so here you go (97kB in 6s):
https://pastebin.com/raw/dakfYRzp

[Animosity022]

In that log, you still have not added --ignore-checksum so you'll get checksums.

command

2024/09/10 16:05:01 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "sync" "Z:\\" "F:\\" "--progress" "-vv" "--dry-run" "--log-file" "C:\\Logs\\rclone.log"]

2024/09/10 16:05:01 DEBUG : CT/eub71bj1mnvsq5d5ii8og69bks/b32845ub5kcgk5ba09ovuj1no8: md5 = 638cbbefe9541abfa0a0037bc52ef431 OK

[rcRsc]

Nope:

https://pastebin.com/raw/aBQG6nXW

[asdffdsa (jojothehumanmonkey)]

@rcRsc

it's pretty obviously an issue with official rclone.

re251sg67nnhm135oad319cpmk/058srvaumm56v2l6q44gc5m8mqujceocescn9putobrv1udqhke0: Modification times differ by 1h59m59.0498134s: 2023-07-05 08:24:09.9501866 +0200 CEST, 2023-07-05 10:24:09 +0200 CEST
re251sg67nnhm135oad319cpmk/058srvaumm56v2l6q44gc5m8mqujceocescn9putobrv1udqhke0: md5 = fc9a1bbf90f7172b148fc8501549c2d7 OK
re251sg67nnhm135oad319cpmk/058srvaumm56v2l6q44gc5m8mqujceocescn9putobrv1udqhke0: Skipped update modification time as --dry-run is set (size 1.271Ki)
re251sg67nnhm135oad319cpmk/058srvaumm56v2l6q44gc5m8mqujceocescn9putobrv1udqhke0: Unchanged skipping

ok, i think this is the issue?

Local Filesystem Unexpected MD5 Calculation

if the sizes are equal and the modtimes are not equal, then what is rclone to do? the flags determine rclone's behavior. i could be wrong, not the first time. let's wait and see what others comment

Local Filesystem Unexpected MD5 Calculation

The thinking behind this logic is this We've detected the file is the same size but has a different modtime. Let's try to avoid an expensive copy by comparing checksums first and if the checksums are the same, just update the modtime. I take your point that on a local -> local copy this is perhaps counterproductive as making checksums for the source and the dest is probably about as expensive as copying the file. This algorithm is explained several times in the rclone docs, but not where an…

might test --size-only
and read about --modify-window

not sure, if and how using --use-server-modtime affects that.
i would pick a single affected file and do some more testing.

[rcRsc]

Thank you!

@asdffdsa

not sure, if and how using --use-server-modtime affects that.

unfortunately still the same

When one-way syncing and "file is the same size but has a different modtime" then the logic should be clear: No matter what, copy source to dest. - thats why I do one-way-sync (I want source and dest to be identical).

[asdffdsa (jojothehumanmonkey)]

@rcRsc

unfortunately still the same

did you test --size-only --dry-run

[rcRsc]

No, I didn't as --size-only is unreliable in my environment (e.g. Acronis TrueImage edits tibx files during "incremental backup" to different content but identical size, same for Veracypt containers).

--dry-run is what i used all of this day (see debug logs), still the same.

[Animosity022]

@rcRsc

Nope:

Huh? It's your output:

No flag

How about we take one file:

• do a copy without the flag with -vv
• do a copy with the flag with -vv

Share the output and we can compare the logs from one file just like I did.