--ignore-checksum but still high CPU usage and reading from destination

[mgutt (Marc Gutt)]

What is the problem you are having with rclone?

rlcone is running on destination server. It fetches through SFTP from the source server. I copied all files a few days ago. Now I want to start a final run and transfer only new files.

What is your rclone version (output from rclone version)

1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Unraid

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --dry-run --ignore-checksum copy synosftp:/marc/ /mnt/user/Marc/

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/04/28 09:20:43 NOTICE: #recycle/Manictime Screenshots/2020-04-18/2020-04-18_23-51-48_02-00_1294_1407_978075_1.jpg: Not copying as --dry-run
2020/04/28 09:20:43 NOTICE: #recycle/Manictime Screenshots/2020-04-18/2020-04-18_23-51-49_02-00_1294_1407_978076_1.jpg: Not copying as --dry-run
2020/04/28 09:20:43 NOTICE: #recycle/Manictime Screenshots/2020-04-18/2020-04-18_23-52-16_02-00_1294_1407_978078_1.jpg: Not copying as --dry-run
2020/04/28 09:20:44 NOTICE: #recycle/Manictime Screenshots/2020-04-18/2020-04-18_23-52-27_02-00_1294_1407_978081_1.jpg: Not copying as --dry-run
2020/04/28 09:20:44 NOTICE: #recycle/Manictime Screenshots/2020-04-18/ugrlyv2a.jfq.ignore: Not copying as --dry-run

Until here it was really fast. Then after several seconds this appeared:

2020/04/28 09:21:28 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 4181 / 14197, 29%
Transferred: 5847 / 5847, 100%
Elapsed time: 59.6s
Checking:

• Filme/Film1.mkv: checking
• Filme/Film2.mkv: checking
• Filme/Film3.mkv: checking
• Filme/Film4.mkv: checking
• Filme/Film5.mkv: checking
• Filme/Film6.mkv: checking
• Filme/Film7.mkv: checking
• Smartphone/OfficeLens/…d_2018-11-14_10-10.zip: checking

What does "checking" mean? Shouldn't it only check the time and size of the files?

The disk status shows massive read load on the corresponding disk of the destination server:

[Animosity022]

Can you post the debug log of when the issue actually happened with the full log? You can try with that one file if you that's easier.

Checking is when it compared source to destination. What command did you run when you got that output of checking?

[mgutt (Marc Gutt)]

The command is as posted above:

rclone -vv --dry-run --ignore-checksum copy synosftp:/marc/ /mnt/user/Marc/

[Animosity022]

Can you share the full log?

[mgutt (Marc Gutt)]

I used a smaller subfolder and its still running, but it starts with some interesting things:

rclone -vv --dry-run --ignore-checksum copy synosftp:/marc/Filme_Revise /mnt/user/Marc/Filme_Revise
2020/04/28 15:13:39 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "-vv" "--dry-run" "--ignore-checksum" "copy" "synosftp:/marc/Filme_Revise" "/mnt/user/Marc/Filme_Revise"]
2020/04/28 15:13:39 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/04/28 15:13:39 DEBUG : sftp://user@1.2.3.4:22//marc/Filme_Revise: New connection 1.2.3.4:45500->1.2.3.4:22 to "SSH-2.0-OpenSSH_7.4"
2020/04/28 15:13:40 DEBUG : datei.txt: Size and modification time the same (differ by 0s, within tolerance 1s)
2020/04/28 15:13:40 DEBUG : datei.txt: Unchanged skipping
2020/04/28 15:13:40 DEBUG : Film (2009)/Film (2009).mkv: Modification times differ by 3515h6m32.548441729s: 2019-11-29 10:04:35 -0800 PST, 2020-04-23 22:11:07.548441729 -0700 PDT
2020/04/28 15:13:40 DEBUG : Film (2009)/Film (2009).mkv: Modification times differ by 3515h4m52.940288368s: 2019-11-29 10:03:47 -0800 PST, 2020-04-23 22:08:39.940288368 -0700 PDT
2020/04/28 15:13:40 DEBUG : sftp cmd = /marc/Filme_Revise/Film\ \(2009\)/Film\ \(2009\).mkv
2020/04/28 15:13:40 DEBUG : Film (2009)/Film (2009).mkv: Failed to calculate MD5 hash: Process exited with status 1 (Could not chdir to home directory /var/services/homes/user: No such file or directory
md5sum: /marc/Film_Revise/Film (2009)/Film (2009).mkv: No such file or directory)

1.) Why does it use md5sum although I used –ignore-checksum?
2.) Why does it find /marc/Film_Revise/Film (2009)/Film (2009).mkv while checking the mod-time but not while trying to create the md5sum?
3.) Why does it read the file from the local disk (it seems to check the local checksum) although there is no md5 checksum from the remote to compare with?
4.) Why does /var/services/homes/user not exist? EDIT: Ok, this is related to Synology (source server). I was able to solve this error message through this hint, but it has no influence to the other errors.

EDIT: I tried this command:

rclone -vv md5sum synosftp:"/marc/Filme_Revise/Film (2009)/Film (2009).mkv"`

Same error:

2020/04/28 15:41:56 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "-vv" "md5sum" "synosftp:/marc/Filme_Revise/Film (2009)/Film (2009).mkv"]
2020/04/28 15:41:56 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/04/28 15:41:56 DEBUG : sftp://user@1.2.3.4:22//marc/Filme_Revise/Film (2009)/Film (2009).mkv: New connection 1.2.3.4:45558->1.2.3.4:22 to "SSH-2.0-OpenSSH_7.4"
2020/04/28 15:41:57 DEBUG : Film (2009) 1.mkv: Excluded
2020/04/28 15:41:57 DEBUG : Film (2009) 2.mkv: Excluded
2020/04/28 15:41:57 DEBUG : Film (2009) 3.mkv: Excluded
2020/04/28 15:41:57 DEBUG : Film (2009) 4.mkv: Excluded
2020/04/28 15:41:57 DEBUG : sftp cmd = /marc/Filme_Revise/Film\ \(2009\)/Film.mkv
2020/04/28 15:41:57 DEBUG : Film (2009).mkv: Failed to calculate MD5 hash: Process exited with status 1 (md5sum: /marc/Filme_Revise/Film (2009)/Film (2009).mkv: No such file or directory)
                                  Film (2009).mkv
2020/04/28 15:41:57 DEBUG : 12 go routines active
2020/04/28 15:41:57 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "-vv" "md5sum" "synosftp:/marc/Filme_Revise/Film (2009)/Film (2009).mkv"]

If I type in a wrong file name it says "directory not found"?!

rclone -vv md5sum synosftp:"marc/Filme_Revise/Film (2009)/does not exist.mkv"
2020/04/28 15:51:32 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "-vv" "md5sum" "synosftp:marc/FilmeMakeMKV_Revise/Verblendung (2009)/does not exist.mkv"]
2020/04/28 15:51:32 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/04/28 15:51:33 DEBUG : sftp://user@1.2.3.4:22/marc/Filme_Revise/Film (2009)/does not exist.mkv: New connection 1.2.3.4:45568->1.2.3.4:22 to "SSH-2.0-OpenSSH_7.4"
2020/04/28 15:51:33 ERROR : : error listing: directory not found
2020/04/28 15:51:33 Failed to md5sum with 2 errors: last error was: directory not found

EDIT: Ok, if I understand this explanation correct the --ignore-checksum setting only influences the check after the transfer is completed and not the checksum check for files that already exist on the destination. This can not be disabled. Sad.

But finally it does not explain why md5sum on the source is not able to find the file.

[Animosity022]

Is it a ftp remote? The remote needs to support md5sum, which ftp does not.

[mgutt (Marc Gutt)]

The remote is SFTP, so it supports md5sum, but I found some more information especially related to Synology here:

https://rclone.org/sftp/#sftp-path-override

This allows checksum calculation when SFTP and SSH paths are different. This issue affects among others Synology NAS boxes.

I'll play around with this setting to find out which one will solve the md5sum not found error.

But the more interesting thing is the --sftp-disable-hashcheck setting:
https://rclone.org/sftp/#sftp-disable-hashcheck

Disable the execution of SSH commands to determine if remote file hashing is available

These special SFTP settings are not part of the flags page so I didn't found them directly:
https://rclone.org/flags/

I will use --sftp-disable-hashcheck to solve my problem, but there are still open questions:

1.) Why does rclone check the local md5sum although the remote md5sum check failed (it will create a md5sum without being able to compare it to something)? Shouldn't rclone quit with a "fatal error" in such a situation?
2.) Why isn't there a global disable checksum setting?

[Animosity022]

You'd have to share your config. I setup a SFTP test and works without much of a problem.

[SFTP]
type = sftp
host = localhost
user = felix
port = 4022
pass = something
md5sum_command = md5sum
sha1sum_command = sha1sum

and a test.

felix@gemini:~$ rclone md5sum SFTP:plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb -vv
2020/04/28 19:32:41 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "md5sum" "SFTP:plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb" "-vv"]
2020/04/28 19:32:41 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/04/28 19:32:42 DEBUG : sftp://felix@localhost:4022/plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb: New connection 127.0.0.1:36296->127.0.0.1:4022 to "SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2"
<removed a bunch of excluded files>
2020/04/28 19:32:42 DEBUG : sftp://felix@localhost:4022/plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb: checking "md5sum" command: "d41d8cd98f00b204e9800998ecf8427e  -"
2020/04/28 19:32:42 DEBUG : sftp://felix@localhost:4022/plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb: checking "sha1sum" command: "da39a3ee5e6b4b0d3255bfef95601890afd80709  -"
2020/04/28 19:32:42 DEBUG : Saving config "md5sum_command" = "md5sum" in section "SFTP" of the config file
2020/04/28 19:32:42 DEBUG : Saving config "sha1sum_command" = "sha1sum" in section "SFTP" of the config file
2020/04/28 19:32:42 DEBUG : sftp cmd = plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb
2020/04/28 19:32:42 DEBUG : sftp output = "607e4710fafd587aa95779047186ad11  plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb\n"
2020/04/28 19:32:42 DEBUG : sftp hash = "607e4710fafd587aa95779047186ad11"
607e4710fafd587aa95779047186ad11  plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb
2020/04/28 19:32:42 DEBUG : 12 go routines active
2020/04/28 19:32:42 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "md5sum" "SFTP:plexmediaserver_1.18.9.2571-e106a8a91_amd64.deb" "-vv"]'

When I ran it, it added the md5sum to my config file. I'd assume md5sum is present and accessible on the Synology.

[Animosity022]

@mgutt

1.) Why does rclone check the local md5sum although the remote md5sum check failed (it will create a md5sum without being able to compare it to something)? Shouldn't rclone quit with a "fatal error" in such a situation?

Apologies if I missed it somewhere in the logs, but it shouldn't if the remote doesn't support a md5sum. My example is I use an encrypted remoted, which doesn't support them so it does not md5sum the local file before sending it. It should only calculate the md5sum if both sides support it to my understanding.

@mgutt

2.) Why isn't there a global disable checksum setting?

You can use --size-only if you wanted to just compare sizes and depending the remote, there might be other options.

[ncw (Nick Craig-Wood)]

@mgutt

1.) Why does rclone check the local md5sum although the remote md5sum check failed (it will create a md5sum without being able to compare it to something)? Shouldn't rclone quit with a "fatal error" in such a situation?

It starts the remote and local checksums at the same time in parallel so if the remote checksum fails it is still committed to calculate the local checksum.

Rclone is doing a checksum check in your case because the files are the same size but the modification times are different so rclone wants to make sure they are the same or not before transmitting them.

@mgutt

2.) Why isn't there a global disable checksum setting?

There are lots of checksum disables rclone help flags sum to see them!

However there isn't a way of disabling this particular checksum. This was also discussed in this thread

Error while syncing with Azure blob storage [X-Ms-Error-Code: [InvalidBlobOrBlock]]

It is saving you uploading a 1TB file unnecessarily so it isn't all bad! I think using the --no-check-dest flag will work and upload everything regardless! That might not be what you want though. I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.52 Let me know if it goes wrong again!

It might be that --ignore-checksum should skip this, or there should be a new flag...

[mgutt (Marc Gutt)]

Thank you for the detailed explanation.

The transfer is already finished with --ignore-checksum. As I hoped, it updated the wrong timestamps. As checksum was disabled, it should transfered these files again. But this is really ok, as it saved many steps:

• did not calculate checksums on both sides (much less CPU load)
• did not produce read and writes on the HDDs at the same time

I'm really lucky that I accidentally used SFTP and not an SMB mount or similar where it wouldn't be possible to disable checksums.

But still a final question: Does rclone really check the checksum of all destination files after it transfered them? Because it was really fast in transfering the files as long the destination was empty and I do not remind heavy reads on the HDDs while rclone was running.

@Animosity022
Thank you for the hint, but --size-only wouldn't have solved my issue as my first transfer through FTP created many files with the server timestamp and not the file's mod-time. By checking only the size those timestamps would be still wrong.

[ncw (Nick Craig-Wood)]

@mgutt

But still a final question: Does rclone really check the checksum of all destination files after it transfered them? Because it was really fast in transfering the files as long the destination was empty and I do not remind heavy reads on the HDDs while rclone was running.

If it can, yes. You disabled checksums for you SFTP remote so it won't there.