New feature suggestion: Use local file system .md5/.sha1/.svf/etc. checksum files as MD5/SHA1/CRC32/etc. checksum source

Feature
#1

Hello. It would be very useful thing if someone would make that feature for rclone.

As you know, NTFS file system do not support checksums. Some remotes explicitly require that file checksum would be provided before the upload. Those checksum files can be generated with Total Commander, HashCheck Shell Extension and other similar utilities.

So rclone should be able to load any .md5/.sha1/.svf/etc. file in the folder and check if it has checksum for the file which is about to be uploaded. Those checksum files seems to have same structure between each other. Maybe there are some minor differences, like comments or Unix format. That feature could be enabled by using "--enable-local-checksum-files" or similar flag. It should prefer checksum file which completely matches (with or without file extension) filename of the checksumed file. If no checksum for particular file is found or provided checksum is incorrect for some reason, it should fall back to regular file upload behaviour. There should also be declaration for manual checksum file if they span many folders. For e.g.: "--use-local-checksum-file Stuff.md5". The problem is that such multi-level directory checksum files are relative and rclone should figure out for which file which checksum should be applied. But that shouldn't be big problem as that checksum file itself should be in correct relative directory position. Also there should be flag which sets if you want your checksum files to be uploaded together with other files: "--upload-local-checksum-files true/false".

Could I expect that such feature would be implemented in the near future? That would potentially save me quite a lot of time while uploading.

#2

hi,
that would be a useful feature.
rclone is half way there as rclone can create such a list using https://rclone.org/commands/rclone_md5sum/
i use double commander, which can also generate checksums.

fyi -NTFS does not support checksums, but REFS supports checksums for metadata and files.
great file system.

#3

I mainly want to use that for crypted/chunked remote. If all crypted/chunked files are checksummed before upload, I can save a lot of time when uploading them in parallel because upload speed is limited per file on the remote I use. The remote explicitly requires checksums for files before upload so this solution is really necessary. I can't use ReFS as it is not supported (can't create) with Windows 10 Home/Pro. It is only supported with Enterprise/Pro for Workstations editions.

Even better solution for me would be if rclone could generate crypted/chunked remote local cache (precache) with fully precalculated required checksums (better also saved locally to checksum files so it can be resumed later) for all newly generated files in precache. Then upload everything normally in parallel from that new precache. You can partly achieve that functionality with rclone's local file system storage functions. But the key element of saving&reading checksums of the files is missing. There could be "--local-destination-create-checksum-file md5/sha1/sfv" or similar flag to enable checksum file generation directly while transferring files locally to local crypted/chunked destination (precache): "rclone copy c:\stuff localcrypt:stuff --local-destination-create-checksum-file md5/sha1/sfv". Then I would use "rclone copy localcrypt:stuff remote:encstuff --enable-local-checksum-files --transfers 25" to transfer that "precache" to my remote effectively. That is quite convoluted process. Basically I want that rclone could generate (preferably while copying/syncing for better efficiency) and read checksum files. That should let me achieve the things I want.

#4

I'm pretty sure there is an issue about this already...