I'm using rclone with Rclone Browser v1.8.0 (kapitainsky releases), but anyway seems that it isn't a rclone command.
I would like to copy and perform a checksum with each copied file. Maybe at the end of each transfer or maybe when the task finish.
Not sure if it's possible put a command into the Rclone Browser's preferences because i never could test it with rclone.exe through command line.
I know that I can use sync, but i accidentally lost some files and i feel more confortable using copy command.
Right now what's I'm doing is create a task inside Rclone Browser. For example copy a folder from my local drive to Google Drive, wait until folder is complete and perform again the same task. That's the only way that I've found to perform a check of the files of that folder.
The check is performed so quick and I'm not sure which kind of check is doing, and if it's the case of checksum, which kind? (CRC32, MD5,SHA1)
For 300 GB of checks it took a little more than 1 hour.
Is there any way to log the checksum of each file?
each time rclone has finished copying a file, it compares checksums.
each storage system has it own kind of checksum. https://rclone.org/overview/
"Google Drive MD5"
You can certainly force checksum comparisons by using --checksum, but if your goal is simply to be sure that the file arrived safely then rclone checksums this automatically. It will not not allow a half-finished or corrupted file to be stored - nor remove a source file (in case of a MOVE or SYNC command) before it has verified that the file was delivered successfully.
The point of --checksum is not to improve transfer-safety, but to improve the accuracy of detecting for example that a file has changed (or to be able to use --track-renames to map data that needs to be moved). rclone normally determines if it needs to make a change to a file based on a combination of it's size and modtime (ie. if both are unchanged it assumes the file is unchanged). Theoretically you could have a file change but get exact the same size and modtime - but this would just be extremely unlikely unless you tampered with the metadata to make it so... So in short, this is very rarely actually needed.
It also has the cost of needing to hash every local file (read all the data) of all files it compares, regardless of if it actually ends up needing to transfer them or not. This is slow, inefficient and potentially puts a lot of extra wear on the harddrives.
I will be happy to answer followup questions, but TLDR it sounds like you are fretting about data-integrity issues that rclone already covers for you
For the ultra-paranoid, you can go download "hashtab" (if using windows) and manually make your own hash-file and then compare the source to the destination - but assuming you let rclone finish without errors you will find that they hash-match
The check will by default made by cheking size+modtime match (unless you use --checksum as I said)
But let me reiterate that when the transfer was initially made it did get hashed on both ends to ensure a successful transfer. If you try to re-transfer it will just do this much faster check to see if there is any reason to re-upload it or not.
When hash-checks are made in rclone it uses whatever hash-type the cloud-server uses - since it needs to math that to be useful.
Gdrive uses MD5. You can see a list of what types are used by other providers here: https://rclone.org/overview/#features
(rclone supports all of them)
if you really really needed to see this info, it should be displayed using the debug-level log: --log-level DEBUG
or simply: -vv
Before to write the post, I've already tested some things:
1_ "--log-level INFO --log-file=mylogfile.txt" and ""--log-level DEBUG --log-file=mylogfile.txt"" trhough nnsm.exe. But the logs weren't easy to read, so I've tried the following
2_ Adding "--checksum" in the Rclone Browser's preferences, field "Default upload option" without sucess because my goal always was that the checks appears showed inside Rclone Browser GUI.
Like this image:
asdffdsa, I've already read that "copy" perform a checksum, but I wasn't sure at all. Thanks for the confirmation. My only concern was related to the Rclone Browser GUI. because if it happens, why it doesn't appear in the "Checks:" section. That was my doubt and the reason why I've written this post.
I will read about "--backup-dir" command. Thanks!
thestigma
I agreed with every that you answered, and you are on right, I'm ultra-paranoid,
Thanks for recommening me "hashtab" software.
The thing is that my hard drives are dying so I have nothing to loose, but if copy command also perform a MD5 checksum for Google Drive, then is not needed to perform a doublecheck.
And yeah, I will put more effort to read the logs with the command --log-level DEBUG
I think that the commands that i've used for nssm.exe are totally independent that the commands that I've used for Rclone Browser, right?
nssm.exe is only for create a Windows service which mount the drive avoiding to setup rclone manually each that the system starts and anyway I can use Rclone Browser without mounting the service, so I'm pretty sure that I need continue adding commands in the Rclone Browser's preferences.
There is a LOT of stuff that happens in the background that rclone does not show you by default (because as you said - the output becomes so messy that is very tedious and confusing to read).
But yes - all rclone transfer types: move, copy, sync all calculates a hashsum on the read-in of the file while it is sending - and then when the transfer is done it checks it's own calculated hashsum to the hashsum the server provides. If they are not identical then it will discard that attempt and re-try.
It just doesn't bother to show you all this background stuff unless you use the extremely verbose debug-log (-vv). I am not sure it it maybe also shows file-hases for transfers using just -v / --verbose. This is not quite as bad to read as debug-level.
First of all I don't recognize the software you are showing screenshots of...
When you said "rclone browser" I assumed you mean the fairly recently added rclone webGUI - but this looks very different.
Are you using that very outdated third-party rclone browser that stopped being developed a long time ago? You should probably link what you are using so we are on the same page before I attempt to answer your questions about this...
Try just using --verbose or -v. This still gives a ton of information, but it still far easier to read. I am not sure if it shows the hash values spesiifically, but give it a try. Debug-log is really only meant for developers to debug the code. It is far too messy to be practical in regular use.
Correct. But also if your drives are not reliable it is an even worse idea to force --checksum because it would stress them unnecessarily with lots of reads.
Definitely look into using --backup-dir like asdffdsa suggests - because when you have unreliable drives you can never be sure that they don't read your files wrong and you then end up coping this mistake to your backup. If you use --backup-dir then you will keep a copy of any "deleted" files, including those that rclone would overwrite or delete in a copy or sync
Gdrive also has a 30-day trashcan for all deleted files that you could recover from in case of emergency, but backup-dir can be your primary safety mechanism without any automatic emptying - and also in the case of a disaster it is much simpler to restore from than picking through the Gdrive trash which can be very tedious...
After you have read the documentation https://rclone.org/docs/#backup-dir-dir
You can check out here a script than asdffdsa and me created for another user that shows how --backup dir can be used to create dated versioning of all changes that are made.
Then I suspect that this program communicates independently from yor mount (which would make the most sense). In that case it will need to use it's own flag options yes... but please find out what exactly this software is. Either it may be super outdated (I know there existed an rclone browser before) or it is something new that I am now aware of and might be interested in learning more about
So someone revived the project then? I think I actually read about this a while back, but I forgot about it. Thanks for letting me know / reminding me.
I will have to remember to check this out again now and see if it works well. It might be a good convenience-tool now that it is being maintained.
