I suspect rclone is not checksuming files on my end

I have a script that backups a few hard drives to Google Drive. By default rclone will use the mod-time and size to determine if a file on src has been modified recently. This was causing some issues and false positives, so I decided to add the --checksum flag to my command to make it use checksuming instead of mod-time.

The problem is that some checks are going by too fast. I don’t think rclone is actually checksuming anything. Also the skipped files just say

Size of src and dst objects identical

It makes no mention of the checksum at all so I’m not confident it’s doing the right thing.

How can I make sure rclone is in fact using a checksum to check my files?

This is on macOS High Sierra.

Can you share the full command you are running? What version of rclone are you running? If you run the command with the debug logs -vvv on the command line, what is the output?

This is my full script:

 #!/bin/bash

DATE=`date +%Y_%m_%d_%H_%M_%S`

NAME="Archive"
LOG_NAME=$NAME\_$DATE.log
rclone sync /Volumes/Archive AibanezKautschMe_ADF_Crypt:/Backups/iMac-Late2012/Backup/$NAME --backup-dir=AibanezKautschMe_ADF_Crypt:/Backups/iMac-Late2012/SafetyNet/$NAME/$DATE --exclude ".DS_Store" --exclude ".DocumentRevisions-V100/**" --exclude ".TemporaryItems/**" --exclude ".Trashes/**" --exclude ".fseventsd/**" --log-level="DEBUG" --log-file=/Users/andyibanez/rclone_logs/$LOG_NAME --checkers 1 --transfers 4 --checksum

rclone 1.43

My -vv/ --rclone-log=DEBUG ouputs things like this:

2018/12/29 11:47:25 DEBUG : Using config file from “/Users/andyibanez/.config/rclone/rclone.conf”
2018/12/29 11:47:36 DEBUG : .DocumentRevisions-V100: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : .DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : .fseventsd: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : .TemporaryItems: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : .Trashes: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : .DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:36 DEBUG : Music/.DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:42 DEBUG : Music/04 Limited Sazabys/.DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:43 DEBUG : Music/11Eyes/.DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/.DS_Store: Excluded from sync (and deletion)
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/01. message.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/01. message.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/02. My HERO.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/02. My HERO.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/03. Brain sugar.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/03. Brain sugar.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/04. Utopia.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/04. Utopia.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/05. Milestone.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/05. Milestone.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/06. Password.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/06. Password.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/07. Alien.flac: Size of src and dst objects identical
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/07. Alien.flac: Unchanged skipping
2018/12/29 11:47:44 DEBUG : Music/04 Limited Sazabys/04 Limited Sazabys - SOIL [FLAC]/08. Kitchen.flac: Size of src and dst objects identical

It’s not practical to put the full log because it’s huge.

Does rclone use the OS’s hashing tools? If so which ones would it use in a Unix environment? I could try installing them.

OK, turns out encrypted remotes don’t support checksumming on sync. It makes total sense when you think about it.

So for now I will just use the filesize checking. When this backup is done I will run cryptcheck on it.

2 Likes

Oh that’s right! That’s a good find. I never usually care as I’m only doing it for Plex so if something gets corrupt or doesn’t work, I just redownload and upload it but I don’t recall seeing that as a problem before.