"source and destination have no hashes in common" for unencrypted Drive to Local sync

Help and Support
1

I'm using rclone sync to maintain a local backup of my Drive account and am seeing this in my logs:

2020/10/03 09:52:28 NOTICE: Local file system at /shared/backups/rclone/GDrive/eric.dalquist: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only

I've found a bunch of posts that state --checksum doesn't work on crypt remotes or team drives. I don't believe either are applicable here. This is my personal account and it is not encrypted. Also the log shows many entries stating Size and MD5 of src and dst objects identical.

rclone version

$ rclone --version
rclone v1.53.0
- os/arch: linux/amd64
- go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

$ uname -a
Linux backups 5.4.60-1-pve #1 SMP PVE 5.4.60-2 (Fri, 04 Sep 2020 10:24:50 +0200) x86_64 x86_64 x86_64 GNU/Linux

Which cloud storage system are you using? (eg Google Drive)

Google Drive, personal account

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ /usr/bin/rclone sync BACKUP-GDrive-eric_dalquist: /shared/backups/rclone/GDrive/eric.dalquist \
  --progress --fast-list --max-backlog=1000000 --checksum --checkers=16 --tpslimit=5 --transfers=10 \
  --bwlimit=65M --timeout=30s --low-level-retries=2 --retries=1 --delete-after --delete-excluded \
  --use-server-modtime --exclude=Takeout/** -vv \
  --log-file=/shared/backups/rclone/logs/2020-10-03_BACKUP-GDrive-eric_dalquist.log

The rclone config contents with secrets removed.

[BACKUP-GDrive-eric_dalquist]
type = drive
client_id = xxx
client_secret = xxx
scope = drive
use_trash = false
export_formats = ods,odt,odp,svg
acknowledge_abuse = true
token = xxx
root_folder_id = xxx

A log from the command with the -vv flag

The log is huge and I'd rather not share the names of all the files I have in my drive. I tried my best to smartly trim all the noise and still have a useful log.

2020/10/03 10:20:12 DEBUG : rclone: Version "v1.53.0" starting with parameters ["/usr/bin/rclone" "sync" "BACKUP-GDrive-eric_dalquist:" "/shared/backups/rclone/GDrive/eric.dalquist" "--progress" "--fast-list" "--max-backlog=1000000" "--ch
ecksum" "--checkers=16" "--tpslimit=5" "--transfers=10" "--bwlimit=65M" "--timeout=30s" "--low-level-retries=2" "--retries=1" "--delete-after" "--delete-excluded" "--use-server-modtime" "--exclude=Takeout/**" "-vv" "--log-file=/shared/bac
kups/rclone/logs/manual_2020-10-03_BACKUP-GDrive-eric_dalquist.log"]
2020/10/03 10:20:12 DEBUG : Using config file from "/home/edalquist/.config/rclone/rclone.conf"
2020/10/03 10:20:12 INFO  : Starting bandwidth limiter at 65MBytes/s
2020/10/03 10:20:12 INFO  : Starting HTTP transaction limiter: max 5 transactions/s with burst 1
2020/10/03 10:20:12 DEBUG : Creating backend with remote "BACKUP-GDrive-eric_dalquist:"
2020/10/03 10:20:12 DEBUG : Creating backend with remote "/shared/backups/rclone/GDrive/eric.dalquist"
2020/10/03 10:20:16 DEBUG : Copy of 2020 Cookie Pre Orders - Template: No export formats found for "application/vnd.google-apps.form"
2020/10/03 10:20:16 DEBUG : Takeout: Excluded from sync (and deletion)
2020/10/03 10:20:16 DEBUG : Sound Speech Solutions: No export formats found for "application/vnd.google-apps.site"
2020/10/03 10:20:16 DEBUG : 2015 Colorado Spring Break Road Trip: Ignoring unknown document type "application/vnd.google-apps.map"
2020/10/03 10:20:16 DEBUG : Summer Trip: Ignoring unknown document type "application/vnd.google-apps.map"
2020/10/03 10:20:16 DEBUG : PK: Ignoring unknown document type "application/vnd.google-apps.map"
2020/10/03 10:20:16 DEBUG : Untitled map: Ignoring unknown document type "application/vnd.google-apps.map"
2020/10/03 10:20:16 DEBUG : Directions to Unknown road: Ignoring unknown document type "application/vnd.google-apps.map"
...
2020/10/03 10:20:47 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (50) returned no entries
2020/10/03 10:20:47 DEBUG : Google drive root '': Recycled 50 entries
2020/10/03 10:20:47 DEBUG : Google drive root '': Recycled 50 entries
2020/10/03 10:21:10 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
2020/10/03 10:21:13 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (2) returned no entries
2020/10/03 10:21:13 DEBUG : Google drive root '': Recycled 2 entries
2020/10/03 10:21:16 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
2020/10/03 10:21:20 NOTICE: GetRooftopSiteEstimatedActuals (1).csv: Duplicate object found in source - ignoring
2020/10/03 10:21:20 NOTICE: GetRooftopSiteForecast (1).csv: Duplicate object found in source - ignoring
...
2020/10/03 10:21:20 NOTICE: resolutions.txt: Duplicate object found in source - ignoring
2020/10/03 10:21:20 DEBUG : 01-Jan-2020_to_20-Jan-2020 (1).csv: MD5 = 485c7d1b1a2fd6a127f41a0525fa21b7 OK
2020/10/03 10:21:20 DEBUG : 01-Jan-2020_to_20-Jan-2020 (1).csv: Size and MD5 of src and dst objects identical
2020/10/03 10:21:20 DEBUG : 01-Jan-2020_to_20-Jan-2020 (1).csv: Unchanged skipping
2020/10/03 10:21:20 DEBUG : 01-Mar-2020_to_06-Apr-2020 (1).csv: MD5 = bcbe50f488714e0ba8ef7e5509a177f1 OK
2020/10/03 10:21:20 DEBUG : 01-Mar-2020_to_06-Apr-2020 (1).csv: Size and MD5 of src and dst objects identical
2020/10/03 10:21:20 DEBUG : 01-Mar-2020_to_06-Apr-2020 (1).csv: Unchanged skipping
2020/10/03 10:21:20 NOTICE: Calendar making/Scanned_20200509-1739.pdf: Duplicate object found in source - ignoring
2020/10/03 10:21:20 NOTICE: Local file system at /shared/backups/rclone/GDrive/eric.dalquist: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2020/10/03 10:21:20 DEBUG : Alarm Zones.ods: Size of src and dst objects identical
2020/10/03 10:21:20 DEBUG : Alarm Zones.ods: Unchanged skipping
2020/10/03 10:21:20 DEBUG : 15-Nov-2019_to_29-Dec-2019 (1).csv: MD5 = f067db38d19966564e5c990d9ecd0674 OK
2020/10/03 10:21:20 DEBUG : 15-Nov-2019_to_29-Dec-2019 (1).csv: Size and MD5 of src and dst objects identical
2020/10/03 10:21:20 DEBUG : 15-Nov-2019_to_29-Dec-2019 (1).csv: Unchanged skipping
2020/10/03 10:21:20 DEBUG : Broadcast Pioneers Hall of Fame: MD5 = 584fb99249e1fed6bd6cd0bd0af43bca OK
2020/10/03 10:21:20 DEBUG : Broadcast Pioneers Hall of Fame: Size and MD5 of src and dst objects identical
2020/10/03 10:21:20 DEBUG : Broadcast Pioneers Hall of Fame: Unchanged skipping
...
2020/10/03 10:23:01 DEBUG : Work/ACS/projects/xml/xsql/mtf/images/total/8.gif: Unchanged skipping
2020/10/03 10:23:01 DEBUG : Local file system at /shared/backups/rclone/GDrive/eric.dalquist: Waiting for transfers to finish
2020/10/03 10:23:01 DEBUG : Waiting for deletions to finish
2020/10/03 10:23:01 INFO  : There was nothing to transfer
2020/10/03 10:23:01 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:             16041 / 16041, 100%
Elapsed time:      2m48.8s

2020/10/03 10:23:01 DEBUG : 4 go routines active
1 Like
2

That error message is very misleading, I need to fix that. What it means is that one of the items doesn't have a hash

What kind of file is that? Is it a Google Doc maybe? They don't have hashes.

3

/shared/backups/rclone/GDrive/eric.dalquist is the dest directory for the rsync clone

If I read the rest of the DEBUG log right rclone is using checksums but something is triggering that error?

This is running in an LXC container and /shared/backups is a bind mount of the host's ZFS file system.

4

Ah, OK!

Yes rclone is using checksums, but rclone is copying a file without a checksum (which is probably a Google Doc).

It shouldn't give that error in that case.

So you can ignore that error completely and I'll work on fixing rclone so it doesn't print it in this case.

1 Like
5

Sounds good, thanks for the help! Let me know if you want a hand debugging, I can try a build with more logging to try and capture the specific file triggering the message.

(also my reply should say rclone sync not rsync clone)

6

I think this should fix it - can you give it a go? Thanks

v1.54.0-beta.4794.84de1d1c0.fix-checksum-message on branch fix-checksum-message (uploaded in 15-30 mins)

1 Like
7

Looks good, just re-ran and I don't see any checksum errors.

8

Great!

I've merged this to master now which means it will be in the latest beta in 15-30 mins and released in v1.53.2

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.