What is the problem you are having with rclone?
Hasher with GDrive backend. Hasher setup to have MD5, SHA1, SHA256 saved in backend.
In GDrive, there are some old files where GDrive doesn't provide SHA1 / SHA256.
"Note, however, that a small fraction of files uploaded may not have SHA1 or SHA256 hashes especially if they were uploaded before 2018."
Eventhough the SHA1 & SHA256 hashsum are saved in the backend, it doesn't show when running commands.
Expected behaviour, when GDrive does not return SHA1 and SHA256 for old files, Hasher will return the hashsum from the backend.
Run the command 'rclone version' and share the full output of the command.
rclone v1.65.2
- os/version: ubuntu 23.10 (64 bit)
- os/kernel: 6.5.0-1015-oracle (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.6
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Hasher with GDrive as backend
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
$ rclone lsjson --hash 'gdhash:/Google Photos/2013/_2.jpeg'
[
{"Path":"_2.jpeg","Name":"_2.jpeg","Size":165206,"MimeType":"image/jpeg","ModTime":"2014-07-07T00:06:00.000Z","IsDir":false,"Hashes":{"md5":"03c6c126c60cc85620e77757ea538550","whirlpool":"1990038b3b8c61ff8acd4c7fa1073562dc7fd03760e0040b5c7f94a2f6b2da1730e542b1a2e3d7bcb6288efa0c898853f40083d8c90163334b5d62e6580d8290"},"ID":"1_wNymqIvOgZL2UT-dqdsLvubqmMwsvtROg"}
]
The rclone config contents with secrets removed.
[gdrive]
type = drive
client_id = <<client_id_removed>>
client_secret = <<client_secret_removed>>
scope = drive
root_folder_id = <<root_folder_id_removed>>
token = <<token_removed>>
team_drive =
[gdhash]
type = hasher
remote = gdrive:
hashes = md5,sha1,sha256,whirlpool
max_age = off
A log from the command with the -vv
flag
$ rclone lsjson --hash 'gdhash:/Google Photos/2013/_2.jpeg' -vv
2024/02/20 04:37:41 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "lsjson" "--hash" "gdhash:/Google Photos/2013/_2.jpeg" "-vv"]
2024/02/20 04:37:41 DEBUG : Creating backend with remote "gdhash:/Google Photos/2013/_2.jpeg"
2024/02/20 04:37:41 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2024/02/20 04:37:41 INFO : Hasher is EXPERIMENTAL!
2024/02/20 04:37:41 DEBUG : Creating backend with remote "gdrive:/Google Photos/2013/_2.jpeg"
2024/02/20 04:37:44 DEBUG : fs cache: adding new entry for parent of "gdrive:/Google Photos/2013/_2.jpeg", "gdrive:Google Photos/2013"
2024/02/20 04:37:44 DEBUG : hasher::gdhash:/Google Photos/2013: Groups by usage: cached [md5, sha1, whirlpool, sha256], passed [md5, sha1, sha256], auto [md5, sha1, whirlpool, sha256], slow [], supported [md5, sha1, whirlpool, sha256]
2024/02/20 04:37:44 DEBUG : gdrive~hasher.bolt: Opened for reading in 85.8µs
2024/02/20 04:37:44 DEBUG : fs cache: adding new entry for parent of "gdhash:/Google Photos/2013/_2.jpeg", "gdhash:/Google Photos/2013"
[
2024/02/20 04:37:44 DEBUG : gdrive~hasher.bolt: released
2024/02/20 04:37:45 DEBUG : _2.jpeg: pass md5
2024/02/20 04:37:45 DEBUG : _2.jpeg: pass sha1
2024/02/20 04:37:45 DEBUG : gdrive~hasher.bolt: Opened for reading in 81.32µs
2024/02/20 04:37:45 DEBUG : _2.jpeg: cached whirlpool = "1990038b3b8c61ff8acd4c7fa1073562dc7fd03760e0040b5c7f94a2f6b2da1730e542b1a2e3d7bcb6288efa0c898853f40083d8c90163334b5d62e6580d8290"
2024/02/20 04:37:45 DEBUG : _2.jpeg: pass sha256
{"Path":"_2.jpeg","Name":"_2.jpeg","Size":165206,"MimeType":"image/jpeg","ModTime":"2014-07-07T00:06:00.000Z","IsDir":false,"Hashes":{"md5":"03c6c126c60cc85620e77757ea538550","whirlpool":"1990038b3b8c61ff8acd4c7fa1073562dc7fd03760e0040b5c7f94a2f6b2da1730e542b1a2e3d7bcb6288efa0c898853f40083d8c90163334b5d62e6580d8290"},"ID":"1_wNymqIvOgZL2UT-dqdsLvubqmMwsvtROg"}
]
2024/02/20 04:37:45 DEBUG : 7 go routines active
2024/02/20 04:37:45 DEBUG : gdrive~hasher.bolt: released
2024/02/20 04:37:45 Failed to lsjson: database stopped
When running the command to query the back end
rclone returns with SHA1 & SHA256
$ rclone backend dump 'gdhash:' -vv | grep _2.jpeg
2024/02/20 04:42:42 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "backend" "dump" "gdhash:" "-vv"]
2024/02/20 04:42:42 DEBUG : Using config file from "/home/ubuntu/.config/rclone/rclone.conf"
2024/02/20 04:42:42 INFO : Hasher is EXPERIMENTAL!
2024/02/20 04:42:42 DEBUG : Creating backend with remote "gdrive:"
2024/02/20 04:42:42 DEBUG : hasher::gdhash:: Groups by usage: cached [md5, sha1, whirlpool, sha256], passed [md5, sha1, sha256], auto [md5, sha1, whirlpool, sha256], slow [], supported [md5, sha1, whirlpool, sha256]
2024/02/20 04:42:42 DEBUG : gdrive~hasher.bolt: Opened for reading in 61.52µs
ok md5:03c6c126c60cc85620e77757ea538550 sha1:8391e852a8da44f5b8df99149314dd95a3cb36d4 whirlpool:1990038b3b8c61ff8acd4c7fa1073562dc7fd03760e0040b5c7f94a2f6b2da1730e542b1a2e3d7bcb6288efa0c898853f40083d8c90163334b5d62e6580d8290 sha256:963df079beaefcddbb94aa43d65b1eea51d21e2c045d3c1b3921cf6097351018 90h Google Photos/2013/_2.jpeg
2024/02/20 04:42:43 INFO : /home/ubuntu/.cache/rclone/kv/gdrive~hasher.bolt: 19068 records
2024/02/20 04:42:43 DEBUG : 5 go routines active