What is the problem you are having with rclone?
Using rclone md5sum
to recursively produce MD5 hashes for all files in a very large Encrypted Google Drive dirtree
produces multiple errors like the following:
2023/02/18 15:24:43 INFO : can't close account: file already closed
2023/02/18 15:24:43 ERROR : REDACTED10/REDACTED11/REDACTED12.dd.gz: failed to copy file to hasher: failed to authenticate decrypted block - bad password?
So far, after almost 2 weeks running non-stop and having generated the MD5 hashes for 4.3M files (about 1/3 of my total 13M+ files), it has generated 21 'sets' of the above 2 lines.
So far, it has also generated (only once each) two other errors:
2023/02/19 08:50:12 ERROR : REDACTED20/REDACTED21/REDACTED22.JPG: failed to open file REDACTED20/REDACTED21/REDACTED22.JPG: open file failed: googleapi: Error 400: Bad Request, failedPrecondition
and
2023/03/01 10:49:56 ERROR : REDACTED30/REDACTED31/REDACTED32/REDACTED33/REDACTED34/REDACTED35/REDACTED36/REDACTED37/REDACTED38.pdf: failed to open file REDACTED30/REDACTED31/REDACTED32/REDACTED33/REDACTED34/REDACTED35/REDACTED36/REDACTED37/REDACTED38.pdf: open file failed: googleapi: Error 401: Invalid Credentials, authError
The strange thing is, if I run a separate rclone md5sum
for these files,
Run the command 'rclone version' and share the full output of the command.
rclone v1.61.1
- os/version: debian 11.1 (64 bit)
- os/kernel: 6.0.0-4-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Encrypted remote on top of a Google Drive remote.
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone -v --rc --rc-addr 127.0.0.1:5570 --transfers=8 --checkers=32 --low-level-retries=1000 --retries=1000 --drive-chunk-size=64m md5sum --download --output-file ~/ENCRYPTED_GOOGLE_DRIVE.md5 ENCRYPTED_GOOGLE_DRIVE:```
The rclone config contents with secrets removed.
[ENCRYPTED_GOOGLE_DRIVE]
type = crypt
remote = GOOGLE_DRIVE:GOOGLE_DRIVE_ROOT
filename_encryption = standard
password = REDACTED00
password2 = REDACTED01
[GOOGLE_DRIVE]
type = drive
client_id = REDACTED06
client_secret = REDACTED07
token = {"access_token":"REDACTED08","token_type":"Bearer","refresh_token":"REDACTED09","expiry":"2023-02-16T12:18:56.541006812-03:00"}
root_folder_id = REDACTED10
A log from the command with the -vv
flag
This is basically impossible due to the low frequency of these errors and the very very large debug output that would be produced (I probably don't have the disk space to store that much debug output!)