What is the problem you are having with rclone?
hashshum option doesn't try to re-downloaded files if it fails during the hashing.
We are doing quite big hashsum. The server on occasions returns 50X. When problems happens I have:
ERROR file_path/file_name.txt
The files are readable when I try in a later time. The server just has a "bad moment". I think (correct me if I'm wrong!) that rclone doesn't try to re-download the files on the hashum operation which would be fantastic. Let me know if you need the "-vv" log: I have a few more large hashshum's to do the next two weeks.
What is your rclone version (output from rclone version
)
rclone v1.54.0
- os/arch: linux/amd64
- go version: go1.15.7
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Linux 64 bit (Ubuntu 20.04.2 LTS, rclone installed from the .zip)
Which cloud storage system are you using? (eg Google Drive)
S3 based storage (based on Scality)
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone hashsum MD5 spibucket2-readonly:bucket_name --download --output-file verifications/bucket_name.txt --progress --transfers=30
The rclone config contents with secrets removed.
[spibucket2-readonly]
type = s3
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
region =
endpoint = the_end_point
location_constraint =
acl = private
server_side_encryption =
storage_class =
A log from the command with the -vv
flag
I don't have the output of -vv
and since it's a server side thing I'm not able to reproduce right now.
I still have part of the output:
2021-02-18 23:10:31 ERROR : a_file_path/file_name.bin: Failed to open file file_path... : SerializationError: failed to unmarshal error message
status code: 502, request id: , host id:
caused by: UnmarshalError: failed to unmarshal error message
00000000 3c 68 74 6d 6c 3e 0d 0a 3c 68 65 61 64 3e 3c 74 |<html>..<head><t|
00000010 69 74 6c 65 3e 35 30 32 20 42 61 64 20 47 61 74 |itle>502 Bad Gat|
00000020 65 77 61 79 3c 2f 74 69 74 6c 65 3e 3c 2f 68 65 |eway</title></he|
00000030 61 64 3e 0d 0a 3c 62 6f 64 79 20 62 67 63 6f 6c |ad>..<body bgcol|
00000040 6f 72 3d 22 77 68 69 74 65 22 3e 0d 0a 3c 63 65 |or="white">..<ce|
00000050 6e 74 65 72 3e 3c 68 31 3e 35 30 32 20 42 61 64 |nter><h1>502 Bad|
00000060 20 47 61 74 65 77 61 79 3c 2f 68 31 3e 3c 2f 63 | Gateway</h1></c|
00000070 65 6e 74 65 72 3e 0d 0a 3c 68 72 3e 3c 63 65 6e |enter>..<hr><cen|
00000080 74 65 72 3e 6e 67 69 6e 78 3c 2f 63 65 6e 74 65 |ter>nginx</cente|
00000090 72 3e 0d 0a 3c 2f 62 6f 64 79 3e 0d 0a 3c 2f 68 |r>..</body>..</h|
000000a0 74 6d 6c 3e 0d 0a |tml>..|
caused by: expected element type <Error> but have <html>
Thank you very much!