When rclone copying files over >100 MB from Sharefile to remote, MD5 hash differ and transfer fails

tumen · December 20, 2021, 9:36pm

What is the problem you are having with rclone?

When I rclone copy files over 100Mb from Sharefile, the transfer fails due to MD5 hash differ error.
When I do this with --ignore-checksum option, it seems to work fine.
How can I get it working with MD5 checksum option for files over 100MB for integrity check?

What is your rclone version (output from `rclone version`)

rclone v1.55.1

os/type: linux
os/arch: amd64
go/version: go1.16.3
go/linking: static
go/tags: none
-->

Which cloud storage system are you using? (eg Google Drive)

ShareFile

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy -c -v sharefile:path remote:path

The rclone config contents with secrets removed.

Paste config here

A log from the command with the `-vv` flag

 corrupted on transfer: MD5 hash differ "fdfffe6d-1b70-4697-9846-1e270424a681" vs "d19f663cdab5f2f894e66b81d8eb2f8b"'
2021/12/20 21:33:43 DEBUG : 7 go routines active
2021/12/20 21:33:43 Failed to copy with 2 errors: last error was: corrupted on transfer: MD5 hash differ "fdfffe6d-1b70-4697-9846-1e270424a681" vs "d19f663cdab5f2f894e66b81d8eb2f8b"

asdffdsa · December 20, 2021, 9:44pm

hello and welcome to the forum,

fdfffe6d-1b70-4697-9846-1e270424a681
that does not look like a md5 hash, with those dashes
update rclone to latest stable, v1.57.0 and test again.

if that does not fix the issue then

post the redacted remote, rclone config show sharefile
copy a single file over 100MB and post the entire debug log, use -vv, not -v
upload the file without using --checksum and --ignore-checksum
might test using --sharefile-upload-cutoff=5M

tumen · December 20, 2021, 10:10pm

I upgraded to v.157.0 and re-tested, but still seeing the same issue. Here are the rclone config and debug log as requested. Any help would be appreciated.

[sharefile_test]
type = sharefile
root_folder_id = allshared
endpoint = https://xxxxxxx.sharefile.com
token = {xxxxxxxxxxxxxxxxxxx}

2021/12/20 22:03:09 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "-c" "-vv" "sharefile_test:test.file" "remote_path"]
2021/12/20 22:03:09 DEBUG : Creating backend with remote "sharefile_test:test.file"
2021/12/20 22:03:09 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/20 22:03:09 DEBUG : sharefile root 'test.file': Token expired but no uploads in progress - doing nothing
2021/12/20 22:03:09 DEBUG : sharefile_test: Loaded invalid token from config file - ignoring
2021/12/20 22:03:09 DEBUG : Saving config "token" in section "sharefile_test" of the config file
2021/12/20 22:03:09 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2021/12/20 22:03:09 DEBUG : sharefile_test: Saved new token in config file
2021/12/20 22:03:11 DEBUG : fs cache: adding new entry for parent of "sharefile_test:test.file", "sharefile_test:test.file"
2021/12/20 22:03:11 DEBUG : Creating backend with remote "remote_path"
2021/12/20 22:03:11 DEBUG : test.file: Need to transfer - File not found at Destination
2021/12/20 22:03:19 DEBUG : test.file: md5 = 988d957b-ff64-4a3d-8257-97de0ec659fa (sharefile root 'test.file')
2021/12/20 22:03:19 DEBUG : test.file: md5 = d19f663cdab5f2f894e66b81d8eb2f8b (Local file system at remote_path)
2021/12/20 22:03:19 ERROR : test.file: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:19 INFO : test.file: Removing failed copy
2021/12/20 22:03:19 ERROR : Attempt 1/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:19 DEBUG : test.file: Need to transfer - File not found at Destination
2021/12/20 22:03:24 DEBUG : test.file: md5 = 988d957b-ff64-4a3d-8257-97de0ec659fa (sharefile root 'test.file')
2021/12/20 22:03:24 DEBUG : test.file: md5 = d19f663cdab5f2f894e66b81d8eb2f8b (Local file system at remote_path)
2021/12/20 22:03:24 ERROR : test.file: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:24 INFO : test.file: Removing failed copy
2021/12/20 22:03:24 ERROR : Attempt 2/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:24 DEBUG : test.file: Need to transfer - File not found at Destination
2021/12/20 22:03:29 DEBUG : test.file: md5 = 988d957b-ff64-4a3d-8257-97de0ec659fa (sharefile root 'test.file')
2021/12/20 22:03:29 DEBUG : test.file: md5 = d19f663cdab5f2f894e66b81d8eb2f8b (Local file system at remote_path)
2021/12/20 22:03:29 ERROR : test.file: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:29 INFO : test.file: Removing failed copy
2021/12/20 22:03:29 ERROR : Attempt 3/3 failed with 1 errors and: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:03:29 INFO :
Transferred: 442.806 MiB / 442.806 MiB, 100%, 26.236 MiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 19.7s

2021/12/20 22:03:29 DEBUG : 6 go routines active
2021/12/20 22:03:29 Failed to copy: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"

asdffdsa · December 20, 2021, 10:15pm

hmm, that is strange output

what is remote:path, do not see the rclone remote info for that?

tumen · December 20, 2021, 10:21pm

Tried it with shared-upload-cutoff flag and still getting the same error.
remote_path is a shared drive

2021/12/20 22:17:31 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "sharefile_test:test.file" "remote_path" "--sharefile-upload-cutoff=5M" "--retries=1" "-vv"]
2021/12/20 22:17:31 DEBUG : Creating backend with remote "sharefile_test:test.file"
2021/12/20 22:17:31 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/20 22:17:31 DEBUG : sharefile_test: detected overridden config - adding "{JvLx1}" suffix to name
2021/12/20 22:17:31 DEBUG : sharefile root 'test.file': Token expired but no uploads in progress - doing nothing
2021/12/20 22:17:31 DEBUG : sharefile_test{JvLx1}: Loaded invalid token from config file - ignoring
2021/12/20 22:17:31 DEBUG : Saving config "token" in section "sharefile_test" of the config file
2021/12/20 22:17:31 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2021/12/20 22:17:31 DEBUG : sharefile_test{JvLx1}: Saved new token in config file
2021/12/20 22:17:33 DEBUG : fs cache: adding new entry for parent of "sharefile_test:test.file", "sharefile_test{JvLx1}:test.file"
2021/12/20 22:17:33 DEBUG : Creating backend with remote "remote_path"
2021/12/20 22:17:33 DEBUG : test.file: Need to transfer - File not found at Destination
2021/12/20 22:17:39 DEBUG : test.file: md5 = 988d957b-ff64-4a3d-8257-97de0ec659fa (sharefile root 'test.file')
2021/12/20 22:17:39 DEBUG : test.file: md5 = d19f663cdab5f2f894e66b81d8eb2f8b (Local file system at remote_path)
2021/12/20 22:17:39 ERROR : test.file: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:17:39 INFO : test.file: Removing failed copy
2021/12/20 22:17:39 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 22:17:39 INFO :
Transferred: 147.602 MiB / 147.602 MiB, 100%, 27.599 MiB/s, ETA 0s
Errors: 1 (retrying may help)
Elapsed time: 7.7s

2021/12/20 22:17:39 DEBUG : 6 go routines active
2021/12/20 22:17:39 Failed to copy: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"

asdffdsa · December 20, 2021, 10:26pm

so if the file is 99MB, then no problems???

not sure what that exactly means, shared with what?
if that is a rclone remote, please post the details, same as you did for sharefile_test

for a 100MB file, i would make sure able to

copy from local to sharefile and copy from sharefile to local.
copy from local to remote_path and copy from remote_path to local.

tumen · December 20, 2021, 10:30pm

Sorry. remote_path is a network shared drive (e.g. /directory/to/path/dir). I redacted it to "remote_path"

asdffdsa · December 20, 2021, 10:32pm

what kind of network shared drive, samba, nfs or what?
the more details you provide the better, so no need to pull it from you, thanks

i would try copy from sharefile to local.

tumen · December 20, 2021, 10:50pm

Samba drive mounted on the OS. I t tried copying from Sharefile to local. Still the same error.

asdffdsa · December 20, 2021, 11:00pm

ok, no need to test with that samba, just use local.

for that file you just downloaded, i would run rclone md5sum

and rclone check between sharefile folder and the local folder.

and rclone lsf sharefile_test: --format=ph

tumen · December 20, 2021, 11:19pm

rclone md5sum sharefile_test:test.file
988d957b-ff64-4a3d-8257-97de0ec659fa test.file

rclone check sharefile_test:test.file /app/
2021/12/20 23:12:16 ERROR : test.file: File not in Local file system at /app/
2021/12/20 23:12:16 NOTICE: Local file system at /app/: 1 files missing
2021/12/20 23:12:16 NOTICE: Local file system at /app/: 1 differences found
2021/12/20 23:12:16 NOTICE: Local file system at /app/: 1 errors while checking
2021/12/20 23:12:16 Failed to check: 1 differences found

rclone lsf sharefile_test:t --format=ph
test.file;988d957b-ff64-4a3d-8257-97de0ec659fa

I tried copying from sharefile to local app directory, but still getting the same error.

2021/12/20 23:18:53 DEBUG : test.file: md5 = d19f663cdab5f2f894e66b81d8eb2f8b (Local file system at /app)
2021/12/20 23:18:53 ERROR : test.file: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"
2021/12/20 23:18:53 INFO : test.file: Removing failed copy
2021/12/20 23:18:53 ERROR : Attempt 1/1 failed with 1 errors and: corrupted on transfer: md5 hash differ "988d957b-ff64-4a3d-8257-97de0ec659fa" vs "d19f663cdab5f2f894e66b81d8eb2f8b"

asdffdsa · December 20, 2021, 11:27pm

not sure what is going on here, as that md5 hash should not have -
and i do not have a sharefile account, they do no offer a free account.

rclone md5sum sharefile_test:test.file
988d957b-ff64-4a3d-8257-97de0ec659fa test.fil

copying from sharefile to local app directory
about local app directory to sharefile?

and for a one byte text file?

tumen · December 21, 2021, 1:14am

It seems to work for copying over a small file.

asdffdsa · December 21, 2021, 1:19am

ok, i am going to guess what is going on,

once a file is larger than 128MB, rclone will start a multi-part upload, and there is some issue there.

i think we can test that by forcing rclone to do a multi-part upload for smaller file.

to upload a file of approx 5-20MB in size and use
--sharefile-upload-cutoff=1MB
and of course a full debug log.

tumen · December 21, 2021, 6:07pm

I was able to rclone copy a file of 5MB in size with --sharefile-upload-cutoff=1M flag. Here is the a full redacted debug log. Any other suggestions or ideas?

2021/12/21 18:00:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/21 18:00:39 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "--sharefile-upload-cutoff=1M" "-c" "-vv" "--retries=1" "sharefile_test:/Test" "/remote/test"]
2021/12/21 18:00:39 DEBUG : Creating backend with remote "sharefile_test:Test"
2021/12/21 18:00:39 DEBUG : sharefile_test: detected overridden config - adding "{jMuWh}" suffix to name
2021/12/21 18:00:39 DEBUG : sharefile root 'Test': Token expired but no uploads in progress - doing nothing
2021/12/21 18:00:39 DEBUG : sharefile_test{jMuWh}: Loaded invalid token from config file - ignoring
2021/12/21 18:00:40 DEBUG : Saving config "token" = "{"access_token":"xxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxx","expiry":"2021-12-22T02:00:40.242757851Z"}" in section "sharefile_test" of the config file
2021/12/21 18:00:40 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2021/12/21 18:00:40 DEBUG : sharefile_test{jMuWh}: Saved new token in config file
2021/12/21 18:00:40 DEBUG : fs cache: renaming cache item "sharefile_test:/Test" to be canonical "sharefile_test{jMuWh}:Test"
2021/12/21 18:00:40 DEBUG : Creating backend with remote "/remote/test"
2021/12/21 18:00:40 DEBUG : Local file system at /remote/test: Waiting for checks to finish
2021/12/21 18:00:40 DEBUG : Local file system at /remote/test: Waiting for transfers to finish
2021/12/21 18:00:43 DEBUG : test_file2.txt: MD5 = 694f54b964da6084944900b51f17683c OK
2021/12/21 18:00:43 INFO : test_file2.txt: Copied (new)
2021/12/21 18:00:43 INFO :
Transferred: 4.920M / 4.920 MBytes, 100%, 2.009 MBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3.5s
2021/12/21 18:00:43 DEBUG : 7 go routines active

asdffdsa · December 21, 2021, 6:18pm

that is an old version, best to use latest stable, v1.57.0
i could be wrong, but i see the problem as what causes the md5 hash to look like
fdfffe6d-1b70-4697-9846-1e270424a681 to have dashes.
is it sharefile or rclone?
might try to upload a file over 128MiB using --sharefile-upload-cutoff=1M
if possible, enclose the debug logs with three backticks so it looks like this.

2021/12/21 18:00:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/21 18:00:39 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "--sharefile-upload-cutoff=1M" "-c" "-vv" "--retries=1" "sharefile_test:/Test" "/remote/test"]

tumen · December 21, 2021, 6:44pm

Upgraded to the latest stable and tried again. Still getting the same error and the hash still has dashes.

btw, is it possible rclone caching the hash somewhere? What is this doing? fs cache: renaming cache item

2021/12/21 18:39:34 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "--sharefile-upload-cutoff=1M" "-c" "-vv" "--retries=1" "sharefile_test:/Test" "/remote/test"]
2021/12/21 18:39:34 DEBUG : Creating backend with remote "sharefile_test:/Test"
2021/12/21 18:39:34 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2021/12/21 18:39:34 DEBUG : sharefile_test: detected overridden config - adding "{f81XM}" suffix to name
2021/12/21 18:39:34 DEBUG : sharefile root '/Test': Token expired but no uploads in progress - doing nothing
2021/12/21 18:39:34 DEBUG : sharefile_test{f81XM}: Loaded invalid token from config file - ignoring
2021/12/21 18:39:35 DEBUG : Saving config "token" in section "sharefile_test" of the config file
2021/12/21 18:39:35 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2021/12/21 18:39:35 DEBUG : sharefile_test{f81XM}: Saved new token in config file
2021/12/21 18:39:36 DEBUG : fs cache: renaming cache item "sharefile_test:/Test" to be canonical "sharefile_test{f81XM}:/Test"
2021/12/21 18:39:36 DEBUG : Creating backend with remote "/remote/test"
2021/12/21 18:39:36 DEBUG : Local file system at /remote/test: Waiting for checks to finish
2021/12/21 18:39:36 DEBUG : Local file system at /remote/test: Waiting for transfers to finish
2021/12/21 18:39:41 DEBUG : test_file.zip: md5 = c103d268-fb40-4357-bd27-c5ff4e251e7f (sharefile root '/Test')
2021/12/21 18:39:41 DEBUG : test_file.zip: md5 = 7a1ea3ed267ee8b1a6fd06a565cd9755 (Local file system at /remote/test)
2021/12/21 18:39:41 ERROR : test_file.zip: corrupted on transfer: md5 hash differ "c103d268-fb40-4357-bd27-c5ff4e251e7f" vs "7a1ea3ed267ee8b1a6fd06a565cd9755"
2021/12/21 18:39:41 INFO  : test_file.zip: Removing failed copy
2021/12/21 18:39:42 DEBUG : testfile.txt: md5 = 16433c32-f134-4d88-99b1-29200f4f9416 (sharefile root '/Test')
2021/12/21 18:39:42 DEBUG : testfile.txt: md5 = baee33d303187f14bd4d5a113a638f0c (Local file system at /remote/test)
2021/12/21 18:39:42 ERROR : testfile.txt: corrupted on transfer: md5 hash differ "16433c32-f134-4d88-99b1-29200f4f9416" vs "baee33d303187f14bd4d5a113a638f0c"
2021/12/21 18:39:42 INFO  : testfile.txt: Removing failed copy
2021/12/21 18:39:42 ERROR : Attempt 1/1 failed with 2 errors and: corrupted on transfer: md5 hash differ "16433c32-f134-4d88-99b1-29200f4f9416" vs "baee33d303187f14bd4d5a113a638f0c"
2021/12/21 18:39:42 INFO  :
Transferred:      262.986 MiB / 262.986 MiB, 100%, 43.826 MiB/s, ETA 0s
Errors:                 2 (retrying may help)
Elapsed time:         7.9s

2021/12/21 18:39:42 DEBUG : 6 go routines active
2021/12/21 18:39:42 Failed to copy with 2 errors: last error was: corrupted on transfer: md5 hash differ "16433c32-f134-4d88-99b1-29200f4f9416" vs "baee33d303187f14bd4d5a113a638f0c"

asdffdsa · December 21, 2021, 8:46pm

not possible.
fs cache: renaming cache item is just some confusing terminology for how rclone calls remotes.
as that would be in any debug log, you can just ignore that.

ok, let's see where that strange md5 hash is coming from.

for test_file.zip that is in sharefile, try both of these.

rclone md5sum sharefile_test:Test/test_file.zip
rclone md5sum sharefile_test:Test/test_file.zip --download

tumen · December 21, 2021, 9:14pm

It appears that the dashes are coming from first one:

c103d268-fb40-4357-bd27-c5ff4e251e7f test_file.zip
7a1ea3ed267ee8b1a6fd06a565cd9755 test_file.zip

asdffdsa · December 21, 2021, 9:15pm

sorry, please run the commands with debug logs and post the logs

and run rclone md5sum -vv on the local test_file.zip