siddiquis3
October 5, 2021, 1:08pm
1
when we are copying large file size around 56GB or 100GB md5sum checksum failed with status exit .
rclone version)
rclone v1.56.0
os/version: redhat 7.9 (64 bit)
os/kernel: 3.10.0-1160.36.2.el7.x86_64 (x86_64)
Local server copy using SFTP
rclone copy /tmp remote:tmp)
Paste command here
rclone copy --sftp-host=remote_server :sftp:/datasync --sftp-user=minio --sftp-pass='' /datasync --log-file=/var/log/rclone/rclone_pull.log -v
#### The rclone config contents with secrets removed.
<!-- You should use 3 backticks to begin and end your paste to make it readable. -->
Paste config here
rclone copying data using SFTP from remote server without any Specific config
#### A log from the command with the `-vv` flag
<!-- You should use 3 backticks to begin and end your paste to make it readable. Or use a service such as https://pastebin.com or https://gist.github.com/ -->
Paste log here
2021/10/04 13:39:07 DEBUG : sftp cmd = /edwdata/xfer_data/mozart/out/my_file_mid_20210929.csv
2021/10/04 13:39:07 DEBUG : my_file_mid_20210929.csv: Failed to calculate md5 hash: wait:remote commadn exited without exit status or exit signal ( )
2021/10/04 13:52:58 INFO : my_file_large_20210929.csv: Multi-thread Copied (new)
2021/10/04 13:52:58 INFO :
Transferred: 61.098Gi / 61.098 GiByte, 100%, 26.825 KiByte/s, ETA 0s
Transferred: 3 / 3, 100%
Elapsed time: 16m4.1s
2021/10/04 13:52:58 DEBUG : 3 go routines active
asdffdsa
October 5, 2021, 1:22pm
2
hello,
the exit code is the recommended way to check for errors.
another option, which i use is:
siddiquis3
October 5, 2021, 1:52pm
3
yes , I agree , exit code , but why rclone didnt complete the checksum for large files , if this not fit in rclone in-built timing as large file taking long time for checksum is there any other checks we can also add to ensure the data copied gracefully without relying on checksum , this needed our architect is stuck on this error only
asdffdsa
October 5, 2021, 1:55pm
4
can you post the full debug log?
ncw
October 5, 2021, 2:11pm
5
This looks like it is the md5sum taking so long the ssh session times out.
Try setting --timeout 1h and see if that helps.
siddiquis3
October 5, 2021, 2:33pm
6
here is full logs without timout flag .
2021/10/05 16:18:39 DEBUG : Creating backend with remote ":sftp:incoming/large_file_56GB"
2021/10/05 16:30:52 DEBUG : 4 go routines active
asdffdsa
October 5, 2021, 2:45pm
7
about the sftp server, is that from minio or what?
siddiquis3
October 5, 2021, 2:46pm
8
ncw:
--timeout
I tried with --timeout=1h still same happened
siddiquis3
October 5, 2021, 2:47pm
9
minio is user from which it is copying data from remote linux server
asdffdsa
October 5, 2021, 2:49pm
10
if not minio, then who makes that sftp server, bitvise, rebex for what?
if you copy that same file, using another sftp client, does it upload, does the client perform the checksum?
siddiquis3
October 5, 2021, 2:52pm
11
yes this in built feature in rclone copy using SFTP from remote Linux to Local Linux machine , other plain SFTP doesn't use md5sum by default
asdffdsa
October 5, 2021, 2:55pm
12
sorry, not sure i understand what you wrote.
the sftp server, who makes it, Cerberus, openssh, solarwinds or what?
does that sftp server support checksum on file uploads?
siddiquis3
October 5, 2021, 3:01pm
13
asdffdsa:
openssh
that server is RHEL 7.9 Linux machine and yes that support md5sum checksum , for smaller size file it is working fine
asdffdsa
October 5, 2021, 3:15pm
14
perhaps tryrclone check source dest --download
ncw
October 6, 2021, 10:52am
15
Ah, looks like this is the sftp backend tidying up unused connection which it shouldn't do.
Try this fix.
v1.57.0-beta.5681.3df0b2fe1.fix-sftp-timeout on branch fix-sftp-timeout (uploaded in 15-30 mins)
system
October 9, 2021, 10:53am
16
This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.
ncw
October 15, 2021, 3:22pm
17
I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.57
1 Like