Rclone copy logs not captured with any error when cancel explicitly or network lost during copy

Help and Support
1

What is the problem you are having with rclone?

rclone sync --sftp-host=SFTPSERVER :sftp:/datasync/ --sftp-user=cloud-ops --sftp-pass='' /opt/datasync  --log-file=/var/log/rclone/rclone_rep.log  -vvv  -P

While I am doing rclone copy , i am capturing the logs in log file .
what I have notice , during file transfer if I do cntrl+c or I lost the network , there is no alert written in logs that cancelled copy or network lost something like .

What is your rclone version (output from rclone version)

rclone v1.55.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

RHEL 7.9

Which cloud storage system are you using? (eg Google Drive)

Local SFTP copy server to server

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --sftp-host=SFTPSERVER :sftp:/datasync/ --sftp-user=cloud-ops --sftp-pass='' /opt/datasync  --log-file=/var/log/rclone/rclone_rep.log  -vvv  -P (with --log-level DEBUG ) also we tried

The rclone config contents with secrets removed.

Local server to server copy so no rclone config required

A log from the command with the -vv flag

2021/08/12 17:31:58 DEBUG : data1/data1/7: Unchanged skipping
2021/08/12 17:31:58 DEBUG : data1/data1/8: Size and modification time the same (differ by 0s, within tolerance 1s)
2021/08/12 17:31:58 DEBUG : data1/data1/8: Unchanged skipping
2021/08/12 17:31:58 DEBUG : Local file system at /opt/SP/edwdata/datasync: Waiting for checks to finish
2021/08/12 17:31:58 DEBUG : Local file system at /opt/SP/edwdata/datasync: Waiting for transfers to finish
2021/08/12 17:31:58 DEBUG : bigfiles/file1: Starting multi-thread copy with 4 parts of size 2.500G
2021/08/12 17:31:58 DEBUG : bigfiles/file1: multi-thread copy: stream 4/4 (8053063680-10737418240) size 2.500G starting
2021/08/12 17:31:58 DEBUG : bigfiles/file1: multi-thread copy: stream 1/4 (0-2684354560) size 2.500G starting
2021/08/12 17:31:58 DEBUG : bigfiles/file1: multi-thread copy: stream 2/4 (2684354560-5368709120) size 2.500G starting
2021/08/12 17:31:58 DEBUG : bigfiles/file1: multi-thread copy: stream 3/4 (5368709120-8053063680) size 2.500G starting
2021/08/12 17:32:13 DEBUG : data3/data3/file.out: multi-thread copy: stream 3/4 (1073741824-1610612736) size 512M finished
2021/08/12 17:32:14 DEBUG : data3/data3/file.out: multi-thread copy: stream 2/4 (536870912-1073741824) size 512M finished
2021/08/12 17:32:14 DEBUG : data2/data2/dag.file: multi-thread copy: stream 3/4 (1073741824-1610612736) size 512M finished
2021/08/12 17:32:15 DEBUG : data2/data2/dag.file: multi-thread copy: stream 1/4 (0-536870912) size 512M finished
2021/08/12 17:32:17 DEBUG : data3/data3/file.out: multi-thread copy: stream 4/4 (1610612736-2147483648) size 512M finished
2021/08/12 17:32:17 DEBUG : data2/data2/dag.file: multi-thread copy: stream 4/4 (1610612736-2147483648) size 512M finished
2021/08/12 17:32:18 DEBUG : data2/data2/dag.file: multi-thread copy: stream 2/4 (536870912-1073741824) size 512M finished
2021/08/12 17:32:18 DEBUG : data2/data2/dag.file: Finished multi-thread copy with 4 parts of size 512M
2021/08/12 17:32:20 DEBUG : data3/data3/file.out: multi-thread copy: stream 1/4 (0-536870912) size 512M finished
2021/08/12 17:32:20 DEBUG : data3/data3/file.out: Finished multi-thread copy with 4 parts of size 512M
2

Hi Suhail,

Interesting, this is the (redacted) output I see when I disable my WiFi while copying to a SFTP server (from Windows):

PS> rclone copy ./testfolder SFTPserver:testfolder --ignore-times
2021/08/12 15:35:12 ERROR : sftp://REDACTED:testfolder: Discarding closed SSH connection: read tcp REDACTED->REDACTED: wsarecv: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.     
2021/08/12 15:35:23 ERROR : AuthIssue.zip: Failed to copy: Update ReadFrom failed: failed to send packet: EOF
2021/08/12 15:35:23 ERROR : Attempt 1/3 failed with 1 errors and: Update ReadFrom failed: failed to send packet: EOF
2021/08/12 15:35:43 ERROR : : error reading destination directory: List failed: dirExists: couldn't connect SSH: dial tcp: lookup REDACTED: no such host
2021/08/12 15:35:43 ERROR : Attempt 2/3 failed with 1 errors and: List failed: dirExists: couldn't connect SSH: dial 
tcp: lookup REDACTED no such host
2021/08/12 15:36:03 ERROR : : error reading destination directory: List failed: dirExists: couldn't connect SSH: dial tcp: lookup REDACTED: no such host
2021/08/12 15:36:03 ERROR : Attempt 3/3 failed with 1 errors and: List failed: dirExists: couldn't connect SSH: dial 
tcp: lookup REDACTED: no such host
2021/08/12 15:36:03 Failed to copy: List failed: dirExists: couldn't connect SSH: dial tcp: lookup REDACTED: no such host

I don't think rclone has a Ctrl-C handler, so if you do that then you will see nothing but an abrupt end of the log.

3

yeah this could be the case when you stop the WIFI where you trigger the rclone copy command , in my case I am pulling the data from remote server using rclone and network dropped at remote server from I am pulling data not on my server from where I am running command :frowning:

4

Tried that too, similar output with this error message:

A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

So the error messages looks fine to me (on Windows).

Are you able to reproduce your issue using another RHEL client?

5

Yes even I tried on different RHEL machines same thing happened it is always trying/waiting for connection re-established , may be there will be option to how long connection will wait/retry to connection re-established.

6

Sorry, I don’t know how to set SFTP connection time-outs in linux. I mainly use Windows.

What happens if you use the linux sftp command directly, does it time-out with a proper error?

Perhaps, you will be able to work around the issue with the --timeout flag.

7

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.