SFTP transfers of files over a few hundred mb fail

What is the problem you are having with rclone?

SFTP transfers fail with files over a few hundred mb in size

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: debian 12.5 (64 bit)
  • os/kernel: 6.1.0-18-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

sftp

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /mnt/main/backup/Downloads/ backup:/Downloads/ --stats-file-name-length 0 --size-only --transfers 5 -vvP --links --no-update-dir-modtime

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[backup]
type = crypt
remote = server-backup:/home/backup
filename_encryption = off
directory_name_encryption = false
password = XXX
password2 = XXX

[servver-backup]
type = sftp
host = XXX
port = 22
key_file = /root/.ssh/id_ed25519
shell_type = unix
md5sum_command = md5sum
sha1sum_command = sha1sum

A log from the command that you were trying to run with the -vv flag

2024/04/09 09:16:58 DEBUG : sftp://root@hostname/home/backup/Downloads: Not closing 8 unused connections as 5 sessions active
2024/04/09 09:17:58 DEBUG : sftp://root@hostname/home/backup/Downloads: Not closing 8 unused connections as 5 sessions active
2024/04/09 09:18:14 DEBUG : sftp://root@hostname/home/backup/Downloads: Connection failed, closing: connection lost
2024/04/09 09:18:15 DEBUG : LargeFile.zip.buyeger4.partial: Removed after failed upload: connection lost
2024/04/09 09:18:15 ERROR : LargeFile.zip: Failed to copy: Update ReadFrom failed: connection lost

welcome to the forum,

for testing, copy a single file only using the simplest command possible and post the full debug log
rclone copy /mnt/main/backup/Downloads/LargeFile.zip backup:/Downloads/ -vv

make sure there is not an issue with the sftp server?
testing other another sftp client, such as filezilla?

I'm getting the log info now. It can take a bit before it fails. I've transferred over 500gb of data through this same connection with smaller files and have not had any issues.

2024/04/09 09:55:43 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "copy" "/mnt/main/backup/Downloads/debian-12.5.0-amd64-DVD-1.iso" "backup:/Downloads/" "-vvP"]
2024/04/09 09:55:43 DEBUG : Creating backend with remote "/mnt/main/backup/Downloads/debian-12.5.0-amd64-DVD-1.iso"
2024/04/09 09:55:43 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/04/09 09:55:43 DEBUG : fs cache: adding new entry for parent of "/mnt/main/backup/Downloads/debian-12.5.0-amd64-DVD-1.iso", "/mnt/main/backup/Downloads"
2024/04/09 09:55:43 DEBUG : Creating backend with remote "backup:/Downloads/"
2024/04/09 09:55:43 DEBUG : Creating backend with remote "server-backup:/home/backup/Downloads/.bin"
2024/04/09 09:55:44 DEBUG : sftp://root@hostname:22/home/backup/Downloads/.bin: New connection xx.xx.xx.xx:53888->xx.xx.xx.xx:22 to "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2"
2024/04/09 09:55:44 DEBUG : sftp://root@hostname:22/home/backup/Downloads/.bin: Shell type "unix" from config
2024/04/09 09:55:44 DEBUG : sftp://root@hostname:22/home/backup/Downloads/.bin: Using root directory "/home/backup/Downloads/.bin"
2024/04/09 09:55:44 DEBUG : Creating backend with remote "server-backup:/home/backup/Downloads"
2024/04/09 09:55:44 DEBUG : sftp://root@hostname:22/home/backup/Downloads: New connection xx.xx.xx.xx:53904->xx.xx.xx.xx:22 to "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2"
2024/04/09 09:55:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Shell type "unix" from config
2024/04/09 09:55:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Using root directory "/home/backup/Downloads"
2024/04/09 09:55:45 DEBUG : debian-12.5.0-amd64-DVD-1.iso: Need to transfer - File not found at Destination
2024/04/09 09:56:44 DEBUG : sftp://root@hostname:22/home/backup/Downloads/.bin: Closing 1 unused connections
2024/04/09 09:56:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Not closing 0 unused connections as 1 sessions active
2024/04/09 09:57:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Not closing 0 unused connections as 1 sessions active
2024/04/09 09:58:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Not closing 0 unused connections as 1 sessions active
2024/04/09 09:59:45 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Not closing 0 unused connections as 1 sessions active
2024/04/09 10:00:19 DEBUG : sftp://root@hostname:22/home/backup/Downloads: Connection failed, closing: connection lost
2024/04/09 10:00:19 DEBUG : sftp://root@hostname:22/home/backup/Downloads: New connection xx.xx.xx.xx:57506->xx.xx.xx.xx:22 to "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2"
2024/04/09 10:00:20 DEBUG : debian-12.5.0-amd64-DVD-1.iso.nogizay4.partial: Removed after failed upload: connection lost
2024/04/09 10:00:20 ERROR : debian-12.5.0-amd64-DVD-1.iso: Failed to copy: Update ReadFrom failed: connection lost
2024/04/09 10:00:20 ERROR : Attempt 1/3 failed with 1 errors and: Update ReadFrom failed: connection lost
2024/04/09 10:00:20 DEBUG : debian-12.5.0-amd64-DVD-1.iso: Need to transfer - File not found at Destination
2024/04/09 10:00:37 INFO  : Signal received: interrupt
2024/04/09 10:00:38 DEBUG : sftp://root@hostname:22/home/backup/Downloads: New connection xx.xx.xx.xx:51938->xx.xx.xx.xx:22 to "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2"
2024/04/09 10:00:38 INFO  : debian-12.5.0-amd64-DVD-1.iso.siyunul2.partial: Removing failed copy
2024/04/09 10:00:38 INFO  : Exiting...

can you upload debian-12.5.0-amd64-DVD-1.iso using filezilla or another sftp client??

Hmm, looks like that fails as well. Not sure why I didn't think to check that. Thanks. I'll dig into the server side.

sftp> put debian-12.5.0-amd64-DVD-1.iso
Uploading debian-12.5.0-amd64-DVD-1.iso to /root/debian-12.5.0-amd64-DVD-1.iso
debian-12.5.0-amd64-DVD-1.iso                                                                                                                           4%  166MB   1.1MB/s   55:12 ETA
client_loop: send disconnect: Broken pipe

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.