SFTP, SMB transfer failed

[w1q06y3t]

What is the problem you are having with rclone?

I'm currently traveling in Indonesia, and my NAS server is in South Korea. I've posted about this before, but I'm trying to transfer videos from here to my NAS server in Korea using a variety of methods, including FTPS.

In my last post, I was advised to try SMB (over VPN) and SFTP, so I followed that advice, but neither transfer method worked.

For SMB, I used the VPN server app running on my Synology NAS to connect to the network on my NAS with OpenVPN, and then tried to transfer the rclone config over SMB. However, none of the files I tried to transfer ended up on the NAS, only .partial files (which do not appear to be complete files).

When transferring local files to the NAS in Korea via SFTP, some files keep failing to transfer. According to the logs, it seems to be a connection failed issue, but the other files that are being transferred at the same time are transferred just fine, so I have to re-run the command several times. Also, files transferred by protocols other than SFTP (e.g. WebDAV) were retransmitted even though they were already transferred. According to the logs, it seems to be due to different modification times, but I don't know why the same file would have different modification times.

Of course, the network here in Indonesia is very slow (around 1MB/s) and we have periodic power outages. I have an alternative of uploading files to Google Drive and synchronizing them back to my NAS (Google Drive transfers well and without errors, thankfully), but if Google Drive is not available, I would like to find a way to upload to a NAS in another country normally in a poor network environment.

The bottom line is that if I use SFTP to transfer files, some files will be transferred correctly. However, some files don't transfer, so I have to manually compare the file list and retry.

Meanwhile, if I use SMB to transfer files, it's impossible to transfer files at all. Only partial files are created and the entire file is not transferred. The partial files also vary in size: 19MB, 190MB, etc.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.0

• os/version: darwin 14.2.1 (64 bit)
• os/kernel: 23.2.0 (arm64)
• os/type: darwin
• os/arch: arm64 (ARMv8 compatible)
• go/version: go1.21.4
• go/linking: dynamic
• go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

SFTP and SMB(via VPN)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "/Users/yhlee/Desktop/업로드용 임시 폴더" "nas_sftp:Backup/2023 인니 여행 저장" --transfers 40 -P -vv --exclude "**.py" --exclude ".DS_Store" --exclude "**.txt" --log-file=log_SFTP2.txt

rclone copy "/Users/yhlee/Desktop/업로드용 임시 폴더" "nas_smb:Backup/2023 인니 여행 저장/smb_test2" --transfers 40 -P -vv --exclude "**.py" --exclude ".DS_Store" --exclude "**.txt" --log-file=log_SMB2.txt

The rclone config contents with secrets removed.

[nas_sftp]
type = sftp
host = nas.stdl.kr
user = synolit
port = 4929
pass = **
shell_type = unix
md5sum_command = none
sha1sum_command = none

[nas_smb]
type = smb
host = 10.8.0.1
user = synolit
pass = **

A log from the command with the -vv flag

log for SFTP : https://pastebin.com/dzDUVh2T
log for SMB : 2023/12/27 11:37:29 DEBUG : rclone: Version "v1.65.0" starting with parameters [ - Pastebin.com

[kapitainsky]

@w1q06y3t

--transfers 40

Remove this flag - default is 4 and I am not sure whether 40 would work well for sftp/smb on fibre 1GB connection. and you are on something much slower and for sure with much higher latency. Given low quality of your Internet I would even try lower values:

--checkers 4 --transfers 2 or --checkers 2 --transfers 1

As you are transferring files between two machines you control I would suggest to use program like syncthing. You will get much better experience over low quality connections than with rclone. The key difference is that rclone is not able to resume interrupted transfers - which is disastrous for large files and not 100% stable connection. syncthing sends files in chunks and only missing/corrupted chunk has to be retried.

[w1q06y3t]

Hello.

Thank you for your response. There was no significant effect of reducing the number of concurrent transfers when transferring SFTP files via rclone.

I'm going to try the program you recommended instead, as it's important for me to synchronize files securely and without loss at this point. However, in the long run, I think it would be good to have the SFTP and SMB errors fixed, as I don't think this is the intended behavior of the program, and not all networks using rclone are fast and reliable, so it would be more beneficial for users if this was fixed.