SSH_FX_FAILURE when copying file with nonstandard characters to sftp remote with ntfs drive

mcbnr · October 26, 2023, 5:23pm

What is the problem you are having with rclone?

rclone copy fails when using sftp to transfer to a remote with an attached ntfs drive and the filename has nonstandard characters, e.g 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ. The file can be created with this name on the remote, and sftp put successfully transfers it.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2
- os/version: Microsoft Windows 10 Enterprise LTSC 2021 21H2 (64 bit)
- os/kernel: 10.0.19044.3570 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

sftp server with attached ntfs storage

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone copy t remote:t

The folder t contains the file with nonstandard characters

The rclone config contents with secrets removed.

type = sftp
host = 192.168.1.16
user = odroid
shell_type = unix

A log from the command with the `-vv` flag

2023/10/26 19:10:38 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "copy" "-vv" "--retries" "1" "t" "odroid:t"]
2023/10/26 19:10:38 DEBUG : Creating backend with remote "t"
2023/10/26 19:10:38 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2023/10/26 19:10:38 DEBUG : fs cache: renaming cache item "t" to be canonical "//?/C:/Users/user/Desktop/New folder/t"
2023/10/26 19:10:38 DEBUG : Creating backend with remote "odroid:t"
2023/10/26 19:10:38 DEBUG : Creating backend with remote "odroid_home:/mnt/storage/t"
2023/10/26 19:10:38 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: New connection 192.168.1.26:16009->192.168.1.16:22 to "SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4"
2023/10/26 19:10:39 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: Shell type "unix" from config
2023/10/26 19:10:39 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: Using root directory "/mnt/storage/t"
2023/10/26 19:10:39 DEBUG : fs cache: renaming cache item "odroid:t" to be canonical "odroid_home:/mnt/storage/t"
2023/10/26 19:10:39 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮   ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/26 19:10:39 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: Waiting for checks to finish
2023/10/26 19:10:39 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: Waiting for transfers to finish
2023/10/26 19:10:39 ERROR : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮   ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/10/26 19:10:39 INFO  : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮   ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.jufecut9.partial: Failed to remove failed partial copy: stat failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/10/26 19:10:39 ERROR : Attempt 1/1 failed with 1 errors and: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/10/26 19:10:39 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.7s

2023/10/26 19:10:39 DEBUG : 14 go routines active
2023/10/26 19:10:39 DEBUG : sftp://odroid@192.168.1.16:22//mnt/storage/t: Closing 1 unused connections
2023/10/26 19:10:39 Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)

asdffdsa · October 26, 2023, 5:31pm

the D:\files\char is NTFS

rclone copy D:\files\char hetznersbox01_sftp:char -vv --retries=1 --inplace
2023/10/26 13:29:45 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "copy" "D:\\files\\char" "hetznersbox01_sftp:char" "-vv" "--retries=1" "--inplace"]
2023/10/26 13:29:45 DEBUG : Creating backend with remote "D:\\files\\char"
2023/10/26 13:29:45 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2023/10/26 13:29:45 DEBUG : fs cache: renaming cache item "D:\\files\\char" to be canonical "//?/D:/files/char"
2023/10/26 13:29:45 DEBUG : Creating backend with remote "hetznersbox01_sftp:char"
2023/10/26 13:29:46 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:51595->65.108.23.238:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/26 13:29:46 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell type "unix" from config
2023/10/26 13:29:46 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Relative path resolved to "/home/char"
2023/10/26 13:29:46 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Using root directory "/home/char"
2023/10/26 13:29:47 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/26 13:29:47 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for checks to finish
2023/10/26 13:29:47 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for transfers to finish
2023/10/26 13:29:47 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ"
2023/10/26 13:29:48 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/26 13:29:48 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Remote command result: f49df88cc44eb718c2c3a868a6dcee3b /home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/26 13:29:48 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Parsed hash: f49df88cc44eb718c2c3a868a6dcee3b
2023/10/26 13:29:48 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: md5 = f49df88cc44eb718c2c3a868a6dcee3b OK
2023/10/26 13:29:48 INFO  : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Copied (new)
2023/10/26 13:29:48 INFO  :
Transferred:        1.361 KiB / 1.361 KiB, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         2.8s

2023/10/26 13:29:48 DEBUG : 12 go routines active
2023/10/26 13:29:48 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Closing 1 unused connections

c:\data\rclone>rclone ls hetznersbox01_sftp:char
     1394 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ

kapitainsky · October 26, 2023, 5:38pm

Might be related to transfer to a temporary file approach introduced in rclone v.1.63.0 when temp file prefix can lead to too long file names. Names like in your example can have surprising length in bytes given that some Unicode character require multi bytes encoding and .jufecut9.partial prefix trip it over the top. Or you have no permission to delete/rename files...

Could you try with --inplace flag in your copy command?

mcbnr · October 26, 2023, 5:41pm

Yes, --inplace does it, thanks. What are the trade-offs in using this?

kapitainsky · October 26, 2023, 5:46pm

As per --inplace doc:

This prevents other users of the backend from seeing partially uploaded files in their new names and prevents overwriting the old file until the new one is completely uploaded.

I disable this functionality exactly for the reason you have noticed. I have some files which do not fit with temp prefixes.

asdffdsa · October 26, 2023, 5:53pm

yes, that is in the example i posted

mcbnr · October 26, 2023, 5:53pm

Hopefully this flag can be applied automatically depending on the file in future versions.

kapitainsky · October 26, 2023, 5:55pm

Indeed - I have not noticed this flag. I thought that you only proved that OP file name works with rclone. At the end it depends on sftp server limitations.

kapitainsky · October 26, 2023, 6:00pm

Other interesting thing here is that as per flag docs:

(rclone will make sure the final name is no longer than 100 characters by truncating the original-file-name part if necessary).

IMO there are two scenarios here:

rclone does not count name length in bytes but in characters - which can make 100 chars as big as 400 bytes (some chars need 4 bytes encoding)
the root cause is not length problem - but delete/rename permissions.

ncw · October 28, 2023, 3:41pm

Rclone counts file name length in bytes, however it isn't taking care not to split a UTF-8 character (which can be multiple bytes). I suspect this is causing an invalid character which is causing the destination to reject the invalid unicode.

@mcbnr can you test this theory please?

If you try transferring these files (without --inplace) I would expect one of them to work if my theory is correct!

𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ

ncw · October 28, 2023, 4:14pm

I had a go at fixing this here - can you give it a try?

v1.65.0-beta.7464.e4b510981.fix-refactor-copy on branch fix-refactor-copy (uploaded in 15-30 mins)

asdffdsa · October 28, 2023, 4:51pm

rclone.exe copy d:\files\char hetznersbox01_sftp:char -vv --retries=1 
2023/10/28 12:48:39 DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
2023/10/28 12:48:39 DEBUG : rclone: Version "v1.64.2" starting with parameters ["c:\\data\\rclone\\rclone.exe" "copy" "d:\\files\\char" "hetznersbox01_sftp:char" "-vv" "--retries=1"]
2023/10/28 12:48:39 DEBUG : Creating backend with remote "d:\\files\\char"
2023/10/28 12:48:39 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2023/10/28 12:48:39 DEBUG : fs cache: renaming cache item "d:\\files\\char" to be canonical "//?/d:/files/char"
2023/10/28 12:48:39 DEBUG : Creating backend with remote "hetznersbox01_sftp:char"
2023/10/28 12:48:40 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55703->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:40 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell type "unix" from config
2023/10/28 12:48:40 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Relative path resolved to "/home/char"
2023/10/28 12:48:40 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Using root directory "/home/char"
2023/10/28 12:48:41 DEBUG : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:41 DEBUG : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:41 DEBUG : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:41 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:41 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for checks to finish
2023/10/28 12:48:41 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for transfers to finish
2023/10/28 12:48:41 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ\xcb.powijer9.partial"
2023/10/28 12:48:41 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/aa\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱ�.powijer9.partial
2023/10/28 12:48:41 ERROR : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.powijer9.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5 -r /home/char/aa\\𝓝\\𝓸\\𝓫\\𝓸\\𝓭\\𝔂\\ \\𝓲\\𝓼\\ \\𝓱\\𝓸\\𝓶\\𝓮\\ ᴬ\\ ⱽⁱˢⁱᵗ\\ ᶠʳᵒᵐ\\ ᵗʰᵉ\\ ⱽⁱ\xcb.powijer9.partial": md5: /home/char/aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.powijer9.partial: No such file or directory: Process exited with status 1
2023/10/28 12:48:41 ERROR : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.powijer9.partial: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""
2023/10/28 12:48:41 INFO  : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.powijer9.partial: Removing failed copy
2023/10/28 12:48:42 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55704->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:43 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ\xcb.yilisow5.partial"
2023/10/28 12:48:43 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ\xcb.zetaxec2.partial"
2023/10/28 12:48:43 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱ�.yilisow5.partial
2023/10/28 12:48:43 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/aaa\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱ�.zetaxec2.partial
2023/10/28 12:48:43 ERROR : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.yilisow5.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5 -r /home/char/\\𝓝\\𝓸\\𝓫\\𝓸\\𝓭\\𝔂\\ \\𝓲\\𝓼\\ \\𝓱\\𝓸\\𝓶\\𝓮\\ ᴬ\\ ⱽⁱˢⁱᵗ\\ ᶠʳᵒᵐ\\ ᵗʰᵉ\\ ⱽⁱ\xcb.yilisow5.partial": md5: /home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.yilisow5.partial: No such file or directory: Process exited with status 1
2023/10/28 12:48:43 ERROR : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.yilisow5.partial: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""
2023/10/28 12:48:43 INFO  : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.yilisow5.partial: Removing failed copy
2023/10/28 12:48:43 ERROR : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.zetaxec2.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5 -r /home/char/aaa\\𝓝\\𝓸\\𝓫\\𝓸\\𝓭\\𝔂\\ \\𝓲\\𝓼\\ \\𝓱\\𝓸\\𝓶\\𝓮\\ ᴬ\\ ⱽⁱˢⁱᵗ\\ ᶠʳᵒᵐ\\ ᵗʰᵉ\\ ⱽⁱ\xcb.zetaxec2.partial": md5: /home/char/aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.zetaxec2.partial: No such file or directory: Process exited with status 1
2023/10/28 12:48:43 ERROR : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.zetaxec2.partial: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""
2023/10/28 12:48:43 INFO  : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.zetaxec2.partial: Removing failed copy
2023/10/28 12:48:43 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55705->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:44 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ\xcb.pahaqas4.partial"
2023/10/28 12:48:44 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/a\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱ�.pahaqas4.partial
2023/10/28 12:48:44 ERROR : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.pahaqas4.partial: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5 -r /home/char/a\\𝓝\\𝓸\\𝓫\\𝓸\\𝓭\\𝔂\\ \\𝓲\\𝓼\\ \\𝓱\\𝓸\\𝓶\\𝓮\\ ᴬ\\ ⱽⁱˢⁱᵗ\\ ᶠʳᵒᵐ\\ ᵗʰᵉ\\ ⱽⁱ\xcb.pahaqas4.partial": md5: /home/char/a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.pahaqas4.partial: No such file or directory: Process exited with status 1
2023/10/28 12:48:44 ERROR : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.pahaqas4.partial: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""
2023/10/28 12:48:44 INFO  : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱ�.pahaqas4.partial: Removing failed copy
2023/10/28 12:48:44 ERROR : Attempt 1/1 failed with 8 errors and: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""
2023/10/28 12:48:44 INFO  : 
Transferred:   	    5.148 KiB / 5.148 KiB, 100%, 1.287 KiB/s, ETA 0s
Errors:                 8 (retrying may help)
Elapsed time:         5.5s

2023/10/28 12:48:44 DEBUG : 32 go routines active
2023/10/28 12:48:44 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Closing 3 unused connections
2023/10/28 12:48:45 Failed to copy with 8 errors: last error was: corrupted on transfer: md5 hash differ "b62709f2cfd06c9916ed7860707edca2" vs ""

rclone.exe copy d:\files\char hetznersbox01_sftp:char -vv --retries=1 
2023/10/28 12:48:51 DEBUG : Setting --config "C:\\data\\rclone\\rclone.conf" from environment variable RCLONE_CONFIG="C:\\data\\rclone\\rclone.conf"
2023/10/28 12:48:51 DEBUG : rclone: Version "v1.65.0-beta.7464.e4b510981.fix-refactor-copy" starting with parameters ["d:\\data\\rclone\\exe\\windows\\rclone-v1.65.0-beta.7464.e4b510981.fix-refactor-copy-windows-amd64\\rclone.exe" "copy" "d:\\files\\char" "hetznersbox01_sftp:char" "-vv" "--retries=1"]
2023/10/28 12:48:51 DEBUG : Creating backend with remote "d:\\files\\char"
2023/10/28 12:48:51 DEBUG : Using config file from "C:\\data\\rclone\\rclone.conf"
2023/10/28 12:48:51 DEBUG : fs cache: renaming cache item "d:\\files\\char" to be canonical "//?/d:/files/char"
2023/10/28 12:48:51 DEBUG : Creating backend with remote "hetznersbox01_sftp:char"
2023/10/28 12:48:52 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55708->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:52 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell type "unix" from config
2023/10/28 12:48:52 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Relative path resolved to "/home/char"
2023/10/28 12:48:52 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Using root directory "/home/char"
2023/10/28 12:48:53 DEBUG : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:53 DEBUG : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:53 DEBUG : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:53 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Need to transfer - File not found at Destination
2023/10/28 12:48:53 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for checks to finish
2023/10/28 12:48:53 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Waiting for transfers to finish
2023/10/28 12:48:53 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.zedober6.partial"
2023/10/28 12:48:53 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱˢⁱᵒⁿᵃʳʸ.zedober6.partial
2023/10/28 12:48:54 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Remote command result: b62709f2cfd06c9916ed7860707edca2 /home/char/𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.zedober6.partial
2023/10/28 12:48:54 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.zedober6.partial: Parsed hash: b62709f2cfd06c9916ed7860707edca2
2023/10/28 12:48:54 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: md5 = b62709f2cfd06c9916ed7860707edca2 OK
2023/10/28 12:48:54 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55709->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:54 DEBUG : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.zedober6.partial: renamed to: 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/28 12:48:54 INFO  : 𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Copied (new)
2023/10/28 12:48:55 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.jowiqub7.partial"
2023/10/28 12:48:55 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/a\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱˢⁱᵒⁿᵃʳʸ.jowiqub7.partial
2023/10/28 12:48:55 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Remote command result: b62709f2cfd06c9916ed7860707edca2 /home/char/a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.jowiqub7.partial
2023/10/28 12:48:55 DEBUG : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.jowiqub7.partial: Parsed hash: b62709f2cfd06c9916ed7860707edca2
2023/10/28 12:48:55 DEBUG : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: md5 = b62709f2cfd06c9916ed7860707edca2 OK
2023/10/28 12:48:55 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: New connection 192.168.62.234:55710->xxx.xxx.xxx.xxx:23 to "SSH-2.0-OpenSSH_9.3 FreeBSD-20230719"
2023/10/28 12:48:56 DEBUG : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.jowiqub7.partial: renamed to: a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/28 12:48:56 INFO  : a𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Copied (new)
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.peqaniq3.partial"
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/aaa\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱˢⁱᵒⁿᵃʳʸ.peqaniq3.partial
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Shell path "/home/char/aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.govomiw2.partial"
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Remote command result: b62709f2cfd06c9916ed7860707edca2 /home/char/aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.peqaniq3.partial
2023/10/28 12:48:56 DEBUG : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.peqaniq3.partial: Parsed hash: b62709f2cfd06c9916ed7860707edca2
2023/10/28 12:48:56 DEBUG : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: md5 = b62709f2cfd06c9916ed7860707edca2 OK
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Running remote command: md5 -r /home/char/aa\𝓝\𝓸\𝓫\𝓸\𝓭\𝔂\ \𝓲\𝓼\ \𝓱\𝓸\𝓶\𝓮\ ᴬ\ ⱽⁱˢⁱᵗ\ ᶠʳᵒᵐ\ ᵗʰᵉ\ ⱽⁱˢⁱᵒⁿᵃʳʸ.govomiw2.partial
2023/10/28 12:48:56 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Remote command result: b62709f2cfd06c9916ed7860707edca2 /home/char/aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.govomiw2.partial
2023/10/28 12:48:56 DEBUG : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.govomiw2.partial: Parsed hash: b62709f2cfd06c9916ed7860707edca2
2023/10/28 12:48:56 DEBUG : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: md5 = b62709f2cfd06c9916ed7860707edca2 OK
2023/10/28 12:48:57 DEBUG : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.peqaniq3.partial: renamed to: aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/28 12:48:57 INFO  : aaa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Copied (new)
2023/10/28 12:48:57 DEBUG : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ.govomiw2.partial: renamed to: aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ
2023/10/28 12:48:57 INFO  : aa𝓝𝓸𝓫𝓸𝓭𝔂 𝓲𝓼 𝓱𝓸𝓶𝓮 ᴬ ⱽⁱˢⁱᵗ ᶠʳᵒᵐ ᵗʰᵉ ⱽⁱˢⁱᵒⁿᵃʳʸ: Copied (new)
2023/10/28 12:48:57 INFO  : 
Transferred:   	    5.148 KiB / 5.148 KiB, 100%, 1.295 KiB/s, ETA 0s
Transferred:            4 / 4, 100%
Elapsed time:         6.1s

2023/10/28 12:48:57 DEBUG : 32 go routines active
2023/10/28 12:48:57 DEBUG : sftp://xxx@xxx.your-storagebox.de:23/char: Closing 3 unused connections

mcbnr · October 28, 2023, 5:36pm

this version works no problem for me

ncw · October 29, 2023, 2:05pm

Thanks for testing @mcbnr that looks like it is now working. I tried @asdffdsa test with the new binary and that works too

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.65