Filename changed if using copy command

Suspected Bug
1

What is the problem you are having with rclone?

Filename changed if using copy command.
For example, I have one file named "test？：test.txt", if I use rclone copy command to copy to dropbox, the filename became "testtest.txt".
In hex editor I found that "？：" is originally "ef bc 9f ef bc 9a", but became "ef 80 bf ef 80 ba" after copy to dropbox.
This problem did not happen if I mounted and manually copy to dropbox using explorer.

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: Microsoft Windows 10 IoT Enterprise LTSC 2021 21H2 (64 bit)
  • os/kernel: 10.0.19044.3086 Build 19044.3086.3086 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "Z:\folder" "dropboxe:/"

The rclone config contents with secrets removed.

[frost]
type = drive
scope = drive
team_drive = 

[dropbox]
type = dropbox

[dropboxe]
type = crypt
remote = dropbox:enc
filename_encryption = standard
filename_encoding = base32768

A log from the command with the -vv flag

2023/06/18 23:06:03 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "copy" "Z:\\folder" "dropboxe:/" "-vv"]
2023/06/18 23:06:03 DEBUG : Creating backend with remote "Z:\\folder"
2023/06/18 23:06:03 DEBUG : Using config file from "C:\\Users\\a0193143\\AppData\\Roaming\\rclone\\rclone.conf"
2023/06/18 23:06:03 DEBUG : fs cache: renaming cache item "Z:\\folder" to be canonical "//?/Z:/folder"
2023/06/18 23:06:03 DEBUG : Creating backend with remote "dropboxe:/"
2023/06/18 23:06:03 DEBUG : Creating backend with remote "dropbox:enc"
2023/06/18 23:06:04 DEBUG : Encrypted drive 'dropboxe:/': Waiting for checks to finish
2023/06/18 23:06:04 DEBUG : Encrypted drive 'dropboxe:/': Waiting for transfers to finish
2023/06/18 23:06:04 DEBUG : 琾䦮錏猒㬄㪛焾嚙✟: Uploading chunk 1/1
2023/06/18 23:06:05 DEBUG : 琾䦮錏猒㬄㪛焾嚙✟: Uploading chunk 2/1
2023/06/18 23:06:06 DEBUG : Dropbox root 'enc': Adding "/enc/琾䦮錏猒㬄㪛焾嚙✟" to batch
2023/06/18 23:06:06 DEBUG : Dropbox root 'enc': Batch idle for 500ms so committing
2023/06/18 23:06:06 DEBUG : Dropbox root 'enc': Committing sync batch length 1 starting with: /enc/琾䦮錏猒㬄㪛焾嚙✟
2023/06/18 23:06:07 DEBUG : Dropbox root 'enc': Committed sync batch length 1 starting with: /enc/琾䦮錏猒㬄㪛焾嚙✟
2023/06/18 23:06:07 DEBUG : test?:test.txt: dropbox = 518d7eaaa8e82cda40e4a07948a3f2ee93d1d4244a0e839744a1337d2e9a9c0f OK
2023/06/18 23:06:07 INFO  : test?:test.txt: Copied (new)
2023/06/18 23:06:07 INFO  :
Transferred:             32 B / 32 B, 100%, 10 B/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         4.5s

2023/06/18 23:06:07 DEBUG : 9 go routines active
2023/06/18 23:06:07 INFO  : Dropbox root 'enc': Committing uploads - please wait...
2

Invalid UTF-8 bytes will also be replaced, as they can't be used in JSON strings.

Might be related to above.

echo "? : " | hexdump -C
00000000  3f 20 3a 20 0a                                    |? : .|
00000005

but you have:

echo "？：" | hexdump -C
00000000  ef bc 9f ef bc 9a 0a                              |.......|
00000007

Your names use something called NFD (decomposed) normalisation - you see your string is 4 characters but ends up as 6 code points.

yes they both look the same.... but they are not and can't be used in JSON.

As explorer probably normalized your name. Could you check if it is still ef bc 9f ef bc 9a on Dropbox?

Characters normalisation is still ongoing issue in 2023... in all computer industry. Sometimes things seem to work but when become more complex the break.

3

Yep, it is still ef bc 9f ef bc 9a on Dropbox if I use explorer copy to.

4

Thanks you for checking.

5

To make it sure - you use rclone mount and explorer? Or you have some dropbox client?

6

Also note for others reading - any dropbox limitations are irrelevant here as files are stored in crypt. So my first post was clear shot at the fence.

Normalisation problems remain but for good we do not have to worry about any real remote limitations.

7

Yes, rclone mount and explorer.

And I also tested on Arch Linux (rclone 1.16.2), no problem at all, so this issue might be Windows specified.

1 Like
8

So I dissected it:

your original string "？：" (hex: EF BC 9F EF BC 9A) is composed of two valid UTF8 characters - they are both part of "East Asian Punctuation Marks" category

EFBC9F ？ Ideographic question mark
EFBC9A ： Ideographic colon

`rclone copy' saves them as hex: EF 80 BF EF 80 BA - also two valid UTF8 characters:

EF80BF:
image

EF80BA:
image

Do these two ideograms make any sense in relation to `"？：" (lets ignore for the moment that dropbox cant display them)

9

No, these two ideograms are just random kanji in my opinion, I've never seen before.

10

both

？ EFBC9F and ： EFBC9A

are part of Unicode since time immemorial

so indeed we have some problem with rclone copy characters conversion/normalisation program chain.

11

BIG question is why mount works as it should - BTW we have even more issues on macOS e.g. this thread

Maybe we can fix all of this in one go:)

1 Like
12

I have tried to reproduce it myself with crypt and dropbox - unfortunately I can not see this problem. I am on macOS so things can be different.

Could you please show step by step how to reproduce it on Windows? You say "copy to dropbox". But do you mean copy to crypt hosted by dropbox?
And where you see "testtest.txt"? In mount or by running rclone lsf for example?

13

Sure, I recorded the steps I use.
https://www.youtube.com/watch?v=KwdkMw_lVWw

And yes, copy to crypt hosted by dropbox, but uncrypted is same result in explorer.
I see "testtest.txt" in explorer, and under rclone lsf is "test?:test.txt", which original "？！" became "?!" .
I've also noticed if rclone copy to not encrypted dropbox, "？！" became "?!" too.

1 Like
14

  1. what is HEX representation when you copy this file name from rclone lsf ?

  2. Are you using the latest WinFsp ? - they did a lot of work to improve characters handling lately

15
  1. The hex representation is 74 65 73 74 **3f ef bc 81** 74 65 73 74
  2. I tried the lastest WinFsp, the filename represent in explorer did change, the two character is now ef 80 bf ef bc 81, and I found that the WinFsp version I previously used was 2021 (1.9 I guess).
16

There is workaround for this issue:

rclone copy "Z:\folder" "dropboxe:/" --local-encoding "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

17

I have same problem with onedrive business
If the filename have this character ？, it will be `？ when i try rclone copy that file

18

This is how rclone works.

Here explanation:

When on Windows and you want "？" and "：" to work you have to add to rclone commands:

--local-encoding "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

you can also use:

--local-encoding None

read link I sent.

19

I mean, filename with fullwidth question mark like this (？), replaced to fullwidth question mark too but with this character (`), so become like this (`？)
So i want filename same like original filename when i use rclone copy to upload to my drive.
But, thank you for replying me!

20

I guess you are on Windows which is the most problematic:)

Try with --local-encoding None - it then will use 1 to 1 windows characters without trying to be "clever" and replacing characters.