I need help, Is this a bug?

[PowerWordTree]

Is this a bug?

E:>md aaa
E:>echo x > aaa\中文：中文.txt
E:>E:\Tool\Network\Rclone\rclone.exe mount aaa bbb

E:>dir /b aaa
中文：中文.txt
E:>dir /b bbb
中文中文.txt

[kapitainsky]

could you please test the same as below - not from mount but from some folder on your local disk:

md test
cd test
echo x > 中文：中文.txt
rclone.exe copy .  remote:test
rclone lsf remote:test

[kapitainsky]

In addition please install the latest WinFSP - you have to uninstall your current version first.

[kapitainsky]

Ok I think it is a bug IMO. Here how to reproduce it:

Please note that ？： are two characters:

EFBC9F ？ Ideographic question mark
EFBC9A ： Ideographic colon

Now in empty folder:

> echo a > test？：test.txt
> rclone copy . onedrive:test
> rclone lsf onedrive:test
test?:test.txt

lsf normalised ？：t to ?:

In another cmd window:

>rclone mount onedrive:test mountpoint
>dir /b mountpoint
testtest.txt

result for ?: are some gibberish characters (EF80BF EF80BA)

When I copy test？：test.txt test file to mount directly results are as follows:

rclone lsf onedrive:test
test?:test.txt

>dir /b mountpoint
test？：test.txt

lsf normalises ？：t to ?: but at the mount point I can see original correct characters.

The same issue has been reported also here:

Filename changed if using copy command

What is the problem you are having with rclone? Filename changed if using copy command. For example, I have one file named "test？：test.txt", if I use rclone copy command to copy to dropbox, the filename became "testtest.txt". In hex editor I found that "？：" is originally "ef bc 9f ef bc 9a", but became "ef 80 bf ef 80 ba" after copy to dropbox. This problem did not happen if I mounted and manually copy to dropbox using explorer. Run the command 'rclone version' and share the full output …

@ncw I think it is another case where VFS normalisation would help.

I used onedrive remote in my example but it behaves the same with other remotes.

It seems all works fine on Linux:

$ echo a > test？：test.txt
$ ls
test？：test.txt
$ rclone copy . onedrive:test
$ rclone lsf onedrive:test
test？：test.txt

lsf does not normalise anything

and all is ok on mount:

$ ls mountpoint
test？：test.txt

[ncw (Nick Craig-Wood)]

What is happening here is that the ： is one of the characters rclone uses to escape literal : on windows.

Unfortunately that means it misbehaves in file names. There is an issue (probably many!) about this.

A workaround is to disable the interpretation of those special characters, so first fine out what the encoding is

rclone help flags local-encoding

Then remove the characters you don't want encoded with, say Colon and Question

  --local-encoding "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

This will then mean you can sync files from drive (say) which have a : and ? in them as rclone won't be able to translate them to something you can store on a Windows file system.

There are encoding settings for most backends.

[kapitainsky]

Ok. This rclone "user friendliness" comes at the cost of people who actually use characters used for substitutions. Looks like ？： are quite common in Asia and are perfectly valid Windows characters.

[kapitainsky]

But something is not clear for me:

I do understand that ? in filename on the remote is substituted with ？ when sent to Windows

But why to do it other way around? and substitute valid windows character with invalid one when sending to remote?

> echo a > test？：test.txt
> rclone copy . onedrive:test
> rclone lsf onedrive:test
test?:test.txt

and why on mount ? is corrupted to . It should be ？ isn't it?

And second part of my test - why it all works when using mount only? Does VFS use different logic?

@kapitainsky

When I copy test？：test.txt test file to mount directly results are as follows:

rclone lsf onedrive:test
test?:test.txt

>dir /b mountpoint
test？：test.txt

[ncw (Nick Craig-Wood)]

@kapitainsky

This rclone "user friendliness" comes at the cost of people who actually use characters used for substitutions. Looks like ？： are quite common in Asia and are perfectly valid Windows characters.

Yes, I did not realise this when I made the substitutions. I would have chosen different ones if I had

@kapitainsky

I do understand that ? in filename on the remote is substituted with ？ when sent to Windows

But why to do it other way around? and substitute valid windows character with invalid one when sending to remote?

This is to enable repeated syncing. Rclone needs to be able to match each file at the source with the destination so the matching has to be two way.