Substituted CJK punctuation at filenames in console logs & crypt input

Help and Support
1

What is the problem you are having with rclone?

Some file name characters have been accidentally converted in console and crypt's filename sources.
Like the "test：？.txt" become to "test:?.txt" in rclone console output, and the "test.txt" in opened crypt fs.
So, the console and external tools (e.g. GitHub - Alkl58/RcloneBrowser: Simple cross platform GUI for rclone. Supports macOS, GNU/Linux, BSD family and Windows.) show incorrect results for listing, and some characters in the filename are lost in crypt fs.

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: Microsoft Windows 10 Pro 22H2 (64 bit)
  • os/kernel: 10.0.19045.3636 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

OneDrive Enterprise or Local of rclone.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

New file "D:\test\test：？.txt", then.

rclone.exe copy D:/test local:i:/ -vv

The rclone config contents with secrets removed.

Editing existing "local" remote with options:
- type: local

Option nounc.

A log from the command with the -vv flag

2023/10/30 09:38:59 DEBUG : rclone: Version "v1.64.2" starting with parameters ["C:\\Softwares\\bin\\rclone.exe" "copy" "D:/test" "local:i:/" "-vv"]
2023/10/30 09:38:59 DEBUG : Creating backend with remote "D:/test"
2023/10/30 09:38:59 DEBUG : Using RCLONE_CONFIG_PASS password.
2023/10/30 09:38:59 DEBUG : Using config file from "C:\\Users\\user\\AppData\\Roaming\\rclone\\rclone.conf"
2023/10/30 09:38:59 DEBUG : fs cache: renaming cache item "D:/test" to be canonical "//?/D:/test"
2023/10/30 09:38:59 DEBUG : Creating backend with remote "local:i:/"
2023/10/30 09:38:59 DEBUG : fs cache: renaming cache item "local:i:/" to be canonical "//?/i:/"
2023/10/30 09:38:59 DEBUG : test:?.txt: Need to transfer - File not found at Destination
2023/10/30 09:38:59 DEBUG : Local file system at //?/i:/: Waiting for checks to finish
2023/10/30 09:38:59 DEBUG : Local file system at //?/i:/: Waiting for transfers to finish
2023/10/30 09:38:59 DEBUG : test:?.txt: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2023/10/30 09:38:59 DEBUG : test:?.txt.yaxodok2.partial: renamed to: test:?.txt
2023/10/30 09:38:59 INFO  : test:?.txt: Copied (new)
2023/10/30 09:38:59 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s

2023/10/30 09:38:59 DEBUG : 5 go routines active
2

This is not a bug but expected behaviour:

3

I see the Overview of cloud storage systems now.
I'm confused about this, since the "Colon,Question" in --local-encoding & --onedrive-encoding both, it seems that no conversion is needed, and I see no conversion happening for local NTFS / OneDrive Web targets, only conversion for console output. Additionally, the converted characters are bad in decrypted cryfs.
Fullwidth characters ？ and ： is a bit common in Chinese file names.

4

Yes indeed - so it was not the best decision to use them for encoding - but now we have to live with it.

In order to disable full width characters ？ and ：encoding on local filesystem you have to add below flag to your commands:

--local-encoding "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

and for onedrive:

--onedrive-encoding "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot"

In general you can see all backends default encodings by running:

rclone help flags encoding

To avoid rclone mingling with ？ and ：remove Colon and Question

If this is something you always use add these flags to your rclone.config:

type = local
encoding = "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"

[onedrive]
type = onedrive
client_id = XXX
client_secret = XXX
drive_id = XXX
drive_type = personal
token = XXX
encoding = "Slash,LtGt,DoubleQuote,Asterisk,Pipe,BackSlash,Del,Ctl,LeftSpace,LeftTilde,RightSpace,RightPeriod,InvalidUtf8,Dot"