Rclone mount fails to interact with file with not allowed characters in the filename

Suspected Bug
1

What is the problem you are having with rclone?

If you name a file on gdrive with characters that are not allowed in NTFS filenames, a rclone mount of the cloud drive with vfs caching set to full will fail to interact in any way other than listing the file in the directory due to vfs being unable to create the backing file in the cache

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0
  • os/version: Microsoft Windows 10 Pro 2009 (64 bit)
  • os/kernel: 10.0.22000.675 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount "tknkstash:" V: -vvvvvvvv --vfs-cache-mode full --vfs-read-chunk-size 200M --vfs-cache-max-age 30m --vfs-cache-poll-interval 35m --volname 'The Archive' --attr-timeout 1h --drive-skip-gdocs --drive-skip-shortcuts --dir-cache-time 5h --cache-dir 'E:\rclone' --rc --rc-web-gui --rc-user=admin --rc-pass=admin --transfers 2 --order-by 'size,ascending'

The rclone config contents with secrets removed.

[tknkstash]
type = drive
team_drive = 0AOQu6EMCjp1PUk9PVA
root_folder_id =

A log from the command with the -vv flag

https://gist.github.com/Masamune3210/32c31567d7c1a7296988f029200b616b
2

Hmm, this problem seems familiar...

I think this is probably the same issue: Cannot copy files to VFS-mounted directory with `:` in the name, or subdirectories thereof, from outside the mount. · Issue #5131 · rclone/rclone · GitHub

3

Sounds like it might be the same root issue, yeah. VFS not translating the path or file name in any way causing NTFS to refuse the path creation due to the illegal characters. Could the same translation that rclone does for uploading files to a cloud drive with illegal characters be applied to VFS possibly?

4

Presumably the files don't work if you turn VFS caching off either?

I think there are two problems here

  1. the vfs cache needs to translate characters
  2. rclone mount needs to translate characters

The entire VFS should be doing translations for rclone mount to make the file names legal. However something like rclone serve webdav might or might not need the transations depending on exactly where you are using it. So the VFS probably needs an optional file name translation to translate files into that used by the local backend. This would need to be forced on for rclone mount.

However the VFS mount caching always needs filename translation.

5

The files copy correctly if vfs caching is disabled, so it is seemingly just the fact that the VFS cant write to the cache file that is stopping the operation with it enabled

6

I don't understand this..

If the files can't be written to the NTFS disk (in the VFS cache) then how can you copy them to an NTFS disk? I'm probably mis-understanding something!

7

I'll test it more tonight, didn't have much time last night, but starting a mount with caching disabled seemingly allowed the copy to start

8

So, did some more digging. Seemingly non-vfs is translating the name on file operation, wide colons are fine as they are permitted in NTFS filenames, but standard colons are translated into a dot. This one to be precise,  , if you can even see it as it just shows up as the unicode char not found box in Chrome for me

9

Unicode codepoint f03a is what the normal colon gets translated to, which just appears as a dot, while Unicode codepoint ff1a, which is the wide colon is passed directly through

2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /: Releasedir: fh=0x0
2022/06/03 18:23:01 DEBUG : /: >Releasedir: errc=0
2022/06/03 18:23:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /: Opendir:
2022/06/03 18:23:01 DEBUG : /: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/06/03 18:23:01 DEBUG : /: >OpenFile: fd=/ (r), err=
2022/06/03 18:23:01 DEBUG : /: >Opendir: errc=0, fh=0x0
2022/06/03 18:23:01 DEBUG : /: Releasedir: fh=0x0
2022/06/03 18:23:01 DEBUG : /: >Releasedir: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: OpenEx: flags=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/06/03 18:23:01 DEBUG : ：.nsz: Open: flags=O_RDONLY
2022/06/03 18:23:01 DEBUG : ：.nsz: >Open: fd=：.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /：.nsz: >OpenFile: fd=：.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /：.nsz: >OpenEx: errc=0, fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Flush: fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: >Flush: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Release: fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: >Release: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: OpenEx: flags=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/06/03 18:23:01 DEBUG : ：.nsz: Open: flags=O_RDONLY
2022/06/03 18:23:01 DEBUG : ：.nsz: >Open: fd=：.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /：.nsz: >OpenFile: fd=：.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /：.nsz: >OpenEx: errc=0, fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: Getattr: fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Flush: fh=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: OpenEx: flags=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/06/03 18:23:01 DEBUG : :.nsz: Open: flags=O_RDONLY
2022/06/03 18:23:01 DEBUG : :.nsz: >Open: fd=:.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /:.nsz: >OpenFile: fd=:.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /:.nsz: >OpenEx: errc=0, fh=0x1
2022/06/03 18:23:01 DEBUG : /：.nsz: >Flush: errc=0
2022/06/03 18:23:01 DEBUG : /：.nsz: Release: fh=0x0
2022/06/03 18:23:01 DEBUG : /：.nsz: >Release: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0x1
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Flush: fh=0x1
2022/06/03 18:23:01 DEBUG : /:.nsz: >Flush: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Release: fh=0x1
2022/06/03 18:23:01 DEBUG : /:.nsz: >Release: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: OpenEx: flags=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: OpenFile: flags=O_RDONLY, perm=-rwxrwxrwx
2022/06/03 18:23:01 DEBUG : :.nsz: Open: flags=O_RDONLY
2022/06/03 18:23:01 DEBUG : :.nsz: >Open: fd=:.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /:.nsz: >OpenFile: fd=:.nsz (r), err=
2022/06/03 18:23:01 DEBUG : /:.nsz: >OpenEx: errc=0, fh=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: Getattr: fh=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Flush: fh=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: >Flush: errc=0
2022/06/03 18:23:01 DEBUG : /:.nsz: Release: fh=0x0
2022/06/03 18:23:01 DEBUG : /:.nsz: >Release: errc=0
2022/06/03 18:23:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2022/06/03 18:23:01 DEBUG : /: >Getattr: errc=0
2022/06/03 18:23:01 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF

Thats all the references in the log to the files I used to test

10

Ah, I see, that is WinFSP doing the name translation - see here for details:

So it looks like WinFSP is working around 2. for us (maybe not in an ideal way), so 1. here is more urgent.

Part 1. also isn't too tricky - maybe you could open a new issue on Github about just that and I'll have a go at fixing. Or you could have a go and I'll talk you through it?

11

Made a issue, If I have time soon and its not done by you or someone else first, ill see if I can figure it out or ill ask you for help, but my job just ramped up to mega since summer started so I probably wont have much time to try to help outside of information and testing

1 Like
12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.