Handling of long paths on Windows (>260 characters)

verize · December 7, 2020, 9:53pm

What is the problem you are having with rclone?

I have exactly the same issue as OP in this thread: Windows max path lengh

When copying files with long paths (more than 260 characters) to a remote, I get a load of I/O errors and rclone fails to copy the files. Copying through a mount or copying via CLI both produce this error.

In the linked thread, it was suggested to enable long paths through the registry/group policy editor (gpedit.msc). So I did that, but the errors still keep popping up (OP also mentioned this).

My question is, is this something that Windows users just have to live with? Or is there something I can do to fix the errors?
It would be great if there actually is a way to handle longer paths with rclone on Windows.

Also, sorry for creating a new thread, but the old one drifted off the main question, never reached a solution and got closed due to inactivity.
And please note that I really appreciate all the work you guys do for rclone; this issue is my only gripe with rclone, apart from that it's been the perfect tool for managing all my cloud data.

What is your rclone version (output from `rclone version`)

rclone v1.53.3

os/arch: windows/amd64
go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 Education 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive (with crypt remote)

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = xxxxxxxx
client_secret = xxxxxxxx
scope = drive
token = xxxxxxxx

[gcrypt]
type = crypt
remote = gdrive:/vault
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxx

A log from the command with the `-vv` flag

Simple log (without -vv) when copying file into mount through explorer:

2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: detected external removal of cache file
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: Non-out-of-space error encountered during open
2020/12/07 17:35:25 ERROR : IO error: cache open with O_TRUNC: failed to truncate: open RW handle failed to open cache file: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: detected external removal of cache file
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: Non-out-of-space error encountered during open
2020/12/07 17:35:25 ERROR : IO error: cache open with O_TRUNC: failed to truncate: open RW handle failed to open cache file: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: detected external removal of cache file
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: vfs cache: failed to open item: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.
2020/12/07 17:35:25 ERROR : video/movie/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu) [1080p x265 10bit BD Dual Audio AAC 5.1]/[Arukoru] Steins Gate the Movie - Load Region of Deja Vu (Steins Gate - Fuka Ryouiki no Deja vu).mkv: Non-out-of-space error encountered during open
2020/12/07 17:35:25 ERROR : IO error: cache open with O_TRUNC: failed to truncate: open RW handle failed to open cache file: vfs cache item: check object failed: vfs cache item: open truncate failed: vfs cache: truncate: failed to open cache file: The system cannot find the path specified.

asdffdsa · December 7, 2020, 10:07pm

i know this can be frustrating.

verize · December 7, 2020, 10:11pm

could you elaborate?

asdffdsa · December 7, 2020, 10:17pm

sure, the limitiation is with the crypt
https://rclone.org/crypt/#file-name-encryption-modes
"file names can't be as long (~143 characters)"

but perhaps i am confused, as i was able to copy a file of length 255 to a crypted remote.

verize · December 7, 2020, 10:25pm

Yeah, now that you mention it I remember reading about that limitation in the docs, but I never really thought about it. And I've been uploading files to the crypt remote with paths that are up to 260 chars (definitely over ~143) without any issues.

I just tried again with "rclone copy" and this time it worked out, even though the path is longer than 260. I don't know what I did different from when I tried yesterday, but it worked out. This effectively is a good enough workaround for me.

The question would be, why does it error out when copying to the mount through explorer?

asdffdsa · December 7, 2020, 10:27pm

well, now i am real confused.

verize · December 7, 2020, 10:29pm

maybe the crypt limitation doesn't apply to the length of the path but rather to each individual segment of the path (folder names, file name)

asdffdsa · December 7, 2020, 10:30pm

we have many experts, hopefully one will stop by soon.

ncw · December 8, 2020, 4:53pm

Rclone uses windows UNC paths internally so there shouldn't be any issued with Windows per-se.

However I don't think the VFS cache uses UNC paths which is maybe where you are seeing the issue.

Can you try an experiment for me?

Run your mount command but mount a local directory and see if you have the same problems.

rclone mount --config "C:\Users\user\.config\rclone\rclone.conf" --progress --stats 1m --stats-one-line-date --vfs-cache-mode full --vfs-cache-max-age 336h --vfs-cache-max-size 100G --cache-dir "F:" --fuse-flag --VolumePrefix="\server\gcrypt" -vv C:/test X:

Then try again with --vfs-cache-mode off and see if you have the same problems.

rclone mount --config "C:\Users\user\.config\rclone\rclone.conf" --progress --stats 1m --stats-one-line-date --vfs-cache-mode off --vfs-cache-max-age 336h --vfs-cache-max-size 100G --cache-dir "F:" --fuse-flag --VolumePrefix="\server\gcrypt" -vv C:/test X:

I'm reasonably sure this is the VFS cache not using UNC paths on Windows but I'm not 100% certain. If it is that problem then you'll see problems with vfs cache mode full but not off.

asdffdsa · December 8, 2020, 5:38pm

does this help?

2020/12/08 12:36:52 DEBUG : rclone: Version "v1.53.3" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "mount" "gdrive-a1b2:" "b:\\mount\\rclone\\gdrive-a1b2" "--dir-cache-time=15s" "--rc" "--log-file=.\\log.mount.gdrive-a1b2.txt" "--log-level=DEBUG" "--vfs-cache-mode=full"]
2020/12/08 12:36:52 NOTICE: Serving remote control on http://127.0.0.1:5572/
2020/12/08 12:36:52 DEBUG : Creating backend with remote "gdrive-a1b2:"
2020/12/08 12:36:52 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/12/08 12:36:52 DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
2020/12/08 12:36:52 DEBUG : gdrive-a1b2: Loaded invalid token from config file - ignoring
2020/12/08 12:36:52 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020/12/08 12:36:52 DEBUG : gdrive-a1b2: Saved new token in config file
2020/12/08 12:36:52 DEBUG : vfs cache: root is "\\\\?\\C:\\Users\\user01\\AppData\\Local\\rclone\\vfs\\gdrive-a1b2"
2020/12/08 12:36:52 DEBUG : vfs cache: metadata root is "\\\\?\\C:\\Users\\user01\\AppData\\Local\\rclone\\vfs\\gdrive-a1b2"
2020/12/08 12:36:52 DEBUG : Creating backend with remote "\\\\?\\C:\\Users\\user01\\AppData\\Local\\rclone\\vfs\\gdrive-a1b2"
2020/12/08 12:36:52 DEBUG : fs cache: renaming cache item "\\\\?\\C:\\Users\\user01\\AppData\\Local\\rclone\\vfs\\gdrive-a1b2" to be canonical "//?/C:/Users/user01/AppData/Local/rclone/vfs/gdrive-a1b2"
2020/12/08 12:36:52 DEBUG : fs cache: switching user supplied name "\\\\?\\C:\\Users\\user01\\AppData\\Local\\rclone\\vfs\\gdrive-a1b2" for canonical name "//?/C:/Users/user01/AppData/Local/rclone/vfs/gdrive-a1b2"

verize · December 9, 2020, 12:57am

I just tried both mount commands and I think you're right about the vfs cache.

Cache mode "full"

It errors out with the same I/O error as before.

RkeJhXisZt

Cache mode "off"

It does copy! Although it's really slow (~12 seconds for a 2 MB file on local FS).

ncw · December 9, 2020, 10:15pm

I should really look at the code more before making wild statments...

It looks like the cache backend does support UNC paths

And the log here supports that

However from the log above it looks like the paths haven't been converted to UNC paths

2020/12/09 01:50:35 DEBUG : vfs cache: root is "F:vfs\\local\\C\\test"
2020/12/09 01:50:35 DEBUG : vfs cache: metadata root is "F:vfs\\local\\C\\test"

Looking at the code it seems to be expecting the drive letter to end with a \ - if you don't then isn't it making a relative path? If it isn't making a relative path then the library needs fixing. Some guidance from Windows users here would be helpful

Try using --cache-dir F:\ - hopefully that will work.

asdffdsa · December 9, 2020, 10:20pm

yes, that is how it works on windows
dir c:\ will output the root of the c: drive.
dir c: will out the current working folder.

ncw · December 9, 2020, 10:39pm

Hmm, so that implies the current behaviour is correct.

I guess what rclone should be doing is making the path absolute before trying to turn it into a UNC path

I had a quick go at that

Can you try this please @verize (or @asdffdsa ) and see if you pass in --cache-dir F: it does the right thing?

v1.54.0-beta.4966.019dbb40e.fix-vfs-cachedir on branch fix-vfs-cachedir (uploaded in 15-30 mins)

asdffdsa · December 9, 2020, 10:51pm

the current behaviour is correct, no fix should be needed.
dir c: is the equivalent to dir . and dir .\
the OP made a common mistake or a typo, that was around when i was using DOS.

if i set --cache-dir=c:test, then a folder named test will be created in a sub-folder of the current working folder.

2020/12/09 17:47:06 DEBUG : rclone: Version "v1.53.3" starting with parameters ["c:\\data\\rclone\\scripts\\rclone.exe" "mount" "gdrive-a1b2:" "b:\\mount\\rclone\\gdrive-a1b2" "--dir-cache-time=15s" "--rc" "--log-file=.\\log.mount.gdrive-a1b2.txt" "--log-level=DEBUG" "--vfs-cache-mode=full" "--cache-dir=c:test"]
2020/12/09 17:47:06 NOTICE: Serving remote control on http://127.0.0.1:5572/
2020/12/09 17:47:06 DEBUG : Creating backend with remote "gdrive-a1b2:"
2020/12/09 17:47:06 DEBUG : Using RCLONE_CONFIG_PASS password.
2020/12/09 17:47:06 DEBUG : Using config file from "c:\\data\\rclone\\scripts\\rclone.conf"
2020/12/09 17:47:06 DEBUG : gdrive-a1b2: Loaded invalid token from config file - ignoring
2020/12/09 17:47:07 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020/12/09 17:47:07 DEBUG : gdrive-a1b2: Saved new token in config file
2020/12/09 17:47:07 DEBUG : vfs cache: root is "c:test\\vfs\\gdrive-a1b2"
2020/12/09 17:47:07 DEBUG : vfs cache: metadata root is "c:test\\vfs\\gdrive-a1b2"
2020/12/09 17:47:07 DEBUG : Creating backend with remote "c:test\\vfs\\gdrive-a1b2"
2020/12/09 17:47:07 DEBUG : fs cache: renaming cache item "c:test\\vfs\\gdrive-a1b2" to be canonical "//?/c:/data/rclone/scripts/rr/other/mount/test/vfs/gdrive-a1b2"
2020/12/09 17:47:07 DEBUG : fs cache: switching user supplied name "c:test\\vfs\\gdrive-a1b2" for canonical name "//?/c:/data/rclone/scripts/rr/other/mount/test/vfs/gdrive-a1b2"
2020/12/09 17:47:07 DEBUG : Google drive root '': Mounting on "b:\\mount\\rclone\\gdrive-a1b2"

asdffdsa · December 9, 2020, 11:23pm

imho, there should not be a fix for this, as there is no bug.
in this case, the problem is human error.

at most, perhaps, emit a warning in the debug log

ncw · December 10, 2020, 8:25am

Where the cache directory is created won't change with the patch above

The same thing will happen, however rclone will use UNC paths which will mean paths longer than 260 chars will work properly - you can see it isn't using UNC paths here.

asdffdsa · December 10, 2020, 1:57pm

ok. thanks.....

verize · December 10, 2020, 2:34pm

I just tried the patch and passed in "F:" as the cache dir (without the trailing backslash) and it did use UNC paths.

I also tried the current stable version (v1.53.3) and passed in "F:\" as the cache dir (with a trailing backslash, as suggested by @asdffdsa), and this also caused it to use UNC paths.

I'm not too sure whether I understood the previous messages correctly, but when not adding a backslash to the cache dir (--cache-dir "F:"), rclone didn't treat it as a relative path, but an absolute one, meaning the cache files were created in the root of drive F: Not sure whether this is relevant though.

asdffdsa · December 10, 2020, 2:58pm

the beta seems to be working as intended, to force UNC

v1.53.3
vfs cache: root is "c:vfs\\wasabi01"

v1.54.0-beta.4966.019dbb40e.fix-vfs-cachedir
vfs cache: root is "\\\\?\\c:\\data\\rclone\\scripts\\rr\\other\\mount\\vfs\\wasabi01"