Rclone vfs cache says file name too long

Help and Support
1

What is the problem you are having with rclone?

Rclone vfs cache says file name too long

The strange part is that it was not too long when it was placed in the cache! The file never ever changed. This is happening to hundreds of files, due I believe to the fact that the cache adds to the path to the vfs system when downloading.Why it could download the file and not upload the exact same file is confusing, as the path is still the same size.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
  • os/kernel: 10.0.19044.1706 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Dropbox.

The command you were trying to run (eg rclone copy /tmp remote:tmp)

"C:\Program Files\rclone\rclone.exe"    mount Media: M: --dir-cache-time 9999h      --poll-interval 15s  --cache-dir=D:\t   --vfs-cache-max-size 250G   --log-level INFO --log-file "D:\Videos\rclone-mount.log" --vfs-cache-mode writes --vfs-write-back 1h --vfs-cache-max-age 9999h --network-mode --vfs-cache-poll-interval 1m

The rclone config contents with secrets removed.

[Dropbox UL]
type = dropbox
token = {"access_token":"mine
server_side_across_configs = true
client_id = something
client_secret = whatever

[Media]
type = crypt
remote = Dropbox UL:LFT/M-T
filename_encryption = standard
directory_name_encryption = true
password = wofhijrpfjweifhjqwe;f
password2 = quocfhwfjwogvj[
server_side_across_configs = true

A log from the command with the -vv flag

The is no standard Rclone log, as mount uses a different log.

2022/06/28 00:49:26 ERROR : Movies/Atticus V. The Architect - The Political Assassination Of Don Siegelman (2017) [1080p x264 AAC 2ch] Size 1898 MB - Length 106 Minutes-poster.jpg: Failed to copy: file name too long
2022/06/28 00:49:26 ERROR : Movies/Atticus V. The Architect - The Political Assassination Of Don Siegelman (2017) [1080p x264 AAC 2ch] Size 1898 MB - Length 106 Minutes-poster.jpg: vfs cache: failed to upload try #115, will retry in 5m0s: vfs cache: failed to transfer file from cache to remote: file name too long

This issue only started occurring after I switched to Dropbox, from Google Drive. The path is less than 256 characters, so I don't know why Dropbox is having an issue - unless the encryption method changed and it makes the filenames much longer. The encrypted path is simply Media: - just as it was before.

Thanks.

2

Google drive has very generous file name limits compared to dropbox.

What it looks like is the file name fits on your local disk, but when expanded by the encryption doesn't fit on dropbox.

The file path is 148 chars long which will expand to > 256 bytes with standard encryption.

You could try using filename_encoding = base32768 which I think will work quite well with dropbox that will shorten the encoded names.

4

Will this require re-encrypting, or do I just add it to rclone.conf and go?

5

It will require re-encrypting, yes. So give it a little test and see if it helps first.

It is possible to get rclone to server side move all the files which means you don't need to upload and download with a bit of cleverness.

What you do is make a new crypt backend copied from the old one, but with the new filename_encoding and a new destination directory.

You can then do rclone move oldcrypt: newcrypt: --crypt-server-side-across-configs and it should just move all the files into place for you without upload and download.

6

I am aware of the server-side command, thanks. I figured that I would need to re-crypt, but was hoping no.. lol

How do I make sure that all of the cache is uploaded? I know it spools the beginning of the files, so I don't want to copy or move them back. Is it OK to simply delete, and I will have no loss? I have a couple of directories that are 330+ gigs in size, possibly from an old vfs cache setting of full.

Thanks.

7

hi,

based on your rclone mount command, files in the vfs file cache are uploaded after one hour.
and check the rclone log.

rclone does not do that, perhaps it is the app accessing M:

8

Rclone can't upload the cache until the file names will fit... If you changed the crypt so the file names would fit, then the vfs would upload any "dirty" files when you restarted the mount.

Or alternatively you can rescue the files out of the cache rclone is having trouble uploading.

9

It's not moving them server-side, as they are only transferring at one-half of the speed of my connection, under 5 MiB/s. I am using the following command:

rclone move oldcrypt: newcrypt: --crypt-server-side-across-configs -vv -P --delete-empty-src-dirs --dropbox-batch-mode sync

and I have tried both with and without all of the other commands, including the dropbox-batch-mode command, and it seems to be faster with it as it kept getting too many retries, and it seems to have decreased the retries sightly.

Please advise.

10

I don't think dropbox supports server-side-across-configs . This is making me thing the whole feature is a mis-feature and we should just have a --server-side-across-configs global flag...

11

I attempted --server-side-across-configs with Dropbox for something unrelated to this ticket and found that it didn't worked between different accounts, only within the same Dropbox account. Were you able to have it work? (I used it by placing the value in rclone.conf, not on the command line - if that makes a difference - I wouldn't expect that it should.)

Thanks.

12

Try this, I made a global flag, which is probably what I should have done to start with.

So use --server-side-across-configs you shouldn't need any other --crypt-server-side-across-configs flags.

v1.59.0-beta.6314.1454e091a.fix-server-side-across-configs on branch fix-server-side-across-configs (uploaded in 15-30 mins)

13

That's working like a charm! The only issues are pacer rate limited - too many read/write, but they work themselves out.

Thanks.

14

I must have something misconfigured in using the mount. I am getting this error:

2022/07/08 21:22:00 NOTICE: Encrypted drive 'Media-16:': ChangeNotify was unable to decrypt "䆺齉䣒⬋碁宺蚱㢿食/锗膜✟稞蝼襯跎旾㕙腳觓ꔾ䫳㥄ᔱ䋌井ɟ/渠㢣鬝琑垃蔐蒻泯扟/ᡊ㶧哿澅芨ꖾ畻仹曟/ᣎ⎩ꖇᇲ䑖᧧㬘䁹鉁ԗ璨傾䝳拿乭䥁澔㳑槒奰熛⣟稟崀㺚噄凿護ꍜ闘魄匣駆䮪拲➫姀鋨阠嵺呻纨岂ⲁ裎䟨⬯Ꝙ㤰犺蕊惕猥猖幎昴㾌㦤晊祔囤亨萍駖ᘥ哎ⲧ蹰ʇ/\x06凶雀髊䟭陰炓齎簩安晪啵岒㣤帶㶗奾ɟ": illegal base32768 data at input byte 0
2022/07/08 21:21:15 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

for every file - even if I just created the cache. The Media-16 is the configuration that is using base32768, so it is correct that it is there. It just cannot resync anything back from the virtual cache. It does attempt to clean, as you can see, but that doesn't make a difference, it's the same error.

15

Is it working apart from the change notifies?

Does rclone -R lsf Media-16: show the files?

16 
rclone -R lsf Media-16:Movies
Error: unknown command "Media-16:Movies" for "rclone"
Run 'rclone --help' for usage.
You could use 'rclone selfupdate' to get latest features.

The files and directories are there. I am running the beta version you created, and when I updated to 1.59.0 release version, the --server-side-across-configs was not in the new version.

2022/07/09 18:23:34 Fatal error: unknown flag: --server-side-across-configs
17

Sorry try

rclone lsf -R Media-16:

That's annoying, I intended to put that in the release, but must have missed in somehow :frowning:

I put it in the latest beta

18

The lsf command shows all the files and the directory structure. So, that means that the vfs cache/mount is not decoding because I can manually decode everything with the cryptdecode command.

Thanks for putting it in the beta.

19

So this does sound like a bug in the changenotify code.

You say you can decode the file names with cryptdecode you see in the log but it appears crypt is failing to decode them.

I tried to replicate this but I haven't been able to yet...

Can you try to capture the changenotify response coming from dropbox?

If you run a mount with -vv --dump bodies --log-file rclone.log then upload a file with rclone copy it will generate a changenotify message. Hopefully it will then generate the same error message and we can see what dropbox actually sent.

Can you tell me what that file name decodes to and I can try that actual filename?

21

All of the lines in the mount log that I attempted to decrypt, I was able to decrypt using cryptdecode, so it seems to be an issue with the mount command. The log of the mount command that you asked for is here: rclone.log, but I am not sure if it will help.

Some example lines:

2022/07/15 01:31:59 NOTICE: Encrypted drive 'Media-16:': ChangeNotify was unable to decrypt "䆺齉䣒⬋碁宺蚱㢿食/而㗝齺皓䘋讶䈊蚛尟傂裧鼟畦踜喍苯┚ʟ/渠㢣鬝琑垃蔐蒻泯扟/ᡊ㶧哿澅芨ꖾ畻仹曟/牍绤勘抁蔮嵮ᆅ⡷ᓝ䍰挨ᆍ懅沮ꏼ没惈珠誂䅞酜髵嵡谇藷ᇥ㭙镃嵷㖪ᑞꐃ壃鐇賠盶ꄋ岓▗阌楆菅鷃╹姉ꕵ蜿险欭渮戹Ə/\x06凶雀髊䟭陰炓齎簩安晪啵岒㣤帶㶗奾ɟ": illegal base32768 data at input byte 0
2022/07/15 01:31:59 NOTICE: Encrypted drive 'Media-16:': ChangeNotify was unable to decrypt "䆺齉䣒⬋碁宺蚱㢿食/䚝帊綄ꝍ妐⥦ꃓ睨呃皗異㐐喂踛臉⪃橑ʟ/䑒踕豲䵽飊闗嚉䄬郟/ᡊ㶧哿澅芨ꖾ畻仹曟/囀ꖟ鹱觖橋䑀ᔺ洭峔䌗蹒輌ꀏ墸要沋瑕䈙的摰靿蟥妻壷鐹撵耲⎃滆噿擎冺㮞㘴㩝勀謆ꉌ疬椀咋燹酌㦲頙惆⫣斦瀞蛒ꐧ髲盖苁ڹ畈⤣棡搠侏/\x06凶雀髊䟭陰炓齎簩安晪啵岒㣤帶㶗奾ɟ": illegal base32768 data at input byte 0

Now, on the remote that I was moving the data from, I have this line repeated over and over, presumably for each file I have transferred:

2022/07/15 01:34:17 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:17 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:17 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:35 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:35 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:35 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:34:52 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:09 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:09 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:29 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:29 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:29 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:29 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:29 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:49 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:49 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:35:49 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:05 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:05 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:05 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:05 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:05 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:23 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata
2022/07/15 01:36:23 ERROR : Dropbox root 'j&;/M-T': Unknown type *files.DeletedMetadata

But then I will have a line or two of a valid filename, but when I check it has not been migrated yet. It just errors due to too_many_write_operations.

22

I need a lot with the "NOTICE: Encrypted drive 'Media-16:': ChangeNotify was unable to decrypt" error in it, so you are right, this log doesn't help yet.

Can you tell me what these decode to please so I can try those file names in dropbox myself.

I've not seen those before. It looks like rclone's handling of deleted files is incorrect or there is something strange about your account.

How did you generate those?