Unable to sync long file names on b2

Suspected Bug
1

Hi

I'm trying to sync all my files to B2 crypted remote.
I believe problem is with very long file names. Sync to local file system was ok.

Error: Attempt 1/3 failed with 7 errors and: File name segment must be no more than 250 bytes (400 bad_request)

I'm using:
windows 10 64
rclone v1.37-101-g0158ab69

  • os/arch: windows/386
  • go version: go1.8

Command is:
rclone sync --transfers 32 --fast-list -v "d:\goran" backblazeBackupCrypt:goran --log-file "backupLogRemote.txt"

log file:

log file with -vv parameter is here (I edited out skipped unchanged files, here is the beginning and the end)

--
I've edited the post to update with new findings - rclone doesnt stop - i've run check command,everything is synched except those files

2

A LongPath tool has ability to fix a long name file errors . It could be one best solution.

3

There isn’t an easy way round this at the moment other than renaming some files to have shorter names :frowning:

Or you could zip what you want to backup first.

1 Like
4

@ncw I am getting the same failure on encrypted b2 connections. Isn’t encryption supposed to mangle filenames into something shorter than 250 bytes?

5

Alas encrypting the file names makes them 60% longer. You could try the obfuscation mode or no encryption for the file names.

6

Alas encrypting the file names makes them 60% longer. You could try the obfuscation mode or no encryption for the file names.

That is even more problematic. Why doesn’t the default behavior guarantee a short filename (least common denominator across all store types)?

7

The problem is one of “reversability”. Most algorithms that make short names are “one way” (e.g. md5 or sha hashing). You can go from a long name to a short name and be pretty sure (not 100% sure, but pretty sure) that each long name maps to a unique short name.

But how do you go from the short name back to the long name so rclone ls will work? Without some form of database that keeps track of the mappings, it’s really really hard. When you try to add in “least common denominator” you end up with routines closer to base64 or uuencode which make the result larger than when you started, and that’s what happens with the encrypting of filenames. The “obfuscate” option was added to try and minimize the increase (in most cases the result is only a few characters longer).

8

Thanks for the summary.

What about storing the mapping “dictionary” in a normal file on the remote store? Give this file a short (maybe even a hard-coded) filename. The contents would be encrypted. To reduce the probability of data loss (if the dictionary is lost or corrupt) scatter multiple copies of it across the store. Compression software WinRAR has a similar concept. At least, I think it does :slight_smile:

Thoughts?

9

There are several tickets about this already… It adds more moving parts to rclone which in turn can make it less reliable. At the moment rclone has everything it needs in a single file - the name and the contents - to do anything else complicates things enormously! However I expect one day we will fix this :slight_smile:

10

Request for clarification/confirmation: the rclone error at issue here arises from a b2 limitation, but a limitation exaccerbated by the way rclone does encryption - yes?

(I have just run into the same problem, using an encrypted b2 remote, on Linux.)

11

Yes (there is a b2 limit on file names) and yes the standard mode expands file names rather a lot. However there are other name encryption modes (eg obfuscated) which are shorter if less secure.

12

Thanks, Nick. I would not mind a link to the relevant documentation. I did find this information about 'Configuration Encryption' but it does not seem to impart the needful. Also - a grouse about the often excellent documentation - what does 'configuration encryption' even mean? Does it mean encryption of configuration? Does it mean configuration of encryption?

13

Read the section on filename encryption.
Here is b2's docs on file names.

That is encrypting the config file, so not what you want here!