Backblaze B2: error reading destination directory: File name in UTF8 must be no more than 1024 bytes (400 bad_request)

What is the problem you are having with rclone?

After uploading over 5TB to backblaze B2 without problems, I am faced with the following errors:

2019/11/28 11:48:03 ERROR : : error reading destination directory: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/28 11:48:03 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting files as there were IO errors
2019/11/28 11:48:03 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting directories as there were IO errors
2019/11/28 11:48:03 ERROR : Attempt 1/3 failed with 1 errors and: not deleting files as there were IO errors

This carries on through 3 attempts and fails. It's a long story however I was able to find a list of filenames which have UTF8 characters in their filenames (the names, not the file content, to be clear). I excluded the containing directories from rclone, however I'm still encountering this issue.

I contacted backblaze and they suggested I contact rclone.

What is your rclone version (output from rclone version)

rclone v1.44

  • go version: go1.10.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

os/arch: freebsd/amd64 | 11.2-STABLE

Which cloud storage system are you using? (eg Google Drive)

Backblaze B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --transfers 2 --fast-list --copy-links --log-level NOTICE --log-file /mnt/storage/.rclonebackup/rclonelog.txt --exclude /auto** --exclude /backups/android/temp** --exclude /backups/archived/** --exclude /backups/cz/** --exclude /backups/mac/** --exclude /backups/SENSITIVE/home/SENSITIVE/VirtualBox VMs/** --exclude /games/** --exclude /isos/SENSITIVE/** --exclude /.rclonebackup/** --exclude /SENSITIVE/SENSITIVE/** --exclude /SENSITIVE-VMs/** --exclude /movies/** /mnt/storage freenas-at-backblaze-crypt:/mnt/storage

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

This generated a massive 600MB+ log file. It ends with:

2019/11/29 19:10:53 ERROR : : error reading destination directory: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/29 19:10:53 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for checks to finish
2019/11/29 19:10:53 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for transfers to finish
2019/11/29 19:10:53 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting files as there were IO errors
2019/11/29 19:10:53 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting directories as there were IO errors
2019/11/29 19:10:53 ERROR : Attempt 3/3 failed with 1 errors and: not deleting files as there were IO errors
2019/11/29 19:10:53 Failed to sync: not deleting files as there were IO errors

I am not seeing anything of particular interest immediately before that. Is there something I could search for? I'll keep the log file around.

Thank you!

welcome to forum,

i noticed that you are running a very old version of rclone, perhaps you could update to the latest version and test again?

Definitely update and re-test before anything else.

That said, it sounds like you may be running into a problem of a path that is already very long becoming too long when encrypted (encryption adds some length to the pathname).

If that is the problem that would be a backend limitation, but 1024bytes should be enough to work with if you just un-nest some folders and flatten the hierarchy a little. It would help if you could see if the debug-log identifies a spesific name it is having problems with. zipping down the log and hosting it somewhere (it should only be a few % of the total size once sipped because it's text) would allow us to take a look at it ourselves. Do note that some file and folder names might be exposed in this log however - if that is a problem for your privacy concerns.

But go update first of all. This could just as easily be a bug that was fixed over a year ago.

Okay, I updated to:

rclone v1.50.2

  • os/arch: freebsd/amd64
  • go version: go1.13.4

With the rest the same. I found the massive log file, when stripped of messages talking about files that would be excluded, was a mere 194 lines. Of those 194 lines, these are the only ones of interest (I promise!):

2019/11/29 22:11:37 ERROR : : error reading destination directory: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/29 22:11:37 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for checks to finish
2019/11/29 22:11:37 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for transfers to finish
2019/11/29 22:11:37 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting files as there were IO errors
2019/11/29 22:11:37 ERROR : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': not deleting directories as there were IO errors
2019/11/29 22:11:37 ERROR : Attempt 3/3 failed with 2 errors and: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/29 22:11:37 Failed to sync with 2 errors: last error was: File name in UTF8 must be no more than 1024 bytes (400 bad_request)

Thanks for the quick responses, I wasn't expecting any response until after the weekend.

Rclone support staff never sleeps... (or maybe that's just me... probably just me...)

Shame we don't get a name.
--dump headers
and
--dump body
can be used to expose even more information, but I'm not at all sure that will give us the filename we are looking for here. Besides, enabling these makes the log incredibly verbose and hard to parse. You could check but it is not a pleasant experience unless you are experienced at how to filter and manage huge logfiles already.

How much data are you syncing? If it is not an extreme amount we could do a much dumber but simpler method.

  • Make a safety-copy (locally) of the files you are syncing
  • Run the same command again, except use rclone move instead of rclone sync, and also add --delete-empty-src-dirs
  • When complete, only the files that had problems transferring will remain, so we can get an idea of their file and path names.
  • Restore your backup to get back to normal

I'm sure there is a better way, but I can't think of anything better to find this spesific info.

Of course, this won't fix the problem, but it will allow us to verify if the filename/pathname length is the issue, and which files are the problem. That should make it relatively easy to fix the problem by flattening some of the underlying hierarchy in a reasonable way. Sometimes you just end up with way more nested sub-folders that you realistically need. Hopefully it is not out of the question to make such a change? There are other alternatives, but all have some significant drawbacks...

@ncw it feels like this error should report the filename. Maybe this is something we can add in future? Feel free to chime in here too if you know a better way of finding this info - which you probably do I expect :stuck_out_tongue:

Interesting options. I actually AM experienced in tearing through massive logs, that doesn't mean I want to :smiley:
We're talking around 5.2TB of data. I do have enough space locally to duplicate it, I'll go ahead and get that started now. Wheee! Might take me a while to get back to yaz.

EDIT: Oooh wait this is freebsd with zfs, snapshot clone time! But still I need sleep. Continue this later...

Oh, yikes..
Well, technically you would not have to backup any data that you are SURE you have excluded correctly (and I see that seems to be a lot). Just don't do anything risky. You shouldn't be able to lose any data in any case case (as anything not remaining on local would be on the cloud), but it would be a shame if you had to re-download TBs of data to local again to get back to normal.

Protip: you can manage complex includes/excludes with --exclude-from MyFilterList.txt if you want to keep it more manageable or need to expand it even more in future.

If you wait until tomorrow you are likely to get a response from NCW also (main author) and he very likely can suggest a better and less intrusive way to handle this.

Alright. I already have multiple full local backups of all of this, so i'm not too worried. I've made a ZFS clone and can point rclone at that with the options mentioned. I'll give it a fling with -vv and report back.

So I lied and didn't go to bed!

Changing from sync to move and adding --delete-empty-src-dirs resulted in basically the same stuff. In addition, where this was supposed to move from didn't shrink at all in size, so nothing got deleted.

NOW off to bed :smiley:

2019/11/29 23:50:08 ERROR : : error reading destination directory: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/29 23:50:08 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for checks to finish
2019/11/29 23:50:08 INFO : Encrypted drive 'freenas-at-backblaze-crypt:/mnt/storage': Waiting for transfers to finish
2019/11/29 23:50:08 ERROR : Attempt 3/3 failed with 2 errors and: File name in UTF8 must be no more than 1024 bytes (400 bad_request)
2019/11/29 23:50:08 Failed to move with 2 errors: last error was: File name in UTF8 must be no more than 1024 bytes (400 bad_request)

that error was generated from backblaze servers

https://www.backblaze.com/b2/docs/files.html
Names can be pretty much any UTF-8 string up to 1024 bytes long

If that happened I can only surmise that the entire sync source (not including those excluded) is affected. Ie. the top folder. This seems strange... because if it was really a problem of a too-long name it should normally only affect a small number of the total files and folders - not the entire hierarchy from the top.

I do note that you are not using bucket-based syntax in rclone though. For a non-bucket based system (like say - Google Drive) the syntax would be as you use it:
remotename:/folder/file.txt
for a bucket based system (as I am fairly use B2 is) it should normally be:
remotename:bucketname/folder/filename.txt
I have seen some weird bugs happen before related to people mixing these up, so even if it seemingly working for most of the files I think it is worth investigating.

Can you go to Backblaze's native browser and see what the name of the bucket you use is? Let's see what happens if we use the syntax that I would expect to be correct for this use.

I think the problem is that in your local path there is a path + name which is longer than 1024 bytes long.

Something like this will find the longest paths

rclone lsf -R /mnt/storage | awk '{ print length, $0 }' | sort -n -s | cut -d" " -f2- | tail

Depending on what it turns out to be you could exclude it and back it up separately in a different bucket.

That's what I was thinking as well. Good URL, saved.

Yeah, the whole thing seems strange, right? And if it were an issue at the top level, why was I able to successfully upload for months?

You're correct, it is bucket based. Nice eagle eye, I can go ahead and give that change a try. HOWEVER: I decided to try to find long file names inside the bucket, so i did an ls of the bucket and am saving the output. It's taking a really, REALLY long time. Do you think I should let that complete, and then try the change? Or bail and just try the change?

I know what you're talking about on the backblaze site, it will have the bucket name there and the other info I need to do the change.

Hmm, why aren't you just running the scan locally like NCW suggested? I mean, the affected files presumably aren't even on the cloud, so I'm not sure what that would tell us. Also the local scan will be orders of magnitude faster. Listing basic attributes for 5TB locally is quite manageable.

Traversing through the entire hierarchy like this non-multithreaded for 5TB data on the cloud however would probably take close to an hour.

You're right, I apologize, been a long week. Doing that now!

Okay now we're getting somewhere. I used rclone locally as suggested and looked for long file paths + names. The longest I could find was 320 (333 if you include "/mnt/storage").

Then I switched up the dest to the correct bucket format, and watched the logfile and found a nightmare. Entries like:

2019/11/30 18:28:39 ERROR : backups/mac/Applications/Adobe Premiere Pro CC 2019/Adobe Premiere Pro CC 2019.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/sccore.framework/Versions/A: lstat /mnt/storage/backups/mac/Applications/Adobe Premiere Pro CC 2019/Adobe Premiere Pro CC 2019.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/PhotoshopServer.app/Contents/Frameworks/sccore.framework/Versions/A/sccore: file name too long

That's one line out of over 112,000 that has "file name too long" in it! B2 can't handle symlinks so i was having the target files copied. I'm not sure if it has to do anything with what's going on, but there are too many levels of symlinks. It's pure insanity.

Culprit: Whilst in "/mnt/storage/backups/mac/Applications/Adobe Premiere Pro CC 2019/Adobe Premiere Pro CC 2019.app/Contents/Frameworks/PhotoshopServer.app/Contents"

lrwxr-xr-x 1 root wheel 19B Jul 29 11:17 Frameworks -> ../../../Frameworks

And off to infinity we go.

I excluded the mac backups and transfers are now running for the first time in a few weeks! I'll check back in on it tomorrow, but it looks like progress has been made!

mechanic voice:
Well theeere's your problem...

If you need to have more control over how to handle symlinks with rclone I suggest you look at the "local filesystem" remote.

Usually we do not bother with making a remote for the local filesystem, but if you have a need for special settings and options (such as how to deal with symlinks) you can use it to gain more control:

https://rclone.org/local/

You can simply ask rclone to copy symlinks as link-files (which will revert to symlinks when copied back) and that would solve this issue, but of course - you won't actually backup any linked folders that way.

If you must actually follow and copy what the symlink points to then I don't see any other way around this than to untangle the logical knot of this symlink. It will never be possible to keep sym-link-following enabled + copy when the symlink is recursive upon itself. It will just end up looping forever or until something breaks. This is just a logical error and not a limitation that can be fixed per se.

I nailed down where the symlink hell is. As stated before B2 can't handle symlinks, so you have two choices:

  • Throw out the symlink
  • Instead, copy the target of the symlink

I was doing the later, and that's what caused this mess on B2. So I tracked down the location of this mess to two base directories, and went to delete them in the B2 website... LOOK AT THIS:

You are deleting 4,398,363 File(s) 591.5 GB

Meanwhile, on the source here, we have reality:

  • 37,260 files
  • 4.3GB

Just wanted to illustrate how this can get completely out of control on B2. Obviously it's not an rclone thing, but if it could somehow have a feature added to watch out for this situation, it could save people massive headaches.

That, or, as I am wont to do, I get all the Special corner situations that nobody else gets :slight_smile:

Thanks for your time, I'll post a status update again later.

Hmm, I don't think that is right. Rclone has a third option:
https://rclone.org/local/#links-l
Basically, it can convert symlinks to inert textfiles of extension .rclonelink

This preserves the symlink but renders it inactive while in the cloud storage (because as you say, most clouds can not support links) - but once you copy it back to local storage it will revert back to a symlink - thus making it ideal for backing stuff up.