OneDrive crypt "invalidRequest: The provided name cannot contain leading, or trailing, spaces."

[sigake]

What is the problem you are having with rclone?

I have a OneDrive remote, with a crypt remote pointing to it. As per default, the crypt remote has directory name encryption enabled. I've noticed that creating directories in the crypt with certain names fails with the following error:

Failed to mkdir: failed to make directory: invalidRequest: The provided name cannot contain leading, or trailing, spaces.

This is affecting several directories, almost all with Japanese characters in their names. An example is "∞ループ" (without quotation marks).

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

• os/version: raspbian 10.11 (64 bit)
• os/kernel: 5.10.103-v8+ (aarch64)
• os/type: linux
• os/arch: arm64
• go/version: go1.17.2
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mkdir cryo:∞ループ

The rclone config contents with secrets removed.

[cuone]
type = onedrive
drive_type = business

[cucryo]
type = crypt
remote = one:O
filename_encryption = obfuscate

A log from the command with the -vv flag

2022/03/17 16:09:08 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mkdir" "cucryo:∞ループ" "-vv"]
2022/03/17 16:09:08 DEBUG : Creating backend with remote "cucryo:∞ループ"
2022/03/17 16:09:08 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2022/03/17 16:09:08 DEBUG : Creating backend with remote "cuone:O/220.≇〔〥\u3000"
2022/03/17 16:09:12 DEBUG : Encrypted drive 'cucryo:∞ループ': Making directory
2022/03/17 16:09:14 ERROR : Attempt 1/3 failed with 1 errors and: failed to make directory: invalidRequest: The provided name cannot contain leading, or trailing, spaces.
2022/03/17 16:09:14 DEBUG : Encrypted drive 'cucryo:∞ループ': Making directory
2022/03/17 16:09:15 ERROR : Attempt 2/3 failed with 1 errors and: failed to make directory: invalidRequest: The provided name cannot contain leading, or trailing, spaces.
2022/03/17 16:09:15 DEBUG : Encrypted drive 'cucryo:∞ループ': Making directory
2022/03/17 16:09:16 ERROR : Attempt 3/3 failed with 1 errors and: failed to make directory: invalidRequest: The provided name cannot contain leading, or trailing, spaces.
2022/03/17 16:09:16 DEBUG : 5 go routines active
2022/03/17 16:09:16 Failed to mkdir: failed to make directory: invalidRequest: The provided name cannot contain leading, or trailing, spaces.

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

i was able to create that folder in a onedrive crypt

rclone mkdir onedrivevb_crypt:∞ループ -vv
DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mkdir" "onedrivevb_crypt:∞ループ" "-vv"]
DEBUG : Creating backend with remote "onedrivevb_crypt:∞ループ"
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : Creating backend with remote "onedrivevb:crypt01/7962t8tflte4tgnmpbs39j7n0o"
DEBUG : Encrypted drive 'onedrivevb_crypt:∞ループ': Making directory
DEBUG : 5 go routines active
rclone lsd onedrivevb_crypt:
          -1 2021-11-11 12:41:27         2 en
          -1 2021-03-12 17:28:48         1 termux
          -1 2022-02-06 11:12:05         1 zork
          -1 2022-03-17 11:39:00         0 ∞ループ

[ncw (Nick Craig-Wood)]

@sigake

filename_encryption = obfuscate

If you try using filename_encryption = standard you probably won't have this problem.

I'm not sure why it is a problem though - I did't think the obfuscate encryption made spaces.

Can you re-run your rclone mkdir test with -vv --dump bodies --low-level-retries 1 --retries 1 and post the output here?

[asdffdsa (jojothehumanmonkey)]

@ncw

If you try using filename_encryption = standard you probably won't have this problem.

why would filename encryption affect directory name encryption?

but since you suggested it, did a quick test using filename_encryption = obfuscate.
even with that, the command seems to work for me.

[onedrivevb_crypt02]
type = crypt
remote = onedrivevb:crypt02
filename_encryption = obfuscate
password = 
password2 = 

rclone mkdir onedrivevb_crypt02:∞ループ -vv
DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mkdir" "onedrivevb_crypt02:∞ループ" "-vv"]
DEBUG : Creating backend with remote "onedrivevb_crypt02:∞ループ"
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : Creating backend with remote "onedrivevb:crypt02/220.∲ヿ【ル"
DEBUG : Encrypted drive 'onedrivevb_crypt02:∞ループ': Making directory
DEBUG : 5 go routines active
rclone lsd onedrivevb_crypt02:
          -1 2022-03-17 12:16:25         0 ∞ループ

[sigake]

I re-ran the test with the parameters you suggested, the output is here: output of rclone mkdir cucryo:∞ループ -vv --dump bodies --low-level-retries 1 --retries 1 · GitHub

I did originally try to go with the standard filename encryption and didn't run into this issue then, but I was running into OneDrive's path length limit instead.

[ncw (Nick Craig-Wood)]

Rclone appears to be trying to create the directory 220.≇〔〥 which indeed has a trailing unicode space a \u3000.

It looks like this is allowed on onedrive personal but not on onedrive business.

@sigake

I did originally try to go with the standard filename encryption and didn't run into this issue then, but I was running into OneDrive's path length limit instead.

Ah.

Try setting standard filename encryption and also setting filename_encoding to base32768 - this was specifically designed with onedrive in mind.

Crypt

You'll need the recently released rclone 1.58 for that.

[asdffdsa (jojothehumanmonkey)]

@ncw

It looks like this is allowed on onedrive personal but not on onedrive business.

hi, all the examples i posted, up above, was using onedrive business.
all seem to have worked.

[onedrivevb]
type = onedrive
client_id = 
client_secret = 
region = global
token = 
drive_id = 
drive_type = business

[ncw (Nick Craig-Wood)]

@asdffdsa

all the examples i posted, up above, was using onedrive business.
all seem to have worked.

Try making this directory - cut and paste it as it has a funny unicode space on the end which looks like a normal space but isn't - 220.≇〔〥

Try #2

@ncw

the directory 220.≇〔〥 which

Try #3

The forum software is determined to eat that space! Try cut and pasting from my post above: "Rclone appears to be trying to create the directory..."

[asdffdsa (jojothehumanmonkey)]

maybe i am confused.

@sigake

An example is "∞ループ" (without quotation marks).

i used what the OP posed, as is.
i tested with and without filename_encryption = obfuscate

[ncw (Nick Craig-Wood)]

That's not the actual directory being made by crypt, that's the unencrypted name. You can see the encrypted name in the log above.

[sigake]

Thanks for the suggestion, after a little testing it looks like base32768 filename encoding does let me avoid running into the OneDrive path length limit. Still, I'm curious if the filename obfuscation producing spaces is considered a bug, and if it's feasible to fix?

[ncw (Nick Craig-Wood)]

It's not a space it is some sort of unicode equivalent...

It is unfortunate that obfuscate produces these but difficult to change in a backwards compatible way. What do you think @sweh ?

The other alternative would be to encode the trailing unicode space like we already encode a trailing normal space in the OneDrive backend. This is possible but has backwards compatibility implications too.

[sweh (Stephen Harris)]

Hmm. It's be easy to add a IsSpace() test to the obfuscateSegment() routine

                    if !utf8.ValidRune(newRune) {
                            _, _ = result.WriteRune(obfuscQuoteRune)
                            _, _ = result.WriteRune(runeValue)
                    } else {
                            _, _ = result.WriteRune(newRune)
                    }

If it's the last character and newRune is a whitespace then quote the original rune instead.

However, as you say, that could break existing files. The deobfuscate routine would still work, but filename matching might break on services that allow whitespace endings.

Similarly expanding the OneDrive character kludge mechanism to handle unicode spaces might let filename matching work (we'd need to test all the variations of whitespace to see if OneDrive rejects them all; if so we know no existing file has that pattern! And we'd also need to test other backends also using that kludge...) ugh... that's getting complicated.

I guess it might be possible to add an advanced boolean obfuscate_whitespace (hmm, needs a better name!) which to allow the user to turn on the IsSpace(newRune) test in obfuscateSegment(). That'll run the risk of breaking files if the user specifically turns it on, but won't impact anything else.

EDIT: Thinking on it more; the theoretical correct place (maybe with a global "utf8_whitespace" option) would be in the filename kludging routine. But because this doesn't use a reversible mapping, it'd break the obfuscate routines (the rune expected wouldn't be the rune seen). So I think it'd have to be done as a layering violation in the obfuscate routines itself.

Aside: Windows actually does allow for whitespace at the end of file... in cygwin you can create a file "hello.txt " with a space at the end. But then windows explorer and CMD and similar just can't handle it! Stoopid Microsoft; stoopid!

[ncw (Nick Craig-Wood)]

@sweh

However, as you say, that could break existing files. The deobfuscate routine would still work, but filename matching might break on services that allow whitespace endings.

Yes I think this could be changed in the obfuscate routines, but I think that is probably too much of a backwards compatible break.

With the benefit of hindsight we should have avoided those unicode space like characters!

@sweh

Thinking on it more; the theoretical correct place (maybe with a global "utf8_whitespace" option) would be in the filename kludging routine. But because this doesn't use a reversible mapping, it'd break the obfuscate routines (the rune expected wouldn't be the rune seen). So I think it'd have to be done as a layering violation in the obfuscate routines itself.

Actually the encoding is reversible: Overview of cloud storage systems - it lets you store any filename on any backend so you can use onedrive to back up your linux computer and not have to worry about all the characters in filenames Windows doesn't allow.

We'd have to find new mapping characters for the various unicode spaces though. Currently we map ascii onto ␠ in RightSpace escaping (There is an escaping mechanism if that character is in existing files).

My feeling is that this is the edgiest of edge cases though and I'm not sure it is worth fixing since there is a good workaround.

What do you think?

[sweh (Stephen Harris)]

I wonder if there's an easier answer.. instead of replacing the trailing whitespace with an alternate character, add an extra non-space character. So that way we know it's not whitespace ended. Then Decode just needs to look for that extra character and remove it

Then we don't need to find 6 or 8 or whatever whitespace alternatives!

[ncw (Nick Craig-Wood)]

Ah, so make a "remove this character" escape sequence.

That would work for trailing unicode space at least and would be much easier than finding so many unicode equivalents.

That said, I think I'm going to park this issue in "too difficult for too little gain" as in general there is very little use of unicode spaces, and even less of them trailing a file name. The obfuscate thing is unfortunate but there is a good workaround using the specially designed for onedrive base32768 encoding scheme.