Base32768 encoded filenames appear to contain characters that result in a 400 error

Suspected Bug
1

What is the problem you are having with rclone?

base32768 encoded filenames appear to include invalid (non-ASCII) characters

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.0-beta.6487.66ed0ca72
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 5.4.0-122-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.2
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone move dbcrypt-tv:/TV dbcrypt-tv-new:/TV -P --server-side-across-configs --crypt-server-side-across-configs --checkers 1 -vv --log-file /home/seed/rclone_db_db.log --tpslimit 13 --tpslimit-burst 13 --transfers 1 --fast-list --delete-empty-src-dirs --dump headers,requests --retries 1

The rclone config contents with secrets removed.

[DBTV]
type = dropbox
client_id = <REDACTED>
client_secret = <REDACTED>
token = <REDACTED>
server_side_across_configs = true

[dbcrypt-tv]
type = crypt
remote = DBTV:/Media-TV-old
filename_encryption = standard
password = <REDACTED>
password2 = <REDACTED>
directory_name_encryption = true
server_side_across_configs = true

[dbcrypt-tv-new]
type = crypt
remote = DBTV:/Media-TV
password = <REDACTED>
password2 = <REDACTED>
directory_name_encryption = true
filename_encoding = base32768
filename_encryption = obfuscate
server_side_across_configs = true

A log from the command with the -vv flag

log file contents via pastebin

2

Base 32768 encoding should have non ASCII characters. They should all be valid unicode/ utf-8 characters though.

What is the problem you are seeing? And how do you know it is to do with the base 32768 encoding?

3

Understood, thanks!

It appears that Dropbox returns a 400 error on any 'POST /2/files/move_v2' requests whose 'to_path' parameter contains unusual (seemingly non-ASCII) characters, and this seems to occur for me only when using the base32768 encoding. Path names generated using standard encryption do not appear to include non-ASCII characters in the path string representation.

For additional context, in the rclone command originally provided, a server-side move is being attempted to rename objects from standard encryption filenames to base32768 encoding, so in the logs, the string representations of each are displayed as part of the request. The log I previously attached was generated to try to be minimal (all ~25 transfer requests are errors) and show the various characters, encoded/encrypted strings and original/plain, that seem to result in a 400 error.

It's possible that base32768 is a red herring in this. It's also possible Dropbox requires the use of JSON-style "\uXXXX" escape codes for non-ASCII characters in the HTTP body (versus just the header) for this API call. I can't be certain, but I see that all my errors are occurring when base32768 is enabled with Dropbox when rclone generates 'to_path' names that include non-ASCII characters (which doesn't seem to occur when standard encryption is used).

Here are some examples of the errors/files:

2022/10/16 15:34:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/16 15:34:51 DEBUG : HTTP REQUEST (req 0xc001311200)
2022/10/16 15:34:51 DEBUG : POST /2/files/move_v2 HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.60.0-beta.6487.66ed0ca72
Content-Length: 504
Authorization: XXXX
Content-Type: application/json
Dropbox-Api-Path-Root: {".tag": "namespace_id", "namespace_id": "2928283745"}
Accept-Encoding: gzip

{"from_path":"/Media-TV-old/amc2nm47uq9p3speqdvgodvgss/1lab9rduf5uhv4jsvdj753ed70/fiu0vifqdd60gk175n8qjlfbhs/bcjql9s8lud4r5gdcra4233a6lrnu4fq68mam9i3ro3j1kaq273bmj39nidq4gvetvaqdqrbbf7hdu972m4um29iu8j1he5beja9na6fkaosic310rhmg54cu5qg5m581ldshdbm67fp3fshb428udt91s","to_path":"/Media-TV/170.Ya/9.TEE kBLx (1908)/234.lxtLHG 90/165.ZKK qHRD - r34d40 - h kNUD xNT, xNT⁳QD oDQEDBS, h sGHMJ [vdack-4313O g597 dZb6 8.4]-msA.LJU","allow_shared_folder":false,"autorename":false,"allow_ownership_transfer":false}
2022/10/16 15:34:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/16 15:34:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/16 15:34:51 DEBUG : HTTP RESPONSE (req 0xc001311200)
2022/10/16 15:34:51 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 57
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: text/plain; charset=utf-8
Date: Sun, 16 Oct 2022 19:34:51 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 3c2a52f7542a427593aa027b09e86ce3
X-Dropbox-Response-Origin: far_remote

2022/10/16 15:34:51 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/16 15:34:51 ERROR : All Rise (2019)/Season 01/All Rise - S01E17 - I Love You, You’re Perfect, I Think [WEBDL-1080p H264 EAC3 5.1]-NTb.mkv: Couldn't move: move failed: Error in call to API function "files/move:2": Bad Request

2022/10/16 15:34:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/16 15:34:53 DEBUG : HTTP REQUEST (req 0xc0012bff00)
2022/10/16 15:34:53 DEBUG : POST /2/files/move_v2 HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.60.0-beta.6487.66ed0ca72
Content-Length: 575
Authorization: XXXX
Content-Type: application/json
Dropbox-Api-Path-Root: {".tag": "namespace_id", "namespace_id": "2928283745"}
Accept-Encoding: gzip
 
{"from_path":"/Media-TV-old/amc2nm47uq9p3speqdvgodvgss/bhp82npo5i9l9uvcmopu5f0mpfspcjkll4opa0c6drrm95s517og/8o0ps6r7ntfspj3spipeqaqat4/0q36ijkv38skkbjiln7ke65d11mop4dilo7jubqdbpak2rfu4mrcm66adi9rcco0aev55jt4jklbjg7d7on6921n3484nc7ai9l9aecpassvf8iu4rpil8n1vvf16oq7jipn4hcg45mhq8ak11grnbhl57bgs2b576hhfr51rlci0dda6s20","to_path":"/Media-TV/170.Ya/13.cxPQ k' iLRA (6456)/238.pBxPLK 49/193.Idvw Q' Orxg - V49H40 - F54 Udfh Wuxfn+Slnh₏v Shdn Sdfh Wuxfn Sduw LL [ZHEGO-5424s a608 HDF7 6.4]-QWe.pny","allow_shared_folder":false,"autorename":false,"allow_ownership_transfer":false}
2022/10/16 15:34:53 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/16 15:34:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/16 15:34:53 DEBUG : HTTP RESPONSE (req 0xc0012bff00)
2022/10/16 15:34:53 DEBUG : HTTP/2.0 400 Bad Request
Content-Length: 57
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Security-Policy: sandbox allow-forms allow-scripts
Content-Type: text/plain; charset=utf-8
Date: Sun, 16 Oct 2022 19:34:53 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 5b6cc9e3df3d4a8aaa1c341553fa3a7e
X-Dropbox-Response-Origin: far_remote
 
2022/10/16 15:34:53 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/16 15:34:53 ERROR : Fast N' Loud (2012)/Season 05/Fast N' Loud - S05E06 - C10 Race Truck+Pike’s Peak Pace Truck Part II [WEBDL-1080p X264 EAC3 2.0]-NTb.mkv: Couldn't move: move failed: Error in call to API function "files/move:2": Bad Request

Thank you!

4

I don't know whether this is relevant:

It is referenced from this request of a different type, but is similar in that an error is being generated when non-ASCII characters are included directly rather than encoded using JSON-style "\uXXXX" escape codes as suggested above: Upload file with special characters will get INVAL... - Dropbox Community

5

I think this is probably a red herring since

  • dropbox works fine with UTF-8 characters - the unit tests test that
  • there aren't many UTF-8 characters in your test file names
{
  "from_path": "/Media-TV-old/amc2nm47uq9p3speqdvgodvgss/1lab9rduf5uhv4jsvdj753ed70/fiu0vifqdd60gk175n8qjlfbhs/bcjql9s8lud4r5gdcra4233a6lrnu4fq68mam9i3ro3j1kaq273bmj39nidq4gvetvaqdqrbbf7hdu972m4um29iu8j1he5beja9na6fkaosic310rhmg54cu5qg5m581ldshdbm67fp3fshb428udt91s",
  "to_path": "/Media-TV/170.Ya/9.TEE kBLx (1908)/234.lxtLHG 90/165.ZKK qHRD - r34d40 - h kNUD xNT, xNT⁳QD oDQEDBS, h sGHMJ [vdack-4313O g597 dZb6 8.4]-msA.LJU",
  "allow_shared_folder": false,
  "autorename": false,
  "allow_ownership_transfer": false
}

In fact there is only one odd character in there the ⁳ which is \u2073 which I don't think is a valid unicode character but I'm not sure...

I think you've managed to confuse rclone by including both of these. I suggest you comment out the filename_encryption = obfuscate line, then you really will get base32768 characters which are unicode, but dropbox should deal with them fine.

6

Thank you for the suggestion. For the ~25 files above, using filename_encoding = base32768 without filename_encryption worked without generating an error.

Having already done the originally configured move operation on many thousands of files, I am now testing whether I get an error on any of those when configured with filename_encoding = base32768 only and will report back.

If I may ask, do you have any theories about what Dropbox is not liking/not expecting about the filenames that prompted the errors I saw? Is is that the generated names are somehow not UTF-8? I'm especially curious to try to understand if there are other things that I need to be aware of in terms of original file naming limitations when using this config.

7

I would have thought that you won't be able to access them.

You could also try just using filename_encryption = obfuscate without filename_encoding = base32768

The one I examined had an illegal unicode character in it. I'm not sure why exactly but I suspect an interaction between obfuscate and base32768 caused that.

If you use base64/base32768 then the original file names are completely obscured and dropbox won't care. If you use obfuscate then there can be parts of original file names which cause trouble - that might be what is happening here. Can you tell what the character was in the original file name?

8

I'm configuring a separate crypt remote as you described and so can access each variation separately. The source remote is stored on GDrive using filename_encryption = standard, and those files are all fine. The issues that I am seeing are arising when trying to convert to these particular crypt configs. The motivation for moving from filename_encryption = standard was to gain additional path length, as I'm finding that I'm unable to migrate existing files from GDrive to Dropbox due to Dropbox's relatively shorter maximum filename length. Hence, I've been trying to implement your suggestion in this post.

Great to know. When both config options are specified, does it seem that the obfuscate option is taking precedence? From what I can glean, the base32768 encoded output looks very different than the examples I saw with obfuscate, and the filenames being generated when I combined the two looked far more like the latter.

Yes, the ERROR line in the original log emits the plain filename and I've been eyeballing to see which characters are causing it. From eyeballing, I've seen this occur:

  • ’ becomes ⁳
  • ‘ becomes ⁲
  • … becomes ₟
  • ’ becomes ₏
    (among others).
9

My test is still running, but I thought that I really should of course ask to confirm that using filename_encoding = base32768 with Dropbox would be a seemingly reasonable thing to do (as opposed to being ill advised or just something that might work)?

Of course, I understand that there are vastly fewer people using this configuration so it doesn't have remotely the same testing as filename_encryption = standard.

Thank you!

10

I think it should work but @Max-Sum may have some other ideas?

11

I saw the testing done by @azmi . It seems fine to use base32k for Dropbox if dropbox behaves reasonably on unicode, unlike OneDrive.
Happy to see the feature actually helps someone.

12

Is the implication not only that you can't store certain code points, but that you can also have a silent failure with base32768 for paths that contain certain code points, wherein a write would seem to complete without error but in fact not be committed?

13

I don't like the silent failure - that shouldn't happen.

Can you run

rclone touch -vv --dump bodies azmi-dbx:/good_stuff/foo --retries 1 --low-level-retries 1

And post the output?

14 

azmi@stiorra:~$ rclone touch -vv --dump bodies azmi-dbx:/good_stuff/ꡀ --retries 1 --low-level-retries 1
2022/10/19 20:25:38 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "touch" "-vv" "--dump" "bodies" "azmi-dbx:/good_stuff/ꡀ" "--retries" "1" "--low-level-retries" "1"]
2022/10/19 20:25:38 DEBUG : Creating backend with remote "azmi-dbx:/good_stuff/"
2022/10/19 20:25:38 DEBUG : Using config file from "/home/azmi/.config/rclone/rclone.conf"
2022/10/19 20:25:38 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/10/19 20:25:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:38 DEBUG : HTTP REQUEST (req 0xc00005e600)
2022/10/19 20:25:38 DEBUG : POST /2/users/get_current_account HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip

2022/10/19 20:25:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:38 DEBUG : HTTP RESPONSE (req 0xc00005e600)
2022/10/19 20:25:38 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 19 Oct 2022 20:25:38 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id:REDACTED
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 514
REDACTED
2022/10/19 20:25:38 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:38 DEBUG : Dropbox root '': Using root namespace "REDACTED"
2022/10/19 20:25:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:38 DEBUG : HTTP REQUEST (req 0xc00005e900)
2022/10/19 20:25:38 DEBUG : POST /2/files/get_metadata HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 117
Authorization: XXXX
Content-Type: application/json
Dropbox-Api-Path-Root: {".tag": "namespace_id", "namespace_id": "REDACTED"}
Accept-Encoding: gzip

{"path":"/good_stuff","include_media_info":false,"include_deleted":false,"include_has_explicit_shared_members":false}
2022/10/19 20:25:38 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : HTTP RESPONSE (req 0xc00005e900)
2022/10/19 20:25:39 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 19 Oct 2022 20:25:38 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: REDACTED
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 177

{".tag": "folder", "name": "good_stuff", "path_lower": "/good_stuff", "path_display": "/good_stuff", "parent_shared_folder_id": "REDACTED", "id": "id:REDACTED", "shared_folder_id": "REDACTED", "sharing_info": {"read_only": false, "parent_shared_folder_id": "REDACTED", "shared_folder_id": "REDACTED", "traverse_only": false, "no_access": false}}
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : fs cache: renaming cache item "azmi-dbx:/good_stuff/" to be canonical "azmi-dbx:good_stuff"
2022/10/19 20:25:39 DEBUG : Touch time 2022-10-19 20:25:39.242606667 +0000 UTC m=+1.106478490
2022/10/19 20:25:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:39 DEBUG : HTTP REQUEST (req 0xc000820000)
2022/10/19 20:25:39 DEBUG : POST /2/files/get_metadata HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 121
Authorization: XXXX
Content-Type: application/json
Dropbox-Api-Path-Root: {".tag": "namespace_id", "namespace_id": "REDACTED"}
Accept-Encoding: gzip

{"path":"/good_stuff/ꡀ","include_media_info":false,"include_deleted":false,"include_has_explicit_shared_members":false}
2022/10/19 20:25:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : HTTP RESPONSE (req 0xc000820000)
2022/10/19 20:25:39 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 19 Oct 2022 20:25:39 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id:REDACTED
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 105

{".tag": "folder", "name": "\ua840", "path_lower": "/good_stuff/\ua840", "path_display": "/good_stuff/\ua840", "parent_shared_folder_id": "REDACTED", "id": "id:REDACTED", "sharing_info": {"read_only": false, "parent_shared_folder_id": "REDACTED", "traverse_only": false, "no_access": false}}
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : Dropbox root 'good_stuff': Touching non-recursively files in directory "ꡀ"
2022/10/19 20:25:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:39 DEBUG : HTTP REQUEST (req 0xc000820600)
2022/10/19 20:25:39 DEBUG : POST /2/files/list_folder HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 223
Authorization: XXXX
Content-Type: application/json
Dropbox-Api-Path-Root: {".tag": "namespace_id", "namespace_id": "REDACTED"}
Accept-Encoding: gzip

{"path":"/good_stuff/ꡀ","recursive":false,"include_media_info":false,"include_deleted":false,"include_has_explicit_shared_members":false,"include_mounted_folders":false,"limit":1000,"include_non_downloadable_files":false}
2022/10/19 20:25:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : HTTP RESPONSE (req 0xc000820600)
2022/10/19 20:25:39 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Wed, 19 Oct 2022 20:25:39 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id:REDACTED
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 204

{"entries": [], "cursor": "REDACTED", "has_more": false}
2022/10/19 20:25:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/10/19 20:25:39 DEBUG : 6 go routines active
2022/10/19 20:25:39 INFO  : Dropbox root 'good_stuff': Commiting uploads - please wait...
azmi@stiorra:~$ rclone ls azmi-dbx:/good_stuff
        0 foo
azmi@stiorra:~$

[azmi-dbx]
type = dropbox
15

I tried this with the latest beta and it worked just fine - can you verify that?

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.