Base32768 filename encoding with crypt + dropbox remote

What is the problem you are having with rclone?

No problem. A question as preventive maintenance.

We are beginning upload of several million files to a Dropbox remote wrapped in a crypt remote in rclone. We are using filename_encoding and have a number of files with very long filenames.

A few hundred of the longest named files generate filenames which are too long for Dropbox with either base32 or base64 encoding (the Failed to copy: file name too long error).
An upload test of the same files using base32768 is successful. Excellent so far.

Digging through rclone and other forums there is some discussion of Dropbox character limitations for Unicode. Indeed running rclone test info shows 4-byte character incompatibility.
Yet I don't see reference to specific/repeatable problems with base32768 encoding of filenames and Dropbox.

maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 255 // for 2 byte unicode characters
maxFileLength = 255 // for 3 byte unicode characters
maxFileLength = -1 // for 4 byte unicode characters
canWriteUnnormalized = true
canReadUnnormalized   = true
canReadRenormalized   = true
canStream = true

Questions:
Are there known incompatibilities with a crypt remote using filename_encoding = base32768 and an underlying Dropbox remote?
If the source filenames contain no emoji or 4-byte Unicode characters, might base32768 nonetheless create a 4-byte character when encoding?

On the rclone.org/overview page you do mention Some cloud storage systems might have restrictions on the characters that are usable in file or directory names. When rclone detects such a name during a file upload, it will transparently replace the restricted characters with similar looking Unicode characters.. But I'm not sure if that applies also to a remote wrapped in a crypt remote.

We've run a test uploading 10,000 or so files with base32768 encoding and so far not run into any issues. But there are in total several million files, thus would prefer to not hit it at the 3,888,888th file. :wink:

If you have no report of specific incompatibilities of dropbox + crypt w filename_encoding = base32768 then brilliant. We will forge ahead.

As always, thank you for the amazing tool and the excellent support forum.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0-beta.6938.243bcc9d0

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-67-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Dropbox Business , primarily team folders dropbox_remote:/

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy local:/path/to/folder dropbox_crypt:/path/to/folder -vvP

The rclone config contents with secrets removed.

[dropbox_crypt]
type = crypt
password = blah
remote = dropbox:
directory_name_encryption = false
filename_encoding = base32768```

#### A log from the command with the `-vv` flag  
No errors with base32768. With base32 or base64 the error for excessively long files is

2023/04/05 09:50:05 Failed to copy with 10 errors: last error was: file name too long

I don't know of any.

The source file name can contain emoji but the base32768 encoding will never contain emoji and all code points in it are UTF-16.

You can check out the design of the encoding here: GitHub - qntm/base32768: Binary-to-text encoding highly optimised for UTF-16

It does apply, yes, but the characters we are talking about are ASCII characters like ?*: which the crypt name encodings will never generate.

There are 32768 possible UTF-16 characters in the base32768 encoding. If you've uploaded 10,000 files it is very likely you've already uploaded all possible characters!

I made a selection of files with all characters from the encoding here: https://pub.rclone.org/base32768.zip

These files upload fine to dropbox so I think you will have no problem with base32768 encoding.

In fact I feel so confident of that I changed the crypt docs to recommend base32768 for onedrive and dropbox!

3 Likes

You are a king, as ever. Thank you. Fast and to the point.

Onward and 32768upward!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.