Gdrive to Dropbox migration - File names too long

What is the problem you are having with rclone?

I'm migrating from a crypted gdrive remote to a crypter dropbox remote.
Files correctly uploaded to gdrive gives file name too long when syncing to dropbox.
how can I avoid this problem?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
  • os/kernel: 10.0.22621.1702 Build 22621.1702.1702 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Dropbox both crypted

The command you were trying to run (eg rclone copy /tmp remote:tmp)

"N:\rclone\rclone.exe" sync "Media:00 - Save" "MediaDB:00 - Save" --transfers 8 --checkers 16 --bwlimit 50M --stats 10s --log-file "R:\Powershell Scripts\rclone Scripting\log\Media Sync\20230601_091242.log" --fast-list --checksum --verbose --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats-file-name-length 150 --exclude "desktop.ini" --links --track-renames --track-renames-strategy modtime --retries-sleep 60s

The rclone config contents with secrets removed.

type = drive
scope = drive
root_folder_id = 17MzJZqHdsxdkEzWdWl8gxR-o4bfByJd1
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxQ2oxNWOcwSXrdougGnkuYWhXraL9TH0C2rIlLQJV6WxetJUoYBXqfBxIepwaCgYKAZoSARASFQG1tDrpNXjgBAo2qvlwWs9BnY9zxg0169","token_type":"Bearer","refresh_token":"1//09-6oPiMav80hCgYIARAAGAkSNwF-L9IrCP9DvFHQFIoDxUvj9SnSM3U22vepOC5A5lnDyJ7H-IenNVHS4Dol5av7o6T8T0cjg6A","expiry":"2023-06-01T16:15:08.5453922+02:00"}
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxxxxxxxx

type = crypt
remote = gdrive:01
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

type = dropbox
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx9T9kseSG7M3d3cpC7bwExycYWL0vVaSwMZFcW1ZtPwril4","token_type":"bearer","refresh_token":"Xc96UtAstuoAAAAAAAAAAbkUTmRd4l3_NRPlJgZnLkElhQSLBPwwweYmPaE9f1Td","expiry":"2023-06-01T16:56:51.7511531+02:00"}
client_id = xxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxx

type = crypt
remote = Dropbox:Media
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

A log from the command with the -vv flag

here's a couple of examples:

Line  4274: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 01 - La figlia di Apollo (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long
Line  4275: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 04 - La rivincita di Narciso (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long
Line  4276: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 05 - L'arco di Eros (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long

The answer is here:

you can shorten encrypted names by using base32768 on Dropbox

so in your config will be:

type = crypt
remote = Dropbox:Media
filename_encoding = base32768
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
1 Like

Please note that Dropbox has max path limit 260 characters - using encryption makes it even less. base32768 will make it less painful than default base32 but with path like in your example:

Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 01 - La figlia di Apollo (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv

it can still be a problem. It is 222 characters + 5 characters for Media folder = 227 characters

Maybe it will work - try - but with Dropbox limitations I would recommend to try keep names shorter.

I was going to do some maths to work this out, but then I remembered rclone can do this!

This tests the max file length for a base32768 encoded crypt running on dropbox

$ rclone test info -vv --check-length dropboxsecret:file-length
// dropboxsecret
maxFileLength = 463 // for 1 byte unicode characters
maxFileLength = 231 // for 2 byte unicode characters
maxFileLength = 154 // for 3 byte unicode characters
maxFileLength = 115 // for 4 byte unicode characters

So provided you don't have too many non-ASCII characters you should be able to fit 255 char long file names no problem.

The above is for maximum size of a single path unit.

Despite the docs I don't find any evidence that total path lengths need to be kept under 260 characters. This 508 character path works fine. The individual path elements do need to be kept under 255 unicode characters though.

I think that 260 number is from Windows (without using UNC paths) and dropbox is just being conservative here assuming you might look at your dropbox on Windows and some other platform.

$ echo hello | rclone rcat -vv dropbox:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt
$ rclone cat dropbox:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt

Yes I thought the same that 260 is weirdly low - but as it comes from their documentation I just take it:)

At the end with crypt and names encoding only way to work out max path length is to test as e.g.:




will have very different length after names encoding is applied.

Personally I just try to keep path as flat as possible and do not use ridiculous filenames. Serves me well even with base32

1 Like

Thx both for the reply.
the base32768 remote test say this for example:
2023/06/01 17:58:51 DEBUG : 姜頲ꌧ匧榴⤗✒䂇恟/盼畉钓灙㨴ꌐꍛ䤿䍷㥷又跕ꞙ羞鶂弚彡夏桹耿裼軖㪬赏㞋腓繻螆䠋Ꝕ愨帰紗阠釭禖ⵌ鋷㴇阓矒嬊吂䌄濇䅼蟚牻槰嫚毪蝥玪隅憥剽Ꙙꏈ筦䋈坱蓗磟樨鑯邦ꍾ餈ꂗ挦汫䯧曰幤剒荨圓㚅㫡巍虻醍郣蛝㔙㴘殰鍶慘闌鴮㇋怔洧: Uploading chunk 3/6
is correctly configurated?

yeah it should look like "chinese" - looks legit

Thx..the move is now in progress!

Quick question, could using the base32768 encoding instead of the default encoding cause any problems when retrieving/decrypting the files again?

I've already copied over 80% of my stuff to dropbox so changing the encoding now will upload the rest of the files with the new encoding but what about the other files uploaded previously? Will they need to be retrieved without the filename_encoding = base32768 line in the config?

You need to create a new clone remote and add it the base32768.
After that you have to move all the files from old to new so rclone re-encrypt the files/folders with the new base..

Thanks. I guess I'll get started with that then.

Is base32768 as safe as base32?

No - unlike base32 (which is 100% safe) its implementation contains case sensitive characters (199 out of 32768)

But chances of a real problem are quite small. Based on ncw maths from linked thread:

I also tried harder to calculate the probability of this happening, and I think you'd have to put around 10²¹ files in a single directory before you got a collision, so I don't think it is very likely to happen ever.

The potential issue was raised with base32768 author:

but fix wont come from there.

Solution is to create new "base32768" implementation with case insensitive alphabet. I have done some preliminary work and have needed alphabet but it also requires new go module to implement it - as logic is a bit different than current one.

Unless you have billions of files in some folders chances of problems are very tiny. But choice is yours.

IMHO I would say it is safe (with caveat that in theory not 100%). In computing we rely often on such solutions e.g. hashing rclone is using.

However when with hashing we gain something (speed) in trade off of 100% confidence and there is no known better algorithm in case of file names' base32768 encoding we know 100% safe solution exist and is equal in terms of complexity/speed as current one - if you are a go programmer and would like to help I am happy to share my work so far on it.

Guys how to I get rclone to run this, I've added it to my rclone.conf but I'm not seeing chinese characters like I should

type = crypt
remote = dropbox:stuff
filename_encryption = standard
filename_encoding = base32768
directory_name_encryption = true
password = xxxxxxx
password2 = xxxxxxx

I suggest you start new thread and explain in details as per template what you are doing and what you are seeing.

Impossible to guess from couple of sentences.

Answered my question while starting a thread, my version of rclone needed updating, all is good now

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.