Gdrive to Dropbox migration - File names too long

Bazzu85 · June 1, 2023, 1:31pm

What is the problem you are having with rclone?

I'm migrating from a crypted gdrive remote to a crypter dropbox remote.
Files correctly uploaded to gdrive gives file name too long when syncing to dropbox.
how can I avoid this problem?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

os/version: Microsoft Windows 11 Pro 22H2 (64 bit)
os/kernel: 10.0.22621.1702 Build 22621.1702.1702 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.20.2
go/linking: static
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Dropbox both crypted

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

"N:\rclone\rclone.exe" sync "Media:00 - Save" "MediaDB:00 - Save" --transfers 8 --checkers 16 --bwlimit 50M --stats 10s --log-file "R:\Powershell Scripts\rclone Scripting\log\Media Sync\20230601_091242.log" --fast-list --checksum --verbose --contimeout 60s --timeout 300s --retries 5 --low-level-retries 10 --rc --stats-file-name-length 150 --exclude "desktop.ini" --links --track-renames --track-renames-strategy modtime --retries-sleep 60s

The rclone config contents with secrets removed.

[gdrive]
type = drive
scope = drive
root_folder_id = 17MzJZqHdsxdkEzWdWl8gxR-o4bfByJd1
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxQ2oxNWOcwSXrdougGnkuYWhXraL9TH0C2rIlLQJV6WxetJUoYBXqfBxIepwaCgYKAZoSARASFQG1tDrpNXjgBAo2qvlwWs9BnY9zxg0169","token_type":"Bearer","refresh_token":"1//09-6oPiMav80hCgYIARAAGAkSNwF-L9IrCP9DvFHQFIoDxUvj9SnSM3U22vepOC5A5lnDyJ7H-IenNVHS4Dol5av7o6T8T0cjg6A","expiry":"2023-06-01T16:15:08.5453922+02:00"}
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxxxxxxxx

[Media]
type = crypt
remote = gdrive:01
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Dropbox]
type = dropbox
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx9T9kseSG7M3d3cpC7bwExycYWL0vVaSwMZFcW1ZtPwril4","token_type":"bearer","refresh_token":"Xc96UtAstuoAAAAAAAAAAbkUTmRd4l3_NRPlJgZnLkElhQSLBPwwweYmPaE9f1Td","expiry":"2023-06-01T16:56:51.7511531+02:00"}
client_id = xxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxx

[MediaDB]
type = crypt
remote = Dropbox:Media
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

A log from the command with the `-vv` flag

here's a couple of examples:

Line  4274: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 01 - La figlia di Apollo (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long
Line  4275: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 04 - La rivincita di Narciso (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long
Line  4276: 2023/06/01 10:40:18 ERROR : Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 05 - L'arco di Eros (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv: Failed to copy: file name too long

kapitainsky · June 1, 2023, 1:43pm

The answer is here:

you can shorten encrypted names by using base32768 on Dropbox

so in your config will be:

[MediaDB]
type = crypt
remote = Dropbox:Media
filename_encoding = base32768
password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

kapitainsky · June 1, 2023, 2:27pm

Please note that Dropbox has max path limit 260 characters - using encryption makes it even less. base32768 will make it less painful than default base32 but with path like in your example:

Da sistemare-Rinominare/CARTONI - C'era una volta Pollon (1982)/1ok/C'era una volta Pollon - 01 - La figlia di Apollo (1982 - WEBmux x264 1080p - Ita Jap AC3 - Softsub Ita) [by TAERR Maleno85 Kagome & Jagger T7ST] (1).mkv

it can still be a problem. It is 222 characters + 5 characters for Media folder = 227 characters

Maybe it will work - try - but with Dropbox limitations I would recommend to try keep names shorter.

ncw · June 1, 2023, 3:25pm

I was going to do some maths to work this out, but then I remembered rclone can do this!

This tests the max file length for a base32768 encoded crypt running on dropbox

$ rclone test info -vv --check-length dropboxsecret:file-length
// dropboxsecret
maxFileLength = 463 // for 1 byte unicode characters
maxFileLength = 231 // for 2 byte unicode characters
maxFileLength = 154 // for 3 byte unicode characters
maxFileLength = 115 // for 4 byte unicode characters

So provided you don't have too many non-ASCII characters you should be able to fit 255 char long file names no problem.

The above is for maximum size of a single path unit.

Despite the docs I don't find any evidence that total path lengths need to be kept under 260 characters. This 508 character path works fine. The individual path elements do need to be kept under 255 unicode characters though.

I think that 260 number is from Windows (without using UNC paths) and dropbox is just being conservative here assuming you might look at your dropbox on Windows and some other platform.

$ echo hello | rclone rcat -vv dropbox:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt
$ rclone cat dropbox:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.txt
hello

kapitainsky · June 1, 2023, 3:45pm

Yes I thought the same that 260 is weirdly low - but as it comes from their documentation I just take it:)

At the end with crypt and names encoding only way to work out max path length is to test as e.g.:

/a/b/c/d/e/file.txt

and

/abcde/file.txt

will have very different length after names encoding is applied.

Personally I just try to keep path as flat as possible and do not use ridiculous filenames. Serves me well even with base32

Bazzu85 · June 1, 2023, 3:59pm

Thx both for the reply.
the base32768 remote test say this for example:
2023/06/01 17:58:51 DEBUG : 姜頲ꌧ匧榴⤗✒䂇恟/盼畉钓灙㨴ꌐꍛ䤿䍷㥷又跕ꞙ羞鶂弚彡夏桹耿裼軖㪬赏㞋腓繻螆䠋Ꝕ愨帰紗阠釭禖ⵌ鋷㴇阓矒嬊吂䌄濇䅼蟚牻槰嫚毪蝥玪隅憥剽Ꙙꏈ筦䋈坱蓗磟樨鑯邦ꍾ餈ꂗ挦汫䯧曰幤剒荨圓㚅㫡巍虻醍郣蛝㔙㴘殰鍶慘闌鴮㇋怔洧: Uploading chunk 3/6
is correctly configurated?

kapitainsky · June 1, 2023, 4:21pm

yeah it should look like "chinese" - looks legit

Bazzu85 · June 1, 2023, 5:48pm

Thx..the move is now in progress!

jxmese · June 1, 2023, 9:31pm

Quick question, could using the base32768 encoding instead of the default encoding cause any problems when retrieving/decrypting the files again?

I've already copied over 80% of my stuff to dropbox so changing the encoding now will upload the rest of the files with the new encoding but what about the other files uploaded previously? Will they need to be retrieved without the filename_encoding = base32768 line in the config?

Bazzu85 · June 1, 2023, 9:58pm

You need to create a new clone remote and add it the base32768.
After that you have to move all the files from old to new so rclone re-encrypt the files/folders with the new base..

jxmese · June 1, 2023, 11:01pm

Thanks. I guess I'll get started with that then.

Berzerk90er · June 2, 2023, 8:23am

Is base32768 as safe as base32?

kapitainsky · June 2, 2023, 8:35am

No - unlike base32 (which is 100% safe) its implementation contains case sensitive characters (199 out of 32768)

github.com/rclone/rclone

crypt base32768 file names encoding on case insensitive remotes (e.g. OneDrive) is not 100% safe

opened 10:56AM - 03 Mar 23 UTC

kapitainsky

For various reasons I was trying to understand base32768 in details and I have s…uspicion that base32768 encoding rclone is using is not 100% safe on case insensitive remotes like OneDrive. The reason is that so called safe alphabet used contains both upper and lower case characters. I have raised the question with @Max-Sum as his go encoding procedure is used in rclone. https://github.com/Max-Sum/base32768/issues/1 Chances of upper/lower case collisions are extremely low (but not zero) given that 99% of alphabet used does not have upper and lower cases (asian characters) and it is only used for file names. We would need by chance two different files resulting with base32768 encoding results like: 䏠Ҡ䈠䉀䉠䊀䊠䋀䋠䌀䌠䍀䍠䎀䎠䏀 and 䏠ҡ䈠䉀䉠䊀䊠䋀䋠䌀䌠䍀䍠䎀䎠䏀 In this example Ҡ and ҡ are problematic and as I understand it they are both part of alphabet used by rclone. Maybe I am wrong but I think it would be helpful for somebody more knowledgeable to double check it.

But chances of a real problem are quite small. Based on ncw maths from linked thread:

I also tried harder to calculate the probability of this happening, and I think you'd have to put around 10²¹ files in a single directory before you got a collision, so I don't think it is very likely to happen ever.

The potential issue was raised with base32768 author:

github.com/qntm/base32768

Case folding in base32768

opened 03:12PM - 06 Mar 23 UTC

closed 07:12PM - 07 Mar 23 UTC

ncw

question

We've been using your base32768 encoding in rclone via @Max-Sum 's go port https…://github.com/Max-Sum/base32768 as a way to encode encrypted file names onto cloud storage systems. This seems particularly effective on OneDrive which seems to use UTF-16 internally. However we noticed in this issue https://github.com/rclone/rclone/issues/6803 that there are some characters which can be case folded in the set of 32768 characters. I wrote a little Go program to demonstrate this here: https://go.dev/play/p/SK5G4dnHM6T This prints stuff like ``` Duplicate case folded rune ƃ into Ƃ Duplicate case folded rune ƅ into Ƅ ``` and comes up with the summary ``` Found 521 case folding and 199 duplicate case folding characters out of 32896 ``` Which means that there are 521 characters which have a case folded variant, but more importantly there are 199 characters which have both the upper case and lower case variants in the 32768 characters. This is important because rclone generates file names with these characters and OneDrive is case insensitive. So there is a small chance that two different encrypted file names map to two strings which are the same when compared case insensitively. Now, I think the probability of this is quite small. The minimum length of a file name is 16 bytes (un-encoded) so 128 bits which makes 9 base32768 characters. So there is about a 0.006 chance any given character can be case folded. What we'd like to know is how many of these filenames would we have to put in a directory in order to have a 50% chance of having a case folded collision. I've had a few goes at working this out and I'm coming out with an answer of the order of 10²¹. I'm not sure I trust my maths here but its a big number, that I'm sure of. We've been thinking about making a variant of base32768 which does not include both the upper and lower case versions of any characters which can be case folded. Any thoughts?

but fix wont come from there.

Solution is to create new "base32768" implementation with case insensitive alphabet. I have done some preliminary work and have needed alphabet but it also requires new go module to implement it - as logic is a bit different than current one.

Unless you have billions of files in some folders chances of problems are very tiny. But choice is yours.

kapitainsky · June 2, 2023, 8:50am

IMHO I would say it is safe (with caveat that in theory not 100%). In computing we rely often on such solutions e.g. hashing rclone is using.

However when with hashing we gain something (speed) in trade off of 100% confidence and there is no known better algorithm in case of file names' base32768 encoding we know 100% safe solution exist and is equal in terms of complexity/speed as current one - if you are a go programmer and would like to help I am happy to share my work so far on it.

PogMoThoin22 · June 3, 2023, 6:55pm

Guys how to I get rclone to run this, I've added it to my rclone.conf but I'm not seeing chinese characters like I should

[crypt]
type = crypt
remote = dropbox:stuff
filename_encryption = standard
filename_encoding = base32768
directory_name_encryption = true
password = xxxxxxx
password2 = xxxxxxx

kapitainsky · June 3, 2023, 6:58pm

I suggest you start new thread and explain in details as per template what you are doing and what you are seeing.

Impossible to guess from couple of sentences.

PogMoThoin22 · June 3, 2023, 7:44pm

Answered my question while starting a thread, my version of rclone needed updating, all is good now

system · June 6, 2023, 7:44pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.