So how much did the leaf-names change (roughly) ?
I use 128bit (about 22 or 23 chars) and have no problem with that.
But if NCW says it should not matter... well... he is rarely wrong, so now I am confused 
I just set the crypt password to aaa and did the test again, with zero errors for filename length.
When the crypt password was 100 char passphrase, all these errors happened.
It seems the password length is affecting the filename length to some degree.
Either way, it shouldn't be more than 143 char leaf name?
143 real unencrypted chars in the leaf-name is supposed to be the max, as that should not generate more than 255 encrypted characters. At least that's the way I've understood it.
So if you have longer unencrypted leaf-names than this, that's going to be a problem, but below that it should be fine - and AFAIK the length of the crypt-key should not matter for this. Presumably that just gets hashed in somewhere anyway.
EDIT: after a quick visual inspection of my archive, I picked a random long leaf-name and counted it. That was 128chars. I'm sure there may be a longer one, but they mostly seem to be around this and below. The final size of the encrypted leaf-name should depend on the original name, except it will be a bit longer when encrypted.
above is unencrypted leafname shown in first post above.
unencrypted leafname = 128 char
encrypted result below = 282 char
Something isn't making sense or is this a bug?
I would like to know that too, because this does not jive with my understanding of how it should work.
@ncw ?
EDIT: could you pick a spesific file below 143 chars leaf-name then
(1) encrypt it with a 100char crypt-key
(2) encrypt it with a 4char crypt key
(3) count the length of each encrypted leaf-name and post them here.
if those greatly vary then there is some kind of issue afoot I suspect.
According to the docs on file name encryption
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of which are derived from the user password.
So the key material is of constant length regardless of the length of the password.
So I really wouldn't expect a longer password to make a difference to the length of the file names.
Encrypted file names lengths are this long:
((length_of_filename+1) rounded up to nearest 16 bytes) * 8 / 5 all rounded up to the nearest integer
So a 1 byte file name should be 16*8/5 = 26 characters long.
Likewise a 128 byte file name, +1 rounded up to a multiple of 16 becomes 144, 144*8/5 = 230.4 rounded up is 231 characters which is exactly what I get when I try it...
This (128 chars)
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
Becomes this (231 chars)
25sgjm2r1ajrdl3t0grbu85gaaafohakpuk3dc07nq2gil5edb822u556h0i1pl2fs901pjb6fnvke5utpne1p41j09i9q1e56h7t6tq1pqkcc9ulhbjoickvbdfkmpmq348rnf9pltcdv5apbemqgprddhve8jc33qov7opurr325k5057mk3tebtef957utpujvt4rc7jovuogobf6iucl6p47np9adp4n29g"
Note that if you have unicode characters, eg é then these take up more than one byte and will bring the total length down.
That fits with my current understanding of it (you know, at a slightly less technical level albeit 
)
But it's weird that @lawmanuk reports result inconsistent with that.
If he performs the test I specified we should get a more solid answer to if there is something unexpected going on.
I did some tests and it turns out that the culprit isn't the crypt command, but when its run with chunking.
config is below:
This contains 100 char cyrpt and 3 char crypt.
[local]
type = local
[crpt100-local]
type = crypt
remote = local:
filename_encryption = standard
directory_name_encryption = true
password = xxx100
password2 = xxx100
[chnk-crpt100-local]
type = chunker
remote = crpt100-local:
chunk_size = 1M
hash_type = md5
[crpt3pw-local]
type = crypt
remote = local:
filename_encryption = standard
directory_name_encryption = true
password = xxx3
password2 = xxx3
[chnk-crpt3pw-local]
type = chunker
remote = crpt3pw-local:
chunk_size = 1M
hash_type = md5
The source leaf file is:
357A75YK6EO4IO3E3EY434VXHBHQJEEJG7HOBP3UZSY7ZU2JRXZX37X4YIFPNPKHB6TPVKNJMZW5NDMJ2266P47QJVFFAIXB6JC4ZB5F5LLS6VQ=
This is 112 char long.
when it goes through the 100 char crypt WITHOUT the chunker, there are no errors as below:
2019/11/04 22:54:27 INFO : d/33/BZTFK2NX4C4OQVFZZ7CQOKY2HCUVIR/357A75YK6EO4IO3E3EY434VXHBHQJEEJG7HOBP3UZSY7ZU2JRXZX37X4YIFPNPKHB6TPVKNJMZW5NDMJ2266P47QJVFFAIXB6JC4ZB5F5LLS6VQ=: Copied (new)
When the same source file goes through the 3 char crypt WITH chunker then we get the LONG filenames that make the errors as below
2019/11/04 22:56:36 ERROR : d/33/BZTFK2NX4C4OQVFZZ7CQOKY2HCUVIR/357A75YK6EO4IO3E3EY434VXHBHQJEEJG7HOBP3UZSY7ZU2JRXZX37X4YIFPNPKHB6TPVKNJMZW5NDMJ2266P47QJVFFAIXB6JC4ZB5F5LLS6VQ=: Failed to copy: open /mnt/data/sync/test-data-cyrptom/mrqupgirc9apqh4g34durgucnc/v16cb0h0k313v72o884aubs2nk/4rf2tlgbh9188jkssacnc3qfb8/bl481m1isan9v7c805riiakrv32honddfikc3rbs6gl27g0a67k0/mi46msgd6l3u161tukc1na4p9a1euih56d8toghq9rih1v92450efmbpnja1co0786s3gs5ngqvhpqe3tflikmo05bsq5n6egqunmcm84dsat0f112j3g63874eihkbcnfgvqkcogvhalnr531gu2ltos9p3nhada8df8h0356lmnm2otbnd74g37d95qhf8h78i0v325qcv5u9der1u88ve6hlk3r60ajnvqdk7eccbfonriqhdjllaq3ch4h3p: file name too long
The resulting encrypted leaf file is 256 char long. Other files are longer.
The commands used for the 3 pw char with chunker and crypt.
2019/11/04 22:56:28 DEBUG : rclone: Version "v1.50.0-016-g7bf05631-beta" starting with parameters ["rclone" "sync" "/mnt/data/sync/data-cryptom/" "chnk-crpt50-local:50/" "--fast-list" "-P" "-vv" "--log-file" "50.log"]
and for the 100 pw crypt without chunker =
2019/11/04 22:54:23 DEBUG : rclone: Version "v1.50.0-016-g7bf05631-beta" starting with parameters ["rclone" "sync" "/mnt/data/sync/data-cryptom/" "crpt100-local:100/" "-vv" "-P" "--log-file" "100.log"]
Maybe its the additional information that the chunker adds to filename, although it still shouldn't be that long?
did further test with chunker crypt.
original leaf name is 124 char
3TSFEBDYIT6R2HUB2UB23QD4LOOFLPVLW7GG6QGTCH5CSVYWM2MN6H6HFY6JDBX742Y5Z5KUVBKTD6QOOL7JJY4PF44ZTUQIQJBUAP2O7WBC5NUWC27HWIX7WYBQ
reducing chunk appendage to 3 char -## results in encrypted filename = 256 char
rclone sync /mnt/data/sync/data-cryptom/ chnk-crpt50-local:50b/ --fast-list -P --log-file 50b.log --chunker-name-format *-##
2019/11/04 23:30:12 ERROR : d/45/RG7TCUP4XFCRDQ6A3PHP3YEVXQ2526/3TSFEBDYIT6R2HUB2UB23QD4LOOFLPVLW7GG6QGTCH5CSVYWM2MN6H6HFY6JDBX742Y5Z5KUVBKTD6QOOL7JJY4PF44ZTUQIQJBUAP2O7WBC5NUWC27HWIX7WYBQ====: Failed to copy: open /mnt/data/sync/test-data-cyrptom/c92hs35rokdkqqatmveibrl5vc/v16cb0h0k313v72o884aubs2nk/4gbe0393a49rfoodn06l10hek8/nvk41r1guk0jfengvs77bbu2eno1k243jvqtboq9g8vb8okcasb0/4gdtkkk8okq958pbt61vcvscr5irccp5gdm2pmbld0866muu9m6tuu5ucoq2hfbagqqaclsdu6gjl95fg2sv0qistcdg6j55i85kpgio5jhv3d3r7a91hvi50tchg2meqsj8b0k9163vakgdcqkejio67cvdpvfuqovoe41l525af8efvao6qmpeehgekh30m82t6ru004n4ctojnmr7vekqr84n74s6fi61dv0us2n0e8csteaho7bs0debsurm: file name too long
using standard chunker appendage the encrypted filename = 282 char
rclone sync /mnt/data/sync/data-cryptom/ chnk-crpt50-local:50/ --fast-list -P --log-file 50a.log
2019/11/04 23:34:54 ERROR : d/45/RG7TCUP4XFCRDQ6A3PHP3YEVXQ2526/3TSFEBDYIT6R2HUB2UB23QD4LOOFLPVLW7GG6QGTCH5CSVYWM2MN6H6HFY6JDBX742Y5Z5KUVBKTD6QOOL7JJY4PF44ZTUQIQJBUAP2O7WBC5NUWC27HWIX7WYBQ====: Failed to copy: open /mnt/data/sync/test-data-cyrptom/mrqupgirc9apqh4g34durgucnc/v16cb0h0k313v72o884aubs2nk/4gbe0393a49rfoodn06l10hek8/nvk41r1guk0jfengvs77bbu2eno1k243jvqtboq9g8vb8okcasb0/s4c2qptfsdob2lm6dihd7hu7qlshnu2kpa6pklm2mg02nseklfho6ag0qibv5afdg9t5bncjbqslv7babe438ebi7sqe9gl7ti835g2lq5p1dd7e8hvjpfehd6ac2b1sact5m8qa7nu2mtbgahqodqah0kkg1955l812m0u6foja93l1b25nfs2udgq2v5aarqpqqdejqfaud1c5handtjpnf9n9iojplsc6dg08ec6i98nj1d61e62sutumrbova2j2jhtospcnb6i4lusia71gug: file name too longThanks for the test. This makes more sense at least 
Yes the chunker does effectively add a little more to the name, but it's a few chars right? (is it 4?, 5?)
I can't quite see why that would make the encrypted leaf name jump up so drastically. The chars chunker appends to the name should be no different to any other chars (ie. a leaf-name that was 128char to begin with but not using chunker) - and that should still be well within your limits.
Unfortunately I have extremely limited practical experience with chunker yet. It is still very new and I have literally tested it once. I only really know the main principles it operates by. That kind of makes it difficult to say exactly what the issue might be if it relates to that backend.
It almost sounds like the names are getting translated to unicode in chunker and suddenly take up twice the space or something. There is just too much of an increase here to account for.
NCW gave you the formula for the encrypted name length above. If you input the number of chars in one of your chunker output files (pre encrypt) - does it come out right? I kind of expect it wont based on your data.
If you jump over a 16 byte boundary then you'll suddenly get 16*8/5 = 26 more characters in your file name. That might explain it.
Yes, but I can't reconcile that with a jump from 124char to 256char - assuming I am understanding the data above correctly.
The chunker adds only a few chars to the file name as far as I remember. rclone lsf in the right remote will show the names.
I've checked the chunker without crypt and it only adds ".rclone_chunk.###' to the right of each filename. ie. 17 chars.
Doesn't explain the increase in file size?
It does seem like it shouldn't be able to account for all of it according to your data, no.
It will obviously increase the filename_size somewhat, but not so drastically as you describe.
At this point I don't really have any strong idea about what may be causing the problem.
Sidenote: is the postfix configurable? I haven't played around with chunker enough to know. If it is then I'd consider setting something shorter even if it may not directly the cause of this problem, because that seems unnecessarily long. I assume this is a "magic word" it looks for to identify a chunk, but aside from the numbers at the end that could probably be shortened to just 3-4 uncommonly used characters, making the postfix 6-7 chars instead of 17 to save on namespace.
EDIT: Apprently yes:
--chunker-name-format
The chunker backends wasn't dev'ed by NCW, but contributed by Ivan Andreev - so you may perhaps ask him if he has any knowledge about this? I don't know him personally, and I don't know if he frequents the forum, but you can probably find him on github and make contact there. Assuming that this is a genuine bug in chunker that you can reproduce for him, he is likely to be interested in fixing it.
If you can list with rclone ls the remote immediately under the chunker remote it will show you the file names.
BTW I have this issue in my rclone cache-crypte-mount too.
Think this is cause I have really too long Path.
edit: 280 chars
regards
As in above post, the default postfix is 17 additional char for chunker. I amended this to 3 char as its configuarable.
Original filename was 124 char.
-
124 + 3 char post fix = 127 char unencrypted = 256 encrypted chunker
-
124 + 17 char post fix = 141 char unencrypted = 282 encrypted chunker
Either way, it makes it impossible for me to use the encrypted chunker
Ah, I see the problem. It is that chunker generates intermediate names whch are longer. I did a little experiment with a 127 char file name and chunker tries to save this as a temporary name
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.rclone_chunk.001..tmp_1573122616
which encrypts to this 282 char file name
95ooh9l4s9urukjojv46tq2lgott5upqcuvolbanjpfej46458ltld8vck5pn1i7u0m4hi4j4q82nnq4rnbgun7tgqp35u9871sj5tcefued8dlc5b7q00er37vdt445n1365a9tjptlfgjqb8ukdpuk6ud6aiqdidhvvcv8j2dlebvpriebbe1kbgipnp4j65c4srk3jofs3pir4ibao71ejo9bkhaq1jh6o9li5huffschk4ncqpo56vs4j5opbbeohbfbuccnrud93o2h9dopr8
I used this config (with randomly generated passwords I don't care about
[crypt]
type = crypt
remote = /tmp/secret
filename_encryption = standard
password = igAQKcbmiGoYHJyj8-eWL-FxRZBQ19EIqZTC
password2 = UhLYvwXume37V_w8sR-iaT7k4nTofzjWNUjC
[chunker]
type = chunker
remote = crypt:
chunk_size = 1M
hash_type = md5
And created a 2MB file whose name was 127 a characters and tried to copy it in
$ rclone --config /tmp/rclone.config copy -vv aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa chunker: --retries 1 --low-level-retries 1
2019/11/07 10:35:44 DEBUG : rclone: Version "v1.50.1-024-ge557586e-mount2-v2-beta" starting with parameters ["rclone" "--config" "/tmp/rclone.config" "copy" "-vv" "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" "chunker:" "--retries" "1" "--low-level-retries" "1"]
2019/11/07 10:35:44 DEBUG : Using config file from "/tmp/rclone.config"
2019/11/07 10:35:45 DEBUG : aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: Need to transfer - File not found at Destination
2019/11/07 10:35:45 ERROR : aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa: Failed to copy: open /tmp/secret/mtf0f7hst60i7onogcudk3blf9dp3c18k0vqaoj9gqvpd51q99rkvdphqirrakdt1tpii6rd9em1jn24ghv58f4chdncu92p7q0etdeuk859d837k8thu50amh6mnotb3eh0g09l9b85epk4g0ij643m3nidm5adnsglj1hgclr7ulf5q7qu56krjl75ve91ob7hh97m6ihttc66cp54f7s5vd3neloj5vkoql5b18h9h79ju8motovei30dern4do63f60gu1g5d223jpoph43a64: file name too long
2019/11/07 10:35:45 ERROR : Attempt 1/1 failed with 3 errors and: open /tmp/secret/mtf0f7hst60i7onogcudk3blf9dp3c18k0vqaoj9gqvpd51q99rkvdphqirrakdt1tpii6rd9em1jn24ghv58f4chdncu92p7q0etdeuk859d837k8thu50amh6mnotb3eh0g09l9b85epk4g0ij643m3nidm5adnsglj1hgclr7ulf5q7qu56krjl75ve91ob7hh97m6ihttc66cp54f7s5vd3neloj5vkoql5b18h9h79ju8motovei30dern4do63f60gu1g5d223jpoph43a64: file name too long
2019/11/07 10:35:45 Failed to copy with 3 errors: last error was: open /tmp/secret/mtf0f7hst60i7onogcudk3blf9dp3c18k0vqaoj9gqvpd51q99rkvdphqirrakdt1tpii6rd9em1jn24ghv58f4chdncu92p7q0etdeuk859d837k8thu50amh6mnotb3eh0g09l9b85epk4g0ij643m3nidm5adnsglj1hgclr7ulf5q7qu56krjl75ve91ob7hh97m6ihttc66cp54f7s5vd3neloj5vkoql5b18h9h79ju8motovei30dern4do63f60gu1g5d223jpoph43a64: file name too long
You can then use cryptdecode to work out what the file name is
$ rclone cryptdecode --config /tmp/rclone.config crypt: mtf0f7hst60i7onogcudk3blf9dp3c18k0vqaoj9gqvpd51q99rkvdphqirrakdt1tpii6rd9em1jn24ghv58f4chdncu92p7q0etdeuk859d837k8thu50amh6mnotb3eh0g09l9b85epk4g0ij643m3nidm5adnsglj1hgclr7ulf5q7qu56krjl75ve91ob7hh97m6ihttc66cp54f7s5vd3neloj5vkoql5b18h9h79ju8motovei30dern4do63f60gu1g5d223jpoph43a64
mtf0f7hst60i7onogcudk3blf9dp3c18k0vqaoj9gqvpd51q99rkvdphqirrakdt1tpii6rd9em1jn24ghv58f4chdncu92p7q0etdeuk859d837k8thu50amh6mnotb3eh0g09l9b85epk4g0ij643m3nidm5adnsglj1hgclr7ulf5q7qu56krjl75ve91ob7hh97m6ihttc66cp54f7s5vd3neloj5vkoql5b18h9h79ju8motovei30dern4do63f60gu1g5d223jpoph43a64 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.rclone_chunk.001..tmp_1573122945
@lawmanuk can you please make a new issue on github about this with a link to this forum discussion and I'll see if the author of the chunker backend can come up with some ideas - thanks.
1 Like
Done.
Separately, is there a way to shorten names on encryption, rather than have lengthen even a bit, creating potential for errors?
That way, there can never be a crypt error due to filenames (separate from chunker issue). I know some other projects like cryfs / cryptomator shorten the filenames on purpose. As you said, you don't want to create an index file (although I support any solution which helps to tighten security (of metadata etc.) ), but was wondering if anyway to use some kind of compression algorithm to reduce filename length during crypt? Longshot I know.