[local]
type = local
[crpt50-local]
type = crypt
remote = local:
filename_encryption = standard
directory_name_encryption = true
password = xxx
password2 = xxx
[chnk-crpt50-local]
type = chunker
remote = crpt50-local:
chunk_size = 1M
hash_type = md5
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
small sample below of log
2019/11/03 23:08:49 ERROR : d/45/RG7TCUP4XFCRDQ6A3PHP3YEVXQ2526/3TSFEBDYIT6R2HUB2UB23QD4LOOFLPVLW7GG6QGTCH5CSVYWM2MN6H6HFY6JDBX742Y5Z5KUVBKTD6QOOL7JJY4PF44ZTUQIQJBUAP2O7WBC5NUWC27HWIX7WYBQ====: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/k4elp7j9o7g3557s0m48k4aevk/2i10cnrj2mdo9fplonlm1ll8m9b3bbvb73gthiic8otolnr9k4v0/ptjmikag5jfbda2c4e2hcbeiauf4vmgqkljs76sgucfdvptj9nos3vd264poaif1a92h78m11dtjffnvurvqn0oou3emlo3g8j9pnm6khq4bl3u04n56280ruk1bho9bps4737v4me2988345cm4m5brr4msi6u3me5ittb38q4q29btpovhb482o5cmu92jvn9dnrktuib3i79v3tg1l8funq2dfvno9gef3d9ti73krv25mql8jsmtedcsdmst5mbp7k6it907qru3d1mnfk9j6k: file name too long
2019/11/03 23:08:49 ERROR : d/3D/4W5IBLVFOS64ZQZOVV2C65PGFIRWN7/S6CJR7E5UQBFFN4AGZVRZPG6CQ5CI3JFRR4CR6WDLFGLWMRLUPLWKLGSWDUF5TE3UCJWT6V7L2AGN7Y53V6KDRB42F22RWF22T64KNZLXMCLY===: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/34ai279iqdocdv2at0pflfj8fk/h81rgmdvpi9pd2u7f64kv804obodk82spjb5g7majh376j03223g/64i9qlueqid2vg08trfgl0hnilue15fb4fjh5j09rha3069piqdern9ael55sq8j4vdfht78q516c7au5nkjaqn3itq9cgu8shu38u88pnbenkdttnsms8v1sj74lrpo8g72rm7aq8rvqo777laua97skibfo56lq6ff788iotj7jgmp9cqhpiafmnvvp4eeq5hpb5f6s4atph533vd51ohor8i5tm54lmupesdlket8f7bk757ed4ee9g9gp770: file name too long
2019/11/03 23:08:49 ERROR : d/3R/T3O3IRDROZBWD5EHLM4CYGOEEATFDA/ODVHEX4MLEDHOQ6JOVDTES2H4V2K7OVQ45H23JMEZMSY36CT7O2U4DHXGROW5KJPDCUL3AHPTDCFHEPEQDB7VRULIRUROG7WKSAGOIHVO7P4ARA=: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/llb5u1lhh30g0sba42tmmmandc/0eavbp23gc71c4mdhu49cgsfc9gndiv2suflalavargj9ud9uqd0/lkgl5ir29r7nut6f8j1l2e2k72ilreoq686rbh97pd663rg7thpj10s583t2f4hu2brsvotp0up4hur6ior3knhceem7icaisgrpl2106qh2uho8hgf3v8k5pirh4sv4fjmh2akl2dug59lugit7e1tugl6d49regf86d9hn2urbumum5ltskv0kbkrjqbmgoa3pk1d7jv520g80cb2mp2j56bk7077io1ik12dd4et9ap18eo2kimheo1u79mkr: file name too long
2019/11/03 23:08:49 ERROR : d/3R/T3O3IRDROZBWD5EHLM4CYGOEEATFDA/BEC7HQPSEUPCXWLGQT3ASB2BKQIQWMWQEIC7WQMJ5EI6326SKZIJBY5EAI6PUICZEYSAJI5ICLNRW2NSZ6I734HNYQSL6RMDV5WAXFFA5I======: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/llb5u1lhh30g0sba42tmmmandc/0eavbp23gc71c4mdhu49cgsfc9gndiv2suflalavargj9ud9uqd0/e1309n719j0rkgsm7frru75o757hb8u5tnkd2snu5keo9ves1ol6nju9c58fr5sgmm7o0cn53158b58u2omv5nmo58157hlosaek5d7bnrk72tg0kk72fhvimvejeldn6qtg7bfclbmvdd7orii2e210akpjv7pg9d8p1phj18ve8gfrhaivssejmk4snngrs507ktobr0q4cl1o0bp38rpp8spa278d6rlbgt8hq6qin21k80et3e07fgirjp2t: file name too long
2019/11/03 23:08:49 ERROR : d/33/BZTFK2NX4C4OQVFZZ7CQOKY2HCUVIR/357A75YK6EO4IO3E3EY434VXHBHQJEEJG7HOBP3UZSY7ZU2JRXZX37X4YIFPNPKHB6TPVKNJMZW5NDMJ2266P47QJVFFAIXB6JC4ZB5F5LLS6VQ=: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/aore3ho6cq96c5it0j7tmhqigc/l87rol74icbmqoqm059spefj2n8hk5qge16ht7b42t11gco7bb5g/l5i09j0rjeoj5okkecvt65gsqpq7n4688trvocebbdj1veorbj588i4jqfdgvp76fup4bhj5imn2uqiuksbpegijvung6k0enrtqf7dj6j4lobq6mt3dgt4vpcjamf9182ao7tfh2f2ove58tmlc2qqf4lvvppm4l4a4i66omabgc6aon95kgv5o2qn47jb8b72uce2neqmfs335sro4bkjutfudck5vlh3tiimm4p91o6r4c61269s6ug13h294: file name too long
2019/11/03 23:08:49 ERROR : d/3R/T3O3IRDROZBWD5EHLM4CYGOEEATFDA/F4JKK43ZGVWEAOHETXORXQAK5FNP6PMHGL56KECOT7EO7F6UJ4C6SJLTGMWS5AYXS32ABXULFZAR7OWUTRYKRDRNVC55HG37RNNWABRKN7XO7OA=: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/llb5u1lhh30g0sba42tmmmandc/0eavbp23gc71c4mdhu49cgsfc9gndiv2suflalavargj9ud9uqd0/iilsgmj1se3hv8ba9orpq4bk44qpvf51ds40ah9lgk7mosb1ckclcoh42dktuvut2r3pjdfmmrn2eup1j5ep0j4np0av1ceb8vb3hsjkgug1gsfl2eisiidnsq9jdrd9d7len37j40jdf72jg6jpbj0qtppbt1d3v6k1hlgm20sfculophqus19dss49k755afopng7vckhf5hlpgpup3hj6qmjqkgcbk5lfjhapi2slf0ne963q4vktvfamlcv5: file name too long
2019/11/03 23:08:49 ERROR : d/3R/T3O3IRDROZBWD5EHLM4CYGOEEATFDA/OGWCQPQ5TKRXJPBHYVES75MRLAPS7Y6UIJD6YBLLV7FYEKOHNFI573B2T3DNSRUHLTGMEPDD5RQ7CVDP6V3IMG2XE5GJZR57B2JWHZDZJ6ZNHIQ=: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/llb5u1lhh30g0sba42tmmmandc/0eavbp23gc71c4mdhu49cgsfc9gndiv2suflalavargj9ud9uqd0/c9bdistjo1cuk7d7eui46gp94i7cbkg9dm0a9bnmaifhrnof6bml019f5af8aueskmgo5rf9159po0cl85ob0lc694j7gq51p5nv23rf2530kkdhsl16q0rb8nnehbm7bjgjpfodjsigmccbdnsn9l7mmnff0e86slh4jesof11o32aihjm69n5rghje6b7cnajn1b8safibmrhhssv33174tthlc39g0p4an800781vp3g06l72f0c7atdhtkpl: file name too long
2019/11/03 23:08:49 ERROR : d/4T/WRCY2N4FJZLTC53VVP7ZDH56DSZALN/RRZRFPWNDJCWBB432HATX3O4ATSTQEIIIOS3MUJAXQRWVGGKMGNZ7BYNDUNG2VPXK6A2UT23ZNVUG5S54T6XWBBFZECSQYMDN2PMHG5ZTKKG3LNGRY2L2XOPHE======: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/ho2ae54ncdrums4isbkolvbokg/ida6agmkuru7lo63udarh1vrolm248qca4pcosj7pmvav81uvimg/nhk9o20sjoa6ocsj39gut73p58ko46hb0brhj5brou6dpp3267hc2etbhlv6chohfqulcuch753taupsi7f4maf3ff2viqlo5n4djvj3af28qc6blb49k595lgcq91j19htp7bhpbq7f9det939n2v3nrtuij1d1h5hlokh94oq2died3pfa7jhfknf6na6f335kgu51opsaamdmo576r3rvsh9e93jr3k9hgchati9kmg0j0rrcbmdird9ahvrcdth620fdks2iddh4j8vo4forfg: file name too long
2019/11/03 23:08:49 ERROR : d/4V/ECZICFVQPJ6ZVKIGQEUT2TIDVFGLLC/NEAOCEXATTJNOCSCZVSC5H4UG4RVRAPW5MK3NRMWVXBDUUZUNR5HSK2XNF7ZZD2KDV5QGSLTEAW2X67S4AKRDMP4QETCMF76W6PK3UO2SCBUXNEKQENZ4FHFC65Y47YZ: Failed to copy: open /mnt/data/sync/test-data-cyrptom/9evtnfs6r9i2qo8uh0auhllv4c/8vsik43nl1bq7fuvob65dqlfr8/j54p1lnc31i6g6d9ohr9dhbaso/rt5jj4a7smt6tbvsiqluhpcokk581d0ot3s8mchtbl2m0ohp1rtg/vcvmtm6vnv4d7a3k3chm8qjku2qochl7gsdg7h6sm160jpgi18vmnpa6qqlt3eion9km2fohtkhm5ahg4p3iilq77euve9pbkq7orgjjh6kaclc1ktah1sh25aahrgjqucmo1dbm26j6hkrti0mt09s6p6d7178g4gp84vde6cedf11jle3n03p0bdkq5fspmhkm89rg9du2fm2l3d6i2fmkt81lvuq2mvhdhh1hl0og27fint5sfrk0760atsq5f116avel4kd1ukko99tp73eh18: file name too long
Encrypting does make filenames a little longer than original. Not that much though. I don't have the exact numbers on exactly how much longer unfortunately. I think this is a cryptographic inevitability, but I am not an expert on that topic.
the crypt password does not matter for filename length (99% sure). Note that a longer password is also not stronger encryption. Encryption is always 256bit AES (industry standard). The only purpose to having a long password is that it is long enough to not be easily guessed or brute-forced. 100 characters is placebo as anything over 20 random characters will be more than you need and effectively equally secure.
The maximum path+filename lenght is determined by the underlying filesystem
btrfs apparently has only 255char max length according to wiki - so this is something you will have to keep in mind. If you just flatten your directory structure a little you can usually work fine within those limits though. Otherwise, seek another filesystem that has support for longer names.
256+ filenames aren't really acceptable, and I need ability to upload to different clouds and filesystem's without unnecessary restriction.
The rclone crypt page states that standard crypt filenames will have max 146 characters as below. Not sure why the filenames I'm seeing are much longer than that?
### File name encryption modes
Here are some of the features of the file name encryption modes
Off
* doesn’t hide file names or directory structure
* allows for longer file names (~246 characters)
* can use sub paths and copy single files
Standard
* file names encrypted
* file names can’t be as long (~143 characters)
* can use sub paths and copy single files
* directory structure visible
* identical files names will have identical uploaded names
* can use shortcuts to shorten the directory recursion
Sure, but you probably won't like the "solution" as they are both manual "best practice" solutions to avoid the limitation rather than fixes:
Flatten your directory structure more - ie. avoid too many folders in folders in folders in folders. I assume you already know but the 256char name includes not just the actual filename but the path+leafname, so the directory structure is often much more of the issue than some individual long leaf-name (or "filename" as most people would call it).
Try to avoid unnecessarily long filenames. This is usually less viable because you probably do not name all your files yourself...
If your filesystem can't handle more than 256char that is really the main issue here. unencrypted filenames can also easily break 256 depending on the nesting of the structure, but crypt will eat up a small percentage of that quota also. But yea - you are right - certain could-systems won't handle more than 256char either. I personally just avoid these, but if you can't then I suppose may be forced to work within the limits instead...
Finally I guess there is the nuclear option of using file and folder name obscuration rather than encryption. This will not add any length at all - but then the actual content will not be encrypted at all, which is NOT secure.
Beyond this, to make a "full fix" solution would require some sort of remote backend which can repackage all names into metadata objects - but this is a hard problem with a lot of challenges associated with it. It will interact with a lot of other systems and make syncing much harder - as well as no doubt have performance penalties. No matter what way you solve this it is going to come at a non-trivial cost - but it's no doubt possible to do in some way...
I believe that pertains to the "leaf name" , ie the filename minus the path. In technical terms, here's an example to make it clear:
consider this path to a file: C:\folder1\folder2\folder3\MyFile.txt <--- this is the full name (often just called name) C:\folder1\folder2\folder3\ <---- this is the pathname MyFile.txt <--- this is the leaf-name (most people would collocially refer to this as the filename)
It is this last one (the leaf-name) that can't contain more than 143 characters in encrypted format.
However, the full name (path + leafname) is another thing... on a Gdrive this can be 32767 chars for example. Using encrpytion I still have a limit to my leaf-names, but I could use a just about as complex folder-structure as I wanted because I have so much space to work with.
But on a more limited filesystem this total might just be 255chars (or even less for unicode names). Then you suddenly have sparse room to maneuver, especially when crypt eats up a significant chunk of that total namespace - and you usually to compensate with using a much simpler and flatter firectory structure. This is why I tend to just avoid filesystems and cloud backends that have such restricted names - because it's a hassle an poorly suited to a large complex personal archive. Such small names are usually on either older filesystems anyway - or specialized filesystems not really designed for this type of use. You can certain work with those limits if you just keep mindful of them, but it's annoying.
For your personal use locally, the best solution may just to use another FS format. I am not familiar with btrfs so I don't know what your aim with choosing this particular format was...
Take what I say here with some pinch of salt because I do not claim to be an expert on filesystems. This is accurate to be best of my current knowledge and that is the best I can offer you
I think its possible to have full filename encryption and short filenames. Other projects like cryptomator and cryfs both do this, so I see no reason for such long filenames on rclone which reduce the userbase of filesystems and clouds that it can be used on.
I didn't count them, but yes that does seem weird hmm...
I guess you could test with a standard 128bit key and see if that makes any difference, but I was pretty sure that was not related. I didn't think they should ever be able to be that large.
This is too technical for me to comment on. @ncw would need to comment here. I didn't design any of the encryption. That's above my pay-grade
I think I misread the wiki, and that probably lead to you misunderstanding me also.
255char max leaf-name is common, but some filesystem also have pretty short max-pathname limits. That is the limitation I was mostly referring to. I see now from that list that BTRFS actually is not very restricted in pathname, so I retract my complaint in that regard. It shouldn't be more limited than most other filesystems. Sorry for the confusion - I'm just not very familiar with that one and didn't search well enough for the info.
In your case, it's the leaf-names you are having issue with - so this would apply to most filesystems.
Them being 300+ chars seems abnormal to me. Pretty sure I don't have any encrypted leaf-names this long. How long is an example of some of the longest unencrypted leaf-names here for reference?
Hopefully @ncw can pop in and shed some more light on the technical details here.
I just did a test with 3 char password for crypt = 0 errors for filename being too long.
It seems the password length was the problem, as previous tests were with 100 char password.
I'll have to check how low I need to go. I'm quite sure I went as low as 3 word passphrase and still got filename length errors. Thats too low for errors tbh.
So how much did the leaf-names change (roughly) ?
I use 128bit (about 22 or 23 chars) and have no problem with that.
But if NCW says it should not matter... well... he is rarely wrong, so now I am confused
143 real unencrypted chars in the leaf-name is supposed to be the max, as that should not generate more than 255 encrypted characters. At least that's the way I've understood it.
So if you have longer unencrypted leaf-names than this, that's going to be a problem, but below that it should be fine - and AFAIK the length of the crypt-key should not matter for this. Presumably that just gets hashed in somewhere anyway.
EDIT: after a quick visual inspection of my archive, I picked a random long leaf-name and counted it. That was 128chars. I'm sure there may be a longer one, but they mostly seem to be around this and below. The final size of the encrypted leaf-name should depend on the original name, except it will be a bit longer when encrypted.
I would like to know that too, because this does not jive with my understanding of how it should work. @ncw ?
EDIT: could you pick a spesific file below 143 chars leaf-name then
(1) encrypt it with a 100char crypt-key
(2) encrypt it with a 4char crypt key
(3) count the length of each encrypted leaf-name and post them here.
if those greatly vary then there is some kind of issue afoot I suspect.
This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of which are derived from the user password.
So the key material is of constant length regardless of the length of the password.
So I really wouldn't expect a longer password to make a difference to the length of the file names.
Encrypted file names lengths are this long:
((length_of_filename+1) rounded up to nearest 16 bytes) * 8 / 5 all rounded up to the nearest integer
So a 1 byte file name should be 16*8/5 = 26 characters long.
Likewise a 128 byte file name, +1 rounded up to a multiple of 16 becomes 144, 144*8/5 = 230.4 rounded up is 231 characters which is exactly what I get when I try it...
That fits with my current understanding of it (you know, at a slightly less technical level albeit )
But it's weird that @lawmanuk reports result inconsistent with that.
If he performs the test I specified we should get a more solid answer to if there is something unexpected going on.
)