Fails to sync to encrypted remote due to "Object name contains unsupported characters"

[panomitrius]

What is the problem you are having with rclone?

Fails to sync certain fails on a rclone encrypted remote. It syncs other files fine, but certain files won't get synced. Why is there an issue with the file names when they're encrypted on the remote anyway?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2 - os/version: ubuntu 22.04 (64 bit) - os/kernel: 5.15.0-60-generic (x86_64) - os/type: linux - os/arch: amd64 - go/version: go1.20.2 - go/linking: static - go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Idrive e2 (S3 storage)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

/usr/bin/rclone sync -v --exclude-from excludes.txt "path/to/sync" idrive-crypt:foldername

The rclone config contents with secrets removed.

[idrive-crypt]
type = crypt
remote = idrive:foldername
password = **********
password2 = **********


[idrive]
type = s3
provider = IDrive
access_key_id = **********
secret_access_key = **********
acl = private
endpoint = **********

A log from the command with the -vv flag

Failed to sync with 11 errors: last error was: XMinioInvalidObjectName: Object name contains unsupported characters.
        status code: 400, request id: 174DD0EBBB08442E, host id:

[Animosity022]

I'd imagine the file name contains some invalid characters for that remote which we can figure out with the debug log.

[panomitrius]

Yea, here are some more details from the log..

2023/03/19 14:32:41 DEBUG : /Articles/A Review of the Neurobiological Basis of Trauma-Related Dissociation and Its Relation to Cannabinoid- and Opioid-Mediated Stress Response - a Transdiagnostic, Translational Approach.pdf: Computing md5 hash of encrypted source
        status code: 400, request id: 174DD542F699F10F, host id: 
2023/03/19 14:33:05 ERROR : /Articles/A Review of the Neurobiological Basis of Trauma-Related Dissociation and Its Relation to Cannabinoid- and Opioid-Mediated Stress Response - a Transdiagnostic, Translational Approach.pdf: Failed to copy: XMinioInvalidObjectName: Object name contains unsupported characters.
        status code: 400, request id: 174DD54CC96A621B, host id: 
2023/03/19 14:33:05 DEBUG : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.epub: Computing md5 hash of encrypted source
        status code: 400, request id: 174DD5524DB73D16, host id: 
2023/03/19 14:33:05 DEBUG : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.pdf: Computing md5 hash of encrypted source
        status code: 400, request id: 174DD542C9021881, host id: 
2023/03/19 14:33:05 ERROR : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.pdf: Failed to copy: XMinioInvalidObjectName: Object name contains unsupported characters.
        status code: 400, request id: 174DD55250079A37, host id: 
2023/03/19 14:33:06 ERROR : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.epub: Failed to copy: XMinioInvalidObjectName: Object name contains unsupported characters.
        status code: 400, request id: 174DD5524FA8D0BB, host id: 

[karlh (Karl)]

2023/03/19 14:33:05 ERROR : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.pdf
2023/03/19 14:33:05 ERROR : /Articles/Yoga Workshop/Neurophysiological and neurocognitive mechanisms underlying the effects of yoga-based practices towards a comprehensive theoretical framework.pdf

I thought the file name may be too long. However according to idrive max is 254 characters.

FAQs about IDrive® e2 account management

[Benjamin_L]

I think your file names are too long. Encrypted file names are encoded in base32 by default, which significantly increases the length. iDrive being case sensitive, you can change the encoding to be base64, which will double the max file name length.

[panomitrius]

Cool, I'll give that a try.

[panomitrius]

I've tried with base64 now and it still can't handle "A Review of the Neurobiological Basis of Trauma-Related Dissociation and Its Relation to Cannabinoid- and Opioid-Mediated Stress Response - a Transdiagnostic, Translational Approach.pdf"

[Benjamin_L]

$ echo -e 'A Review of the Neurobiological Basis of Trauma-Related Dissociation and Its Relation to Cannabinoid- and Opioid-Mediated Stress Response - a Transdiagnostic, Translational Approach.pdf'
| base64 | wc -c
252

Hum. Unfortunately, even in base64, this is very close to the max limit. Given that crypt probably add some metadata, you might be out of luck.

[karlh (Karl)]

Please could you upload the file through the ec2 user interface. Would be intereted to know if it uploads.

[panomitrius]

It uploads fine through their interface.

[karlh (Karl)]

Not sure if this adds value but I created a test.
I made a file name yoga et al. and uploaded to B2 with crypt. Here's the file name encrypted.

0ipapumidium985igg8rd2t29hi3lr2db91ub0hr8hrq3r6b0ghfe1n8nr60oiks3s3t46b20rhvu0fdprmpfc8na9i45q3dpdto37fus1ufhnos6osofjma42ajl1569bsrh0qu67kphru127hpm6vdlpge5livp9aam9s40nsgavpubk61c11nhsl1c8u4j8vddun30k9vqgtljs91di3tt56oc8v63cnj66c9o9sif2fdi2fbpd7ajah05hou3fu

Could you make a file with that name and upload in the user interface.
0ipapumidium985igg8rd2t29hi3lr2db91ub0hr8hrq3r6b0ghfe1n8nr60oiks3s3t46b20rhvu0fdprmpfc8na9i45q3dpdto37fus1ufhnos6osofjma42ajl1569bsrh0qu67kphru127hpm6vdlpge5livp9aam9s40nsgavpubk61c11nhsl1c8u4j8vddun30k9vqgtljs91di3tt56oc8v63cnj66.txt (82 Bytes)

(would only upload here if I added .txt)

Thanks

[panomitrius]

@karlh

0ipapumidium985igg8rd2t29hi3lr2db91ub0hr8hrq3r6b0ghfe1n8nr60oiks3s3t46b20rhvu0fdprmpfc8na9i45q3dpdto37fus1ufhnos6osofjma42ajl1569bsrh0qu67kphru127hpm6vdlpge5livp9aam9s40nsgavpubk61c11nhsl1c8u4j8vddun30k9vqgtljs91di3tt56oc8v63cnj66.txt

It uploads fine, but when I download it, the filename is cut to: "0ipapumidium985igg8rd2t29hi3lr2db91ub0hr8hrq3r6b0ghfe1n8nr60oiks3s3t46b20rhvu0fdprmpfc8na9i45q3dpdto37fus1ufhnos6osofjma42ajl1569bsrh0qu67kphru127hpm6vdlpge5livp9aam9s40nsgavpubk61c11nhsl1c8u4j8vddun30k9vqgtljs91di3tt.txt"
So this seems to be the issue.

[karlh (Karl)]

Thanks for that. This is interesting.

Wonder if rclone could have the ability to shorten file names in the future to accommodate picky remotes?

Further reading

Filename too long

According to the docs on file name encryption This uses a 32 byte key (256 bits) and a 16 byte (128 bits) IV both of which are derived from the user password. So the key material is of constant length regardless of the length of the password. So I really wouldn't expect a longer password to make a difference to the length of the file names. Encrypted file names lengths are this long: ((length_of_filename+1) rounded up to nearest 16 bytes) * 8 / 5 all rounded up to the nearest integer So …

[karlh (Karl)]

I created a e2 account and set-up file encryption (1024) and directory encryption (1024). I tried to upload the file and get the same error

2023-03-21 16:53:07 ERROR : sdfjsdfsdkljsiodgjsdfigjjdfilgjdfg/A Review of the Neurobiological Basis of Trauma-Related Dissociation and Its Relation to Cannabinoid- and Opioid-Mediated Stress Response - a Transdiagnostic, Translational Approach.pdf: Failed to copy: XMinioInvalidObjectName: Object name contains unsupported characters.
status code: 400

Tried again with a long string

2023-03-21 16:55:18 ERROR : sdfjsdfsdkljsiodgjsdfigjjdfilgjdfg/ther8ty3486348ty8etn8nvt89eyt8y8934y68vtty8ny6348ny634n6ye48yternrnhyvt3486y34896348976w48enybhtndbgdfjt46734n6384y6b84nt84yat834y68b4y893y76b894tge7twenybt8e: Failed to copy: XMinioInvalidObjectName: Object name contains unsupported characters.

I changed file name to 128 bit encryption but same error.

Of course their user interface allows longer file lengths, so must be their API available to rclone names.

That's all I have to say: just thought it was interesting this limit issue.

[ncw (Nick Craig-Wood)]

Running this shows the limits so it looks like it is limited to 255 characters for the name as you suspect.

$ rclone test info --check-length idrivee2:rclone-length-test
// idrivee2
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 127 // for 2 byte unicode characters
maxFileLength = 85 // for 3 byte unicode characters
maxFileLength = 63 // for 4 byte unicode characters

Note that these are different to the limits on AWS s3

$ rclone test info --check-length s3:rclone-length-test
// s3
maxFileLength = 1024 // for 1 byte unicode characters
maxFileLength = 512 // for 2 byte unicode characters
maxFileLength = 341 // for 3 byte unicode characters
maxFileLength = 256 // for 4 byte unicode characters

Did you try base64 name encoding - that might get you under the limit.

[Muhammad_Abrar]

can you please include the following in [idrive-crypt] and try again:
filename_encryption = off
directory_name_encryption = false

Probably after filename encryption, the resultant file name is exceeding the limit of 254 characters.

[panomitrius]

That would probably work, but I'd like the file names encrypted.

[Muhammad_Abrar]

yeah, this probably requires a change in rclone to handle the scenario a bit differently where the filename after encryption is exceeding the limit of 254 characters.

[ncw (Nick Craig-Wood)]

Try filename_encoding = base64 - I think this will give just enough extra length for you.