Problem with é/è/à in file names

What is the problem you are having with rclone?

rclone can't list files when the folder's name contains accent (é/è/à/ç...)

Run the command 'rclone version' and share the full output of the command.

rclone v1.57.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-100-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls S3:nextcloud/ulysse132/files/Téléphones/

same behavior with rclone mount, the files doesn't appear on the directory tree

The rclone config contents with secrets removed.

[S3]
type = s3
provider = Other
env_auth = false
access_key_id = 
secret_access_key = 
endpoint = s3.pub1.infomaniak.cloud
acl = private
bucket_acl = private
force_path_style = true
v2_auth = true

A log from the command with the -vv flag

rclone ls S3:nextcloud/ulysse132/files/Téléphones/ -vv
<7>DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "S3:nextcloud/ulysse132/files/Téléphones/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/Téléphones/"
<7>DEBUG : Using config file from "/home/ulysse132/.config/rclone/rclone.conf"
<7>DEBUG : Using v2 auth
<7>DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/Téléphones/" to be canonical "S3:nextcloud/ulysse132/files/Téléphones"
<7>DEBUG : 4 go routines active

rclone lsd S3:nextcloud/ulysse132/files/ -vv
<7>DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsd" "S3:nextcloud/ulysse132/files/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/"
<7>DEBUG : Using config file from "/home/ulysse132/.config/rclone/rclone.conf"
<7>DEBUG : Using v2 auth
<7>DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/" to be canonical "S3:nextcloud/ulysse132/files"
           0 2022-03-02 06:48:29        -1 .thumbnails
           0 2022-03-02 06:48:29        -1 Modèles
           0 2022-03-02 06:48:29        -1 Oliver
           0 2022-03-02 06:48:29        -1 Téléphones
           0 2022-03-02 06:48:29        -1 etc
<7>DEBUG : 4 go routines active

Is there any way to configure rclone to accept accent ? It's not possible to rename each folder, as I will not the only one using rclone mount as nextcloud backend.

Thanks a lot for your help !

Rclone should be working fine with accents.

$ echo hello | rclone rcat s3:rclone/nextcloud/ulysse132/files/Téléphones/file.txt
$ rclone ls s3:rclone/nextcloud/ulysse132/files/Téléphones/
        6 file.txt

Are you sure there are files in the directory nextcloud/ulysse132/files/Téléphones/?

What does this do - does it list into the Téléphones directory?

rclone ls S3:nextcloud/ulysse132/files/

I guess this could be something to do with your provider? I tried AWS above.

I'm 100% sure it is not empty, and this for 2 reasons :

  • Here is a screenshot of infomaniak public cloud manager, where you can si 3 folders
    image
  • The files listed here where uploaded through rclone using nextcloud

I'm not using the storage backend without using rclone, this way, it is always up-to-date, normally...

Can you try

Also could the directory name have a trailing space?

In the folder Oliver, I have plenty of files, which means it takes a very long time to list. But inside this folder, I also have other folder with accent. I ran :

rclone lsd S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/
           0 2022-03-02 16:58:46        -1 Etiquette liqueur banane
           0 2022-03-02 16:58:46        -1 Extrudeuse
           0 2022-03-02 16:58:46        -1 Impressions 3D
           0 2022-03-02 16:58:46        -1 Meuble Salon
           0 2022-03-02 16:58:46        -1 PID amélioré
           0 2022-03-02 16:58:46        -1 Ventilo congélateur
           0 2022-03-02 16:58:46        -1 Vélo Bambou - Vélo couché
           0 2022-03-02 16:58:46        -1 Zigbee2MQTT

rclone lsd S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/ has no result just like rclone ls S3:nextcloud/ulysse132/files/Téléphones/ but

rclone ls S3:nextcloud/ulysse132/files/
...
202849 Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/HTML/Velo/Vélo couché/www.atomiczombie.com/wp-content/themes/RadBrad/images/ads/6 Pack Special.png
...

wich means it list inside the directory. And it's not related to trailing space with this example;

Good we have an example to play with

Can you run

rclone lsd -vv --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/

And post (or attach) the output so I can analyse what is going on.

Thank you

 rclone lsd -vv --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/
2022/03/03 06:51:41 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsd" "-vv" "--dump" "bodies" "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/"]
2022/03/03 06:51:41 DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/"
2022/03/03 06:51:41 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/03/03 06:51:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/03/03 06:51:41 DEBUG : Using v2 auth
2022/03/03 06:51:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/03/03 06:51:41 DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/" to be canonical "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché"
2022/03/03 06:51:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/03/03 06:51:41 DEBUG : HTTP REQUEST (req 0xc000593400)
2022/03/03 06:51:41 DEBUG : GET /nextcloud?delimiter=%2F&max-keys=1000&prefix=ulysse132%2Ffiles%2FOliver%2FDocuments%2FProjets+personnels%2FV%C3%A9lo+Bambou+-+V%C3%A9lo+couch%C3%A9%2F HTTP/1.1
Host: s3.pub1.infomaniak.cloud
User-Agent: rclone/v1.57.0
Authorization: XXXX
Date: Thu, 03 Mar 2022 05:51:41 UTC
Accept-Encoding: gzip

2022/03/03 06:51:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/03/03 06:51:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/03/03 06:51:41 DEBUG : HTTP RESPONSE (req 0xc000593400)
2022/03/03 06:51:41 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 03 Mar 2022 05:51:41 GMT
Strict-Transport-Security: max-age=63072000
X-Amz-Id-2: tx0ad01b16890b4d7b8f232-006220576d
X-Amz-Request-Id: tx0ad01b16890b4d7b8f232-006220576d
X-Openstack-Request-Id: tx0ad01b16890b4d7b8f232-006220576d
X-Trans-Id: tx0ad01b16890b4d7b8f232-006220576d

152
<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>nextcloud</Name><Prefix>ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/</Prefix><Marker/><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
0

2022/03/03 06:51:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/03/03 06:51:41 DEBUG : 4 go routines active

There is something weird going on with the encoding here...

So where did that weird encoding come from?

Rclone sent this

If I decode that URL parameter with python it looks correct

>>> import urllib.parse
urllib.parse.unquote_plus('ulysse132%2Ffiles%2FOliver%2FDocuments%2FProjets+personnels%2FV%C3%A9lo+Bambou+-+V%C3%A9lo+couch%C3%A9%2F')
'ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/'
>>> 

So what it looks like is that your provider decoded the URL but interpreted it as latin1 not utf-8

Just to double check the encodings didn't get messed up in the cut and paste can you run the above to a log file and attach the log file here?

rclone lsd -vv --log-file rclone.log --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ 

Thank you

Here you are :slight_smile:
rclone.log (3.8 KB)

Some side information : infomaniak Public Cloud (my provider) is build on OpenStack. I connect using S3 compatible credentials. May be the problem comes from here. I already tried to connect using swift credentials, but I don't succeed. I asked for help to infomaniak.

Thanks for that.

Yes, it looks the same in the rclone.log file - thanks for the confirmation.

I think this is some kind of bug in the cloud provider - a confusion of encodings at some point, but I can't tell exactly where.

So, there is nothing to do ?

That's weird, because these folders were created using rclone...

If the help I asked for to infomaniak don't succeed, I'll move to another solution, such as seafile, where my files are stored using only alphanumerical characters

If you were to set up a crypt backend this will hide the file names from the provider and it will work properly.

I really thought it will work but sadly not, and I really don't understand why...

I rebuild my entire server, create an encrypt backend to a new bucket, put some files and try. After 2h, if I rescan the folders, I can't find any file.

rclone lsd Nextcloud:
          -1 2022-03-04 07:15:16        -1 Coque Personnalisée
          -1 2022-03-04 07:20:58        -1 Documents
          -1 2022-03-04 07:03:59        -1 Modèles
          -1 2022-03-04 07:05:03        -1 Photos
          -1 2022-03-04 07:51:00        -1 Téléphones
rclone ls Next:ulysse132/files/
   567689 Photos/Steps.jpg
   427030 Photos/Vineyard.jpg
   593508 Photos/Birdie.jpg
   167989 Photos/Toucan.jpg
   457744 Photos/Frog.jpg
   797325 Photos/Nextcloud community.jpg
   474653 Photos/Gorilla.jpg
  2170375 Photos/Library.jpg
      150 Photos/Readme.md
     3338 Photos/photos endommagées.txt
 11712256 Nextcloud Manual.pdf
    25150 Documents/Welcome to Nextcloud Hub.docx
     1095 Documents/Example.md
   976625 Documents/Raisons d'utiliser Nextcloud.pdf
      136 Documents/Readme.md
   374008 Documents/Nextcloud flyer.pdf
  1288940 91.jpg
  1429172 Coque Personnalisée/clipper-ship-tall-masts-sailing-nautical.jpg
    89885 Coque Personnalisée/6031905039_65ce7e3d47_b.jpg
  1000812 Coque Personnalisée/sailing-old-rig-sea-water-sailing-vessel-port-2.jpg
   537912 Coque Personnalisée/3953885706_edd02280c0_o.jpg
  1621283 Coque Personnalisée/rope-boat-sea-rigging.jpg
  2112200 Coque Personnalisée/158939597_3b18cd3f3e_o.jpg
   185361 Coque Personnalisée/189893524_a3bf8c22eb_o.jpg
  4190069 Coque Personnalisée/7661299312_042178f60d_o.jpg
  1433088 Coque Personnalisée/water-sails-rope-sunset-sea-cruise.jpg
  1243983 Coque Personnalisée/halyard-rope-boat-sailboat-strings-sea.jpg
  2155261 Coque Personnalisée/dew-rope-rigging-cordage-pods-sailing-vessel-ship.jpg

but rclone ls Next:ulysse132/files/Coque\ Personnalisée/ doesn't return any file

Can you post your config with the secrets removed - that will help me work out what is happening - thank you

[S3]
type = s3
provider = Other
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
endpoint = s3.pub1.infomaniak.cloud
acl = private
bucket_acl = private
force_path_style = true
v2_auth = true

[Nextcloud]
type = webdav
url = https://nextcloud.xxx/remote.php/dav/files/ulysse132/
vendor = nextcloud
user = ulysse132
pass = xxxx

[Next]
type = crypt
remote = S3:next
filename_encryption = obfuscate
password = xxxx

[S3] is my provider
[Next] is the encrypted folder of S3
[Nextcloud] is my nextcloud's webdav server for my user

It might be worth trying

filename_encryption = standard

since the provider seems to be having problem with unicode characters and obfuscate uses unicode characters.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.