ulysse132
(Eric de QUIRPH)
March 2, 2022, 5:51am
1
What is the problem you are having with rclone?
rclone can't list files when the folder's name contains accent (é/è/à/ç...)
Run the command 'rclone version' and share the full output of the command.
rclone v1.57.0
os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.4.0-100-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.17.2
go/linking: static
go/tags: none
Which cloud storage system are you using? (eg Google Drive)
S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone ls S3:nextcloud/ulysse132/files/Téléphones/
same behavior with rclone mount, the files doesn't appear on the directory tree
The rclone config contents with secrets removed.
[S3]
type = s3
provider = Other
env_auth = false
access_key_id =
secret_access_key =
endpoint = s3.pub1.infomaniak.cloud
acl = private
bucket_acl = private
force_path_style = true
v2_auth = true
A log from the command with the -vv
flag
rclone ls S3:nextcloud/ulysse132/files/Téléphones/ -vv
<7>DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "ls" "S3:nextcloud/ulysse132/files/Téléphones/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/Téléphones/"
<7>DEBUG : Using config file from "/home/ulysse132/.config/rclone/rclone.conf"
<7>DEBUG : Using v2 auth
<7>DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/Téléphones/" to be canonical "S3:nextcloud/ulysse132/files/Téléphones"
<7>DEBUG : 4 go routines active
rclone lsd S3:nextcloud/ulysse132/files/ -vv
<7>DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsd" "S3:nextcloud/ulysse132/files/" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/"
<7>DEBUG : Using config file from "/home/ulysse132/.config/rclone/rclone.conf"
<7>DEBUG : Using v2 auth
<7>DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/" to be canonical "S3:nextcloud/ulysse132/files"
0 2022-03-02 06:48:29 -1 .thumbnails
0 2022-03-02 06:48:29 -1 Modèles
0 2022-03-02 06:48:29 -1 Oliver
0 2022-03-02 06:48:29 -1 Téléphones
0 2022-03-02 06:48:29 -1 etc
<7>DEBUG : 4 go routines active
Is there any way to configure rclone to accept accent ? It's not possible to rename each folder, as I will not the only one using rclone mount as nextcloud backend.
Thanks a lot for your help !
ncw
(Nick Craig-Wood)
March 2, 2022, 10:35am
2
Rclone should be working fine with accents.
$ echo hello | rclone rcat s3:rclone/nextcloud/ulysse132/files/Téléphones/file.txt
$ rclone ls s3:rclone/nextcloud/ulysse132/files/Téléphones/
6 file.txt
Are you sure there are files in the directory nextcloud/ulysse132/files/Téléphones/
?
What does this do - does it list into the Téléphones
directory?
rclone ls S3:nextcloud/ulysse132/files/
I guess this could be something to do with your provider? I tried AWS above.
ulysse132
(Eric de QUIRPH)
March 2, 2022, 1:14pm
3
I'm 100% sure it is not empty, and this for 2 reasons :
Here is a screenshot of infomaniak public cloud manager, where you can si 3 folders
The files listed here where uploaded through rclone using nextcloud
I'm not using the storage backend without using rclone, this way, it is always up-to-date, normally...
ncw
(Nick Craig-Wood)
March 2, 2022, 1:50pm
4
Can you try
Also could the directory name have a trailing space?
ulysse132
(Eric de QUIRPH)
March 2, 2022, 4:05pm
5
In the folder Oliver, I have plenty of files, which means it takes a very long time to list. But inside this folder, I also have other folder with accent. I ran :
rclone lsd S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/
0 2022-03-02 16:58:46 -1 Etiquette liqueur banane
0 2022-03-02 16:58:46 -1 Extrudeuse
0 2022-03-02 16:58:46 -1 Impressions 3D
0 2022-03-02 16:58:46 -1 Meuble Salon
0 2022-03-02 16:58:46 -1 PID amélioré
0 2022-03-02 16:58:46 -1 Ventilo congélateur
0 2022-03-02 16:58:46 -1 Vélo Bambou - Vélo couché
0 2022-03-02 16:58:46 -1 Zigbee2MQTT
rclone lsd S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/
has no result just like rclone ls S3:nextcloud/ulysse132/files/Téléphones/
but
rclone ls S3:nextcloud/ulysse132/files/
...
202849 Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/HTML/Velo/Vélo couché/www.atomiczombie.com/wp-content/themes/RadBrad/images/ads/6 Pack Special.png
...
wich means it list inside the directory. And it's not related to trailing space with this example;
ncw
(Nick Craig-Wood)
March 2, 2022, 10:52pm
6
Good we have an example to play with
Can you run
rclone lsd -vv --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/
And post (or attach) the output so I can analyse what is going on.
Thank you
ulysse132
(Eric de QUIRPH)
March 3, 2022, 5:54am
7
rclone lsd -vv --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\ couché/
2022/03/03 06:51:41 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "lsd" "-vv" "--dump" "bodies" "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/"]
2022/03/03 06:51:41 DEBUG : Creating backend with remote "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/"
2022/03/03 06:51:41 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/03/03 06:51:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/03/03 06:51:41 DEBUG : Using v2 auth
2022/03/03 06:51:41 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/03/03 06:51:41 DEBUG : fs cache: renaming cache item "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/" to be canonical "S3:nextcloud/ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché"
2022/03/03 06:51:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/03/03 06:51:41 DEBUG : HTTP REQUEST (req 0xc000593400)
2022/03/03 06:51:41 DEBUG : GET /nextcloud?delimiter=%2F&max-keys=1000&prefix=ulysse132%2Ffiles%2FOliver%2FDocuments%2FProjets+personnels%2FV%C3%A9lo+Bambou+-+V%C3%A9lo+couch%C3%A9%2F HTTP/1.1
Host: s3.pub1.infomaniak.cloud
User-Agent: rclone/v1.57.0
Authorization: XXXX
Date: Thu, 03 Mar 2022 05:51:41 UTC
Accept-Encoding: gzip
2022/03/03 06:51:41 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/03/03 06:51:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/03/03 06:51:41 DEBUG : HTTP RESPONSE (req 0xc000593400)
2022/03/03 06:51:41 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 03 Mar 2022 05:51:41 GMT
Strict-Transport-Security: max-age=63072000
X-Amz-Id-2: tx0ad01b16890b4d7b8f232-006220576d
X-Amz-Request-Id: tx0ad01b16890b4d7b8f232-006220576d
X-Openstack-Request-Id: tx0ad01b16890b4d7b8f232-006220576d
X-Trans-Id: tx0ad01b16890b4d7b8f232-006220576d
152
<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>nextcloud</Name><Prefix>ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/</Prefix><Marker/><MaxKeys>1000</MaxKeys><Delimiter>/</Delimiter><IsTruncated>false</IsTruncated></ListBucketResult>
0
2022/03/03 06:51:41 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/03/03 06:51:41 DEBUG : 4 go routines active
ncw
(Nick Craig-Wood)
March 3, 2022, 9:17am
8
There is something weird going on with the encoding here...
So where did that weird encoding come from?
Rclone sent this
If I decode that URL parameter with python it looks correct
>>> import urllib.parse
urllib.parse.unquote_plus('ulysse132%2Ffiles%2FOliver%2FDocuments%2FProjets+personnels%2FV%C3%A9lo+Bambou+-+V%C3%A9lo+couch%C3%A9%2F')
'ulysse132/files/Oliver/Documents/Projets personnels/Vélo Bambou - Vélo couché/'
>>>
So what it looks like is that your provider decoded the URL but interpreted it as latin1 not utf-8
Just to double check the encodings didn't get messed up in the cut and paste can you run the above to a log file and attach the log file here?
rclone lsd -vv --log-file rclone.log --dump bodies S3:nextcloud/ulysse132/files/Oliver/Documents/Projets\ personnels/Vélo\ Bambou\ -\ Vélo\
Thank you
ulysse132
(Eric de QUIRPH)
March 3, 2022, 9:56am
9
Here you are
rclone.log (3.8 KB)
Some side information : infomaniak Public Cloud (my provider) is build on OpenStack. I connect using S3 compatible credentials. May be the problem comes from here. I already tried to connect using swift credentials, but I don't succeed. I asked for help to infomaniak.
ncw
(Nick Craig-Wood)
March 3, 2022, 10:20am
10
Thanks for that.
Yes, it looks the same in the rclone.log
file - thanks for the confirmation.
I think this is some kind of bug in the cloud provider - a confusion of encodings at some point, but I can't tell exactly where.
ulysse132
(Eric de QUIRPH)
March 3, 2022, 10:46am
11
So, there is nothing to do ?
That's weird, because these folders were created using rclone...
If the help I asked for to infomaniak don't succeed, I'll move to another solution, such as seafile, where my files are stored using only alphanumerical characters
ncw
(Nick Craig-Wood)
March 3, 2022, 3:07pm
12
If you were to set up a crypt backend this will hide the file names from the provider and it will work properly.
ulysse132
(Eric de QUIRPH)
March 4, 2022, 8:21am
13
I really thought it will work but sadly not, and I really don't understand why...
I rebuild my entire server, create an encrypt backend to a new bucket, put some files and try. After 2h, if I rescan the folders, I can't find any file.
rclone lsd Nextcloud:
-1 2022-03-04 07:15:16 -1 Coque Personnalisée
-1 2022-03-04 07:20:58 -1 Documents
-1 2022-03-04 07:03:59 -1 Modèles
-1 2022-03-04 07:05:03 -1 Photos
-1 2022-03-04 07:51:00 -1 Téléphones
rclone ls Next:ulysse132/files/
567689 Photos/Steps.jpg
427030 Photos/Vineyard.jpg
593508 Photos/Birdie.jpg
167989 Photos/Toucan.jpg
457744 Photos/Frog.jpg
797325 Photos/Nextcloud community.jpg
474653 Photos/Gorilla.jpg
2170375 Photos/Library.jpg
150 Photos/Readme.md
3338 Photos/photos endommagées.txt
11712256 Nextcloud Manual.pdf
25150 Documents/Welcome to Nextcloud Hub.docx
1095 Documents/Example.md
976625 Documents/Raisons d'utiliser Nextcloud.pdf
136 Documents/Readme.md
374008 Documents/Nextcloud flyer.pdf
1288940 91.jpg
1429172 Coque Personnalisée/clipper-ship-tall-masts-sailing-nautical.jpg
89885 Coque Personnalisée/6031905039_65ce7e3d47_b.jpg
1000812 Coque Personnalisée/sailing-old-rig-sea-water-sailing-vessel-port-2.jpg
537912 Coque Personnalisée/3953885706_edd02280c0_o.jpg
1621283 Coque Personnalisée/rope-boat-sea-rigging.jpg
2112200 Coque Personnalisée/158939597_3b18cd3f3e_o.jpg
185361 Coque Personnalisée/189893524_a3bf8c22eb_o.jpg
4190069 Coque Personnalisée/7661299312_042178f60d_o.jpg
1433088 Coque Personnalisée/water-sails-rope-sunset-sea-cruise.jpg
1243983 Coque Personnalisée/halyard-rope-boat-sailboat-strings-sea.jpg
2155261 Coque Personnalisée/dew-rope-rigging-cordage-pods-sailing-vessel-ship.jpg
but rclone ls Next:ulysse132/files/Coque\ Personnalisée/
doesn't return any file
ncw
(Nick Craig-Wood)
March 4, 2022, 11:28am
14
Can you post your config with the secrets removed - that will help me work out what is happening - thank you
ulysse132
(Eric de QUIRPH)
March 4, 2022, 12:59pm
15
[S3]
type = s3
provider = Other
env_auth = false
access_key_id = xxxx
secret_access_key = xxxx
endpoint = s3.pub1.infomaniak.cloud
acl = private
bucket_acl = private
force_path_style = true
v2_auth = true
[Nextcloud]
type = webdav
url = https://nextcloud.xxx/remote.php/dav/files/ulysse132/
vendor = nextcloud
user = ulysse132
pass = xxxx
[Next]
type = crypt
remote = S3:next
filename_encryption = obfuscate
password = xxxx
[S3] is my provider
[Next] is the encrypted folder of S3
[Nextcloud] is my nextcloud's webdav server for my user
ncw
(Nick Craig-Wood)
March 4, 2022, 2:17pm
16
It might be worth trying
filename_encryption = standard
since the provider seems to be having problem with unicode characters and obfuscate
uses unicode characters.
system
(system)
Closed
April 3, 2022, 2:18pm
17
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.