Rclone delete does not work on scaleway (S3) remote

J_B · June 13, 2022, 3:54pm

Hi,
rclone delete & ls actions does not show any output (and does not work) while deletefile is working as expected on scaleway (S3) remote.
rclone sync does not remove excluded or not existing files.

Rclone version:

rclone v1.58.1
- os/version: alpine 3.9.4 (64 bit)
- os/kernel: 5.10.104-linuxkit (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none

Simple delete & lsl failling examples:

rclone delete remote:/mypath --min-age 2d

or

rclone lsl remote:/mypath

Deletefile sucessful example:

rclone deletefile remote:mypath/test.txt

Actual remote config:

bash-4.4# rclone config dump
{
    "scaleway": {
        "access_key_id": "xxxxxxxxxxxx",
        "acl": "private",
        "endpoint": "xxxxxxxxxxxx.s3.fr-par.scw.cloud",
        "env_auth": "false",
        "location_constraint": "",
        "provider": "Scaleway",
        "region": "fr-par",
        "secret_access_key": "xxxxxxxxxxxxxx",
        "server_side_encryption": "",
        "storage_class": "STANDARD",
        "type": "s3"
    }
}

Thank you for the help!

asdffdsa · June 13, 2022, 5:01pm

hello,
please post debug logs by adding -vv to the commands.

J_B · June 13, 2022, 5:14pm

Hello,

Here it is:

For delete

2022/06/13 19:11:34 DEBUG : --min-age 2d to 2022-06-11 19:11:34.2259092 +0200 CEST m=-172799.962570499
2022/06/13 19:11:34 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "delete" "remote:xxxxxxxxxxxxxxxx" "--min-age=2d" "-vvv"]
2022/06/13 19:11:34 DEBUG : Creating backend with remote "remote:xxxxxxxxxxxxxxxx"
2022/06/13 19:11:34 DEBUG : Using RCLONE_CONFIG_PASS password.
2022/06/13 19:11:34 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/06/13 19:11:34 DEBUG : Waiting for deletions to finish
2022/06/13 19:11:34 DEBUG : 3 go routines active

For lsl

2022/06/13 19:13:14 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsl" "remote:xxxxxxxxxxxxxxxx" "-vvv"]
2022/06/13 19:13:14 DEBUG : Creating backend with remote "remote:xxxxxxxxxxxxxxxx"
2022/06/13 19:13:14 DEBUG : Using RCLONE_CONFIG_PASS password.
2022/06/13 19:13:14 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/06/13 19:13:15 DEBUG : 3 go routines active

Thx !

asdffdsa · June 13, 2022, 5:17pm

kind of confusing,

what is the exact name of the remote?
scaleway or remote

what is the bucket name?
mypath, xxxxxxxxxxxxxxxx or what?

might be a small detail, but the commands that do not work have a slash
remote:/mypath
and the command that does work, does not have that slash
remote:mypath/test.txt

J_B · June 13, 2022, 5:47pm

It was for the example, in my case the remote is named scaleway.
You can consider the name of the bucket as "xxxxxxxxxxxxxxxx"

I tried both commands, with and without the first "/", rclone seems to automatically remove slashes in canonical butcket name.

asdffdsa · June 13, 2022, 5:54pm

ok, to keep this simple, let's use scaleway:xxx

can you post debug logs for
rclone lsl scaleway:xxx -vv which will not list test.txt
then
rclone deletefile scaleway:xxx/test.txt -vv which will delete that file.

J_B · June 13, 2022, 10:14pm

Sorry for the late reply.

rclone lsl output

rclone lsl scaleway:/xxxxxxxxxxxxxxxx -vv

2022/06/14 00:08:31 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsl" "scaleway:/xxxxxxxxxxxxxxxx" "-vv"]
2022/06/14 00:08:31 DEBUG : Creating backend with remote "scaleway:/xxxxxxxxxxxxxxxx"
2022/06/14 00:08:31 DEBUG : Using RCLONE_CONFIG_PASS password.
2022/06/14 00:08:31 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/06/14 00:08:31 DEBUG : fs cache: renaming cache item "scaleway:/xxxxxxxxxxxxxxxx" to be canonical "scaleway:xxxxxxxxxxxxxxxx"
2022/06/14 00:08:32 DEBUG : 3 go routines active

rclone deletefile output

rclone deletefile scaleway:/xxxxxxxxxxxxxxxx/test.txt -vv

2022/06/14 00:11:57 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "deletefile" "scaleway:/xxxxxxxxxxxxxxxx/test.txt" "-vv"]
2022/06/14 00:11:57 DEBUG : Creating backend with remote "scaleway:/xxxxxxxxxxxxxxxx/test.txt"
2022/06/14 00:11:57 DEBUG : Using RCLONE_CONFIG_PASS password.
2022/06/14 00:11:57 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/06/14 00:12:02 DEBUG : fs cache: adding new entry for parent of "scaleway:/xxxxxxxxxxxxxxxx/test.txt", "scaleway:xxxxxxxxxxxxxxxx"
2022/06/14 00:12:08 INFO  : test.txt: Deleted
2022/06/14 00:12:08 DEBUG : 3 go routines active

asdffdsa · June 13, 2022, 10:21pm

really strange,

one thing i noticed about the endpoint.
--- rclone docs has s3.fr-par.scw.cloud
--- your config is xxxxxxxxxxxx.s3.fr-par.scw.cloud

J_B · June 14, 2022, 2:15pm

Hi,

Yes it's the endpoint recommended by scaleway.
I'm going test without the bucket name as subdomain in the endpoint config.
I will let you know if this changes anything.

Thank you for your time

asdffdsa · June 14, 2022, 2:30pm

might be an issue with virtual-host style versus path-style endpoint

i could be wrong but your config uses virtual-host, but rclone defaults to path-style.
fwiw, i always use virtual-host style

also, do you need to add https:// to the endpoint address?
https://forum.rclone.org/t/sync-copies-everything/25087

"An endpoint represents one end of a communication channel. In the case of Object Storage, your bucket endpoint is a URL eg https://my-bucket.s3.fr-par.scw.cloud which you use when connecting to your bucket.
Note that https://my-bucket.s3.fr-par.scw.cloud represents a virtual-host style endpoint, and
https://s3.fr-par.scw.cloud/my-bucket represents a path-style endpoint.
Both formats are valid and work with any up-to-date SDK or CLI, though virtual-host style is generally preferred."

J_B · June 14, 2022, 2:56pm

Just tried with both types of endpoint but still no output for lsl action or delete:

2022/06/14 16:54:36 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "lsl" "scaleway:xxxxxxxxxxxxxxxx" "-vvv"]
2022/06/14 16:54:36 DEBUG : Creating backend with remote "scaleway:xxxxxxxxxxxxxxxx"
2022/06/14 16:54:36 DEBUG : Using RCLONE_CONFIG_PASS password.
2022/06/14 16:54:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2022/06/14 16:54:36 DEBUG : 3 go routines active

asdffdsa · June 14, 2022, 2:59pm

but are you using the correct endpoint format, as per that link i shared?

J_B · June 14, 2022, 3:33pm

Wonderful!

I changed endpoint format to remove protocol and bucket name : "s3.fr-par.scw.cloud"
I also removed two empty config parameters:

location_constraint=
server_side_encryption=

It works like a charm!

Thank you very much for your help and for your time!

asdffdsa · June 14, 2022, 3:36pm

great!

can you share the working config.

J_B · June 14, 2022, 4:57pm

Here is the correct config.

### Working scaleway config ###
		
[scaleway]
type = s3
provider = Scaleway
env_auth = false
endpoint = s3.fr-par.scw.cloud
access_key_id = API_ACCESS_KEY_ID
secret_access_key = API_SECRET_ACCESS_KEY
region = fr-par
acl = private
storage_class = STANDARD

NB: remember to specify bucket name after remote name in rclone command

### Example for sync action ###

rclone sync /my/src/path/ scaleway:my_bucket_name/my_sub_path/

### Example for ls action ###

rclone lsl scaleway:my_bucket_name/my_sub_path/

system · June 17, 2022, 4:58pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.