Problem syncing S3 bucket to another bucket

enboig · April 16, 2021, 12:18pm

What is the problem you are having with rclone?

I have a S3 account (wasabi), and I want to move one bucket from one location to another. To do so I try to clone all objects between them, but I cannot make it work.

What is your rclone version (output from `rclone version`)

rclone v1.55.0

os/type: linux
os/arch: amd64
go/version: go1.16.2
go/linking: static
go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 64 bits

Which cloud storage system are you using? (eg Google Drive)

S3-Wasabi

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

$ rclone sync --dry-run wasabi:enboig.restic/ wasabisys:enboig.restic-eu/

The rclone config contents with secrets removed.

[wasabi]
type = s3
env_auth = false
access_key_id = ***ROOT ACCOUNT***
secret_access_key = ***ROOT ACCOUNT***
region = us-east-1
endpoint = s3.wasabisys.com
location_constraint = 
acl = 
server_side_encryption = 
storage_class = 

[wasabisys]
type = s3
provider = Wasabi
env_auth = false
access_key_id = ***ROOT ACCOUNT***
secret_access_key = ***ROOT ACCOUNT***
region = eu-central-1
endpoint = s3.eu-central-1.wasabisys.com

A log from the command with the `-vv` flag

2021/04/16 14:14:09 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 19354DDBC0FC3244, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:09 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 19354DDBC0FC3244, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 5F5DDAE86F39DC3B, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:09 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 5F5DDAE86F39DC3B, host id: ***HOST_ID***
2021/04/16 14:14:10 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***
2021/04/16 14:14:10 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:10 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:10 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***
2021/04/16 14:14:10 NOTICE: 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         2.0s

2021/04/16 14:14:10 Failed to sync: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***

asdffdsa · April 16, 2021, 12:41pm

hello,
can you post the complete debg log, or at least the top lines including the exact command

enboig · April 16, 2021, 1:02pm

I thought it was enough with the log I posted

enboig@enboig-GL552VW:~$ rclone sync -vv wasabi:enboig.restic/ wasabisys:enboig.restic-eu/
2021/04/16 14:59:36 DEBUG : Using config file from "/home/enboig/.config/rclone/rclone.conf"
2021/04/16 14:59:36 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "-vv" "wasabi:enboig.restic/" "wasabisys:enboig.restic-eu/"]
2021/04/16 14:59:36 DEBUG : Creating backend with remote "wasabi:enboig.restic/"
2021/04/16 14:59:36 DEBUG : fs cache: renaming cache item "wasabi:enboig.restic/" to be canonical "wasabi:enboig.restic"
2021/04/16 14:59:36 DEBUG : Creating backend with remote "wasabisys:enboig.restic-eu/"
2021/04/16 14:59:36 DEBUG : fs cache: renaming cache item "wasabisys:enboig.restic-eu/" to be canonical "wasabisys:enboig.restic-eu"
2021/04/16 14:59:38 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 28E891F5C3CA555A, host id: whEVlPM7DjJsCkYQmpWVgkcIqsxSjU/eGS1vARjAjvH+sf9THuMD3xk0WJ43PicszROMnUUGnhDA

It appear it is ignoring the user. I tried accessing with rclone rcd --rc-web-gui; and I can list the buckets (all of them), but when I try to browse them nothing is listed.

asdffdsa · April 16, 2021, 1:11pm

i have used wasabi for many years.

not sure why you are getting 403 permission errors.

i always use endpoint, never used region
https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions-

enboig · April 16, 2021, 1:17pm

I think my problem is because the key isn't used, I have set a token so access sholdn't be anonymous

ncw · April 17, 2021, 4:13pm

That looks like the error.

Can you list the buckets with

rclone lsf wasabi:enboig.restic

and

wasabisys:enboig.restic-eu

enboig · April 17, 2021, 5:29pm

No, I can't. The error complains about "Anonymous user"

$ rclone lsf wasabi:enboig.restic
2021/04/17 19:28:09 ERROR : : error listing: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 540A2EA049937B3F, host id: BajtHb6EOK7IHxnnxQX4bl4Na8SxZzLdem+laZxoNkA58TS8UIj17WBJPchuZhPlQcID80p5gpXl
2021/04/17 19:28:09 Failed to lsf with 2 errors: last error was: error in ListJSON: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 540A2EA049937B3F, host id: BajtHb6EOK7IHxnnxQX4bl4Na8SxZzLdem+laZxoNkA58TS8UIj17WBJPchuZhPlQcID80p5gpXl

My user policy is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::restic"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::enboig.restic-eu",
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::984609348793:user/restic"
      },
      "Action": [
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::enboig.restic-eu",
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic"
      ]
    }
  ]
}

ncw · April 18, 2021, 9:39am

That's the problem we need to solve.

First can you check whether it really is using an anonymous user.

Can you do rclone lsf wasabi:enboig.restic -vv --dump bodies --low-level-retries 1 --retries 1 and post the output.

enboig · April 18, 2021, 10:02am

Here it is:

$ rclone lsf wasabi:enboig.restic -vv --dump bodies --low-level-retries 1 --retries 1
2021/04/18 11:58:54 DEBUG : Using config file from "/home/enboig/.config/rclone/rclone.conf"
2021/04/18 11:58:54 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "lsf" "wasabi:enboig.restic" "-vv" "--dump" "bodies" "--low-level-retries" "1" "--retries" "1"]
2021/04/18 11:58:54 DEBUG : Creating backend with remote "wasabi:enboig.restic"
2021/04/18 11:58:54 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/18 11:58:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:54 DEBUG : HTTP REQUEST (req 0xc0003cf100)
2021/04/18 11:58:54 DEBUG : GET /enboig.restic?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1
Host: s3.wasabisys.com
User-Agent: rclone/v1.55.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210418T095854Z
Accept-Encoding: gzip

2021/04/18 11:58:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:55 DEBUG : HTTP RESPONSE (req 0xc0003cf100)
2021/04/18 11:58:55 DEBUG : HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 18 Apr 2021 09:58:55 GMT
Location: https://s3.us-west-1.wasabisys.com/enboig.restic?delimiter=%2F&max-keys=1000&prefix=
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head12)
X-Amz-Bucket-Region: us-west-1
X-Amz-Id-2: 6v66rwCn7j5V5oyQYfMA2wls8g/gTVGWp4i8r92IOEV3h2nRCEYtGDDt2pEwSO2Nt0tcED1KrRl0
X-Amz-Request-Id: 1D89FF58E0E710A5

1c1
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>enboig.restic</Bucket><Endpoint>s3.us-west-1.wasabisys.com</Endpoint><RequestId>1D89FF58E0E710A5</RequestId><HostId>6v66rwCn7j5V5oyQYfMA2wls8g/gTVGWp4i8r92IOEV3h2nRCEYtGDDt2pEwSO2Nt0tcED1KrRl0</HostId></Error>
0

2021/04/18 11:58:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:55 DEBUG : HTTP REQUEST (req 0xc000436100)
2021/04/18 11:58:55 DEBUG : GET /enboig.restic?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.55.0
Referer: https://s3.wasabisys.com/enboig.restic?delimiter=%2F&max-keys=1000&prefix=
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210418T095854Z
Accept-Encoding: gzip

2021/04/18 11:58:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:56 DEBUG : HTTP RESPONSE (req 0xc000436100)
2021/04/18 11:58:56 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 18 Apr 2021 09:58:56 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head05)
X-Amz-Bucket-Region: us-west-1
X-Amz-Id-2: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT
X-Amz-Request-Id: 2D5B7BEF51CF8199

120
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Anonymous user cannot access public access disabled bucket</Message><RequestId>2D5B7BEF51CF8199</RequestId><HostId>x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT</HostId></Error>
0

2021/04/18 11:58:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:56 ERROR : : error listing: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 2D5B7BEF51CF8199, host id: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT
2021/04/18 11:58:56 DEBUG : 6 go routines active
2021/04/18 11:58:56 Failed to lsf with 2 errors: last error was: error in ListJSON: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 2D5B7BEF51CF8199, host id: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT

I think the problem is a endpoint is required.... I will check in some hours

ncw · April 18, 2021, 10:08am

It is clear what is going on with that.

The first request has the Authorization: XXXX header. The response to this is a 301 redirect. Unfortunately this kind of redirect doesn't preserve the extra headers - they wanted a 307/308 redirect for that)

So when rclone retries at the new location it retries without Auth hence the anonymous message.

So I think changing the endpoint to endpoint = s3.us-west-1.wasabisys.com will probably fix the problem.

Also we should encourage Wasabi to use a 307/308 redirect instead of 301.

ncw · April 18, 2021, 10:30am

On further reflection I think Wasabi are donig the right thing as it is a GET request. The Auth is going missing because of rclone or the AWS SDK...

Either way, fixing up the endpoint should fix the problem.

enboig · April 18, 2021, 11:41am

Thanks for your help, now it works perfectly. When I first configured rclone and wasabi they just had one endpoint, so everything was working perfectly.
I will try to reach them to change the redirect.

enboig · April 18, 2021, 11:44am

When using restic (https://restic.net/), I don't specify any endpoint; I don't know how they make it work.

ncw · April 18, 2021, 7:45pm

I think restic must have the main endpoint built in?

Are there endpoints that rclone offers in rclone config correct for Wasabi?

asdffdsa · April 18, 2021, 7:49pm

rclone is missing some endpoints

https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions

ncw · April 18, 2021, 7:50pm

Fancy sending a pr to fix?

asdffdsa · April 18, 2021, 7:51pm

sure, i will do that

system · June 18, 2021, 3:52pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.