Problem syncing S3 bucket to another bucket

What is the problem you are having with rclone?

I have a S3 account (wasabi), and I want to move one bucket from one location to another. To do so I try to clone all objects between them, but I cannot make it work.

What is your rclone version (output from rclone version)

rclone v1.55.0

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.2
  • go/linking: static
  • go/tags: cmount

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04 64 bits

Which cloud storage system are you using? (eg Google Drive)

S3-Wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

$ rclone sync --dry-run wasabi:enboig.restic/ wasabisys:enboig.restic-eu/

The rclone config contents with secrets removed.

[wasabi]
type = s3
env_auth = false
access_key_id = ***ROOT ACCOUNT***
secret_access_key = ***ROOT ACCOUNT***
region = us-east-1
endpoint = s3.wasabisys.com
location_constraint = 
acl = 
server_side_encryption = 
storage_class = 

[wasabisys]
type = s3
provider = Wasabi
env_auth = false
access_key_id = ***ROOT ACCOUNT***
secret_access_key = ***ROOT ACCOUNT***
region = eu-central-1
endpoint = s3.eu-central-1.wasabisys.com


A log from the command with the -vv flag

2021/04/16 14:14:09 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 19354DDBC0FC3244, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:09 ERROR : Attempt 1/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 19354DDBC0FC3244, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 5F5DDAE86F39DC3B, host id: ***HOST_ID***
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:09 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:09 ERROR : Attempt 2/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 5F5DDAE86F39DC3B, host id: ***HOST_ID***
2021/04/16 14:14:10 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***
2021/04/16 14:14:10 ERROR : S3 bucket enboig.restic-eu: not deleting files as there were IO errors
2021/04/16 14:14:10 ERROR : S3 bucket enboig.restic-eu: not deleting directories as there were IO errors
2021/04/16 14:14:10 ERROR : Attempt 3/3 failed with 1 errors and: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***
2021/04/16 14:14:10 NOTICE: 
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         2.0s

2021/04/16 14:14:10 Failed to sync: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: EFB62608816D3A6D, host id: ***HOST_ID***

hello,
can you post the complete debg log, or at least the top lines including the exact command

I thought it was enough with the log I posted

enboig@enboig-GL552VW:~$ rclone sync -vv wasabi:enboig.restic/ wasabisys:enboig.restic-eu/
2021/04/16 14:59:36 DEBUG : Using config file from "/home/enboig/.config/rclone/rclone.conf"
2021/04/16 14:59:36 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "sync" "-vv" "wasabi:enboig.restic/" "wasabisys:enboig.restic-eu/"]
2021/04/16 14:59:36 DEBUG : Creating backend with remote "wasabi:enboig.restic/"
2021/04/16 14:59:36 DEBUG : fs cache: renaming cache item "wasabi:enboig.restic/" to be canonical "wasabi:enboig.restic"
2021/04/16 14:59:36 DEBUG : Creating backend with remote "wasabisys:enboig.restic-eu/"
2021/04/16 14:59:36 DEBUG : fs cache: renaming cache item "wasabisys:enboig.restic-eu/" to be canonical "wasabisys:enboig.restic-eu"
2021/04/16 14:59:38 ERROR : : error reading source directory: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 28E891F5C3CA555A, host id: whEVlPM7DjJsCkYQmpWVgkcIqsxSjU/eGS1vARjAjvH+sf9THuMD3xk0WJ43PicszROMnUUGnhDA

It appear it is ignoring the user. I tried accessing with rclone rcd --rc-web-gui; and I can list the buckets (all of them), but when I try to browse them nothing is listed.

i have used wasabi for many years.

not sure why you are getting 403 permission errors.

i always use endpoint, never used region
https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions-

I think my problem is because the key isn't used, I have set a token so access sholdn't be anonymous

That looks like the error.

Can you list the buckets with

rclone lsf wasabi:enboig.restic

and

wasabisys:enboig.restic-eu

No, I can't. The error complains about "Anonymous user"

$ rclone lsf wasabi:enboig.restic
2021/04/17 19:28:09 ERROR : : error listing: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 540A2EA049937B3F, host id: BajtHb6EOK7IHxnnxQX4bl4Na8SxZzLdem+laZxoNkA58TS8UIj17WBJPchuZhPlQcID80p5gpXl
2021/04/17 19:28:09 Failed to lsf with 2 errors: last error was: error in ListJSON: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 540A2EA049937B3F, host id: BajtHb6EOK7IHxnnxQX4bl4Na8SxZzLdem+laZxoNkA58TS8UIj17WBJPchuZhPlQcID80p5gpXl

My user policy is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": [
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::restic"
      ]
    },
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::enboig.restic-eu",
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic"
      ]
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::984609348793:user/restic"
      },
      "Action": [
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetObject",
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Resource": [
        "arn:aws:s3:::enboig.restic-eu/*",
        "arn:aws:s3:::enboig.restic-eu",
        "arn:aws:s3:::enboig.restic/*",
        "arn:aws:s3:::enboig.restic"
      ]
    }
  ]
}

That's the problem we need to solve.

First can you check whether it really is using an anonymous user.

Can you do rclone lsf wasabi:enboig.restic -vv --dump bodies --low-level-retries 1 --retries 1 and post the output.

Here it is:

$ rclone lsf wasabi:enboig.restic -vv --dump bodies --low-level-retries 1 --retries 1
2021/04/18 11:58:54 DEBUG : Using config file from "/home/enboig/.config/rclone/rclone.conf"
2021/04/18 11:58:54 DEBUG : rclone: Version "v1.55.0" starting with parameters ["rclone" "lsf" "wasabi:enboig.restic" "-vv" "--dump" "bodies" "--low-level-retries" "1" "--retries" "1"]
2021/04/18 11:58:54 DEBUG : Creating backend with remote "wasabi:enboig.restic"
2021/04/18 11:58:54 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/04/18 11:58:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:54 DEBUG : HTTP REQUEST (req 0xc0003cf100)
2021/04/18 11:58:54 DEBUG : GET /enboig.restic?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1
Host: s3.wasabisys.com
User-Agent: rclone/v1.55.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210418T095854Z
Accept-Encoding: gzip

2021/04/18 11:58:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:55 DEBUG : HTTP RESPONSE (req 0xc0003cf100)
2021/04/18 11:58:55 DEBUG : HTTP/1.1 301 Moved Permanently
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 18 Apr 2021 09:58:55 GMT
Location: https://s3.us-west-1.wasabisys.com/enboig.restic?delimiter=%2F&max-keys=1000&prefix=
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head12)
X-Amz-Bucket-Region: us-west-1
X-Amz-Id-2: 6v66rwCn7j5V5oyQYfMA2wls8g/gTVGWp4i8r92IOEV3h2nRCEYtGDDt2pEwSO2Nt0tcED1KrRl0
X-Amz-Request-Id: 1D89FF58E0E710A5

1c1
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>PermanentRedirect</Code><Message>The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.</Message><Bucket>enboig.restic</Bucket><Endpoint>s3.us-west-1.wasabisys.com</Endpoint><RequestId>1D89FF58E0E710A5</RequestId><HostId>6v66rwCn7j5V5oyQYfMA2wls8g/gTVGWp4i8r92IOEV3h2nRCEYtGDDt2pEwSO2Nt0tcED1KrRl0</HostId></Error>
0

2021/04/18 11:58:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:55 DEBUG : HTTP REQUEST (req 0xc000436100)
2021/04/18 11:58:55 DEBUG : GET /enboig.restic?delimiter=%2F&max-keys=1000&prefix= HTTP/1.1
Host: s3.us-west-1.wasabisys.com
User-Agent: rclone/v1.55.0
Referer: https://s3.wasabisys.com/enboig.restic?delimiter=%2F&max-keys=1000&prefix=
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20210418T095854Z
Accept-Encoding: gzip

2021/04/18 11:58:55 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/04/18 11:58:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:56 DEBUG : HTTP RESPONSE (req 0xc000436100)
2021/04/18 11:58:56 DEBUG : HTTP/1.1 403 Forbidden
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Sun, 18 Apr 2021 09:58:56 GMT
Server: WasabiS3/6.2.4542-2021-04-06-384c1a6 (head05)
X-Amz-Bucket-Region: us-west-1
X-Amz-Id-2: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT
X-Amz-Request-Id: 2D5B7BEF51CF8199

120
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Anonymous user cannot access public access disabled bucket</Message><RequestId>2D5B7BEF51CF8199</RequestId><HostId>x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT</HostId></Error>
0

2021/04/18 11:58:56 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/04/18 11:58:56 ERROR : : error listing: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 2D5B7BEF51CF8199, host id: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT
2021/04/18 11:58:56 DEBUG : 6 go routines active
2021/04/18 11:58:56 Failed to lsf with 2 errors: last error was: error in ListJSON: AccessDenied: Anonymous user cannot access public access disabled bucket
        status code: 403, request id: 2D5B7BEF51CF8199, host id: x+xUwQmFpWx+utGLMO2OQ4M2LdVs36jLeymr8Kb9g5UXHzWHBSpNNZRxJ44BbF+Wcvnpq+Jg6DNT

I think the problem is a endpoint is required.... I will check in some hours

It is clear what is going on with that.

The first request has the Authorization: XXXX header. The response to this is a 301 redirect. Unfortunately this kind of redirect doesn't preserve the extra headers - they wanted a 307/308 redirect for that)

So when rclone retries at the new location it retries without Auth hence the anonymous message.

So I think changing the endpoint to endpoint = s3.us-west-1.wasabisys.com will probably fix the problem.

Also we should encourage Wasabi to use a 307/308 redirect instead of 301.

On further reflection I think Wasabi are donig the right thing as it is a GET request. The Auth is going missing because of rclone or the AWS SDK...

Either way, fixing up the endpoint should fix the problem.

Thanks for your help, now it works perfectly. When I first configured rclone and wasabi they just had one endpoint, so everything was working perfectly.
I will try to reach them to change the redirect.

2 Likes

When using restic (https://restic.net/), I don't specify any endpoint; I don't know how they make it work.

I think restic must have the main endpoint built in?

Are there endpoints that rclone offers in rclone config correct for Wasabi?

rclone is missing some endpoints

https://wasabi-support.zendesk.com/hc/en-us/articles/360015106031-What-are-the-service-URLs-for-Wasabi-s-different-storage-regions

Fancy sending a pr to fix?

sure, i will do that

1 Like