Rclone with S3 localstack --> copy fails with Forbidden: Forbidden

Help and Support
1

What is the problem you are having with rclone?

rclone copy fails on a localstack S3 remote

Note that

  • lsd, mkdir work fine
  • I can copy using aws --endpoint-url=http://localstack:5000 s3 cp test/report.clj s3://toto/
  • I tried all s3 options one by one

What is your rclone version (output from rclone version)

tried with:

rclone v1.55.1
- os/type: linux
- os/arch: amd64
- go/version: go1.16.3
- go/linking: static
- go/tags: none

and

rclone v1.56.0-beta.5589.da36ce08e
- os/version: debian 9.11 (64 bit)
- os/kernel: 5.4.0-77-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.16.5
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

64 bit: ubuntu 20.04 but inside a docker container: Debian 9.11 on AMD64

Which cloud storage system are you using? (eg Google Drive)

localstack s3 as provided by:

  localstack:
    image: localstack/localstack:0.9.0
    environment:
      DEFAUT_REGION: us-east-1
      SERVICES: "s3:5000"
      HOSTNAME: "localstack"

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# create the remote
rclone config create localstack s3 region us-east-1 endpoint http://localstack:5000

# command that fails:
./rclone -vv copy  ./git-log.txt localstack:toto/

The rclone config contents with secrets removed.

[localstack]
type = s3
region = us-east-1
endpoint = http://localstack:5000

A log from the command with the -vv flag

2021/07/16 10:00:36 DEBUG : rclone: Version "v1.56.0-beta.5589.da36ce08e" starting with parameters ["./rclone" "-vv" "copy" "./git-log.txt" "localstack:toto/"]
2021/07/16 10:00:36 DEBUG : Creating backend with remote "./git-log.txt"
2021/07/16 10:00:36 DEBUG : Using config file from "/home/docker/.config/rclone/rclone.conf"
2021/07/16 10:00:36 DEBUG : fs cache: adding new entry for parent of "./git-log.txt", "/tmp/rclone-v1.56.0-beta.5589.da36ce08e-linux-amd64"
2021/07/16 10:00:36 DEBUG : Creating backend with remote "localstack:toto/"
2021/07/16 10:00:36 DEBUG : fs cache: renaming cache item "localstack:toto/" to be canonical "localstack:toto"
2021/07/16 10:00:36 ERROR : Attempt 1/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 10:00:36 ERROR : Attempt 2/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 10:00:36 ERROR : Attempt 3/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 10:00:36 INFO  : 
Transferred:   	          0 / 0 Byte, -, 0 Byte/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2021/07/16 10:00:36 DEBUG : 2 go routines active
2021/07/16 10:00:36 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:
2

hello and welcome to the forum,

so with localstack, no need for id/secret and it uses anonymous access?

can you run this command and post the rclone.log

./rclone copy ./git-log.txt localstack:toto --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

3

so with localstack, no need for id/secret and it uses anonymous access?

indeed!

I run it:

rclone copy backend2/config.json localstack:toto/ --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=/tmp/rclone.log

here's what seems to me the relevant part of the log:

<PutObjectResponse xmlns="http://s3.amazonaws.com/doc/2006-03-01"><PutObjectResponse><ETag>&#34;d4547195445dfdb50697c32c28af9dbc&#34;</ETag><LastModified>2021-07-16T14:27:34.436Z</LastModified></PutObjectResponse></PutObjectResponse>
2021/07/16 14:27:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:27:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:27:34 DEBUG : HTTP REQUEST (req 0xc00089a600)
2021/07/16 14:27:34 DEBUG : HEAD /toto/config.json HTTP/1.1
Host: localstack:5000
User-Agent: rclone/v1.55.1

2021/07/16 14:27:34 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:27:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:27:34 DEBUG : HTTP RESPONSE (req 0xc00089a600)
2021/07/16 14:27:34 DEBUG : HTTP/1.1 403 Forbidden
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 14:27:34 GMT
Server: BaseHTTP/0.3 Python/2.7.15
Content-Length: 0

2021/07/16 14:27:34 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:27:34 ERROR : config.json: Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 14:27:34 ERROR : Attempt 1/1 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 14:27:34 INFO  : 
Transferred:   	    5.200k / 5.200 kBytes, 100%, 340.169 kBytes/s, ETA 0s
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2021/07/16 14:27:34 DEBUG : 3 go routines active
2021/07/16 14:27:34 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:

tell met if you need the whole log

Thanks for helping!

4

can you post the full log

5

voila:
rclone copy ./world.txt localstack:toto --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

2021/07/16 14:40:25 DEBUG : Using config file from "/home/docker/.config/rclone/rclone.conf"
2021/07/16 14:40:25 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "./world.txt" "localstack:toto" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--log-file=rclone.log"]
2021/07/16 14:40:25 DEBUG : Creating backend with remote "./world.txt"
2021/07/16 14:40:25 DEBUG : fs cache: adding new entry for parent of "./world.txt", "/tmp"
2021/07/16 14:40:25 DEBUG : Creating backend with remote "localstack:toto"
2021/07/16 14:40:25 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/07/16 14:40:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:40:25 DEBUG : HTTP REQUEST (req 0xc0004a1000)
2021/07/16 14:40:25 DEBUG : HEAD /toto/world.txt HTTP/1.1
Host: localstack:5000
User-Agent: rclone/v1.55.1

2021/07/16 14:40:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:40:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:40:25 DEBUG : HTTP RESPONSE (req 0xc0004a1000)
2021/07/16 14:40:25 DEBUG : HTTP/1.1 403 Forbidden
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 14:40:25 GMT
Server: BaseHTTP/0.3 Python/2.7.15
Content-Length: 0

2021/07/16 14:40:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:40:25 ERROR : Attempt 1/1 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 14:40:25 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2021/07/16 14:40:25 DEBUG : 2 go routines active
2021/07/16 14:40:25 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:
6

for the rclone config, you added region = us-east-1 but i did not see that with the aws command?

try two tests

  1. add --s3-no-head=true
  2. add --s3-no-check-bucket=true - i must use this flag, due to the way i use s3 bucket polices, more locked down than https://rclone.org/s3/#s3-permissions.
7

for the rclone config, you added region = us-east-1 but i did not see that with the aws command?

aws --endpoint-url=http://localstack:5000 --region us-east-1 s3 cp test/report.clj s3://toto/ upload: test/report.clj to s3://toto/report.clj
works fine

  • add --s3-no-head=true 
2021/07/16 14:53:43 DEBUG : Using config file from "/home/docker/.config/rclone/rclone.conf"
2021/07/16 14:53:43 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "./world.txt" "localstack:toto" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--s3-no-head"]
2021/07/16 14:53:43 DEBUG : Creating backend with remote "./world.txt"
2021/07/16 14:53:43 DEBUG : fs cache: adding new entry for parent of "./world.txt", "/tmp"
2021/07/16 14:53:43 DEBUG : Creating backend with remote "localstack:toto"
2021/07/16 14:53:43 DEBUG : localstack: detected overridden config - adding "{NBTUO}" suffix to name
2021/07/16 14:53:43 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/07/16 14:53:43 DEBUG : fs cache: renaming cache item "localstack:toto" to be canonical "localstack{NBTUO}:toto"
2021/07/16 14:53:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:53:43 DEBUG : HTTP REQUEST (req 0xc000854300)
2021/07/16 14:53:43 DEBUG : HEAD /toto/world.txt HTTP/1.1
Host: localstack:5000
User-Agent: rclone/v1.55.1

2021/07/16 14:53:43 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:53:43 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:53:43 DEBUG : HTTP RESPONSE (req 0xc000854300)
2021/07/16 14:53:43 DEBUG : HTTP/1.1 403 Forbidden
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 14:53:43 GMT
Server: BaseHTTP/0.3 Python/2.7.15
Content-Length: 0

2021/07/16 14:53:43 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:53:43 ERROR : Attempt 1/1 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 14:53:43 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2021/07/16 14:53:43 DEBUG : 2 go routines active
2021/07/16 14:53:43 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:

--s3-no-check-bucket=true

2021/07/16 14:55:30 DEBUG : Using config file from "/home/docker/.config/rclone/rclone.conf"
2021/07/16 14:55:30 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "./world.txt" "localstack:toto" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--s3-no-check-bucket=true"]
2021/07/16 14:55:30 DEBUG : Creating backend with remote "./world.txt"
2021/07/16 14:55:30 DEBUG : fs cache: adding new entry for parent of "./world.txt", "/tmp"
2021/07/16 14:55:30 DEBUG : Creating backend with remote "localstack:toto"
2021/07/16 14:55:30 DEBUG : localstack: detected overridden config - adding "{Dn7qA}" suffix to name
2021/07/16 14:55:30 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2021/07/16 14:55:30 DEBUG : fs cache: renaming cache item "localstack:toto" to be canonical "localstack{Dn7qA}:toto"
2021/07/16 14:55:30 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:55:30 DEBUG : HTTP REQUEST (req 0xc00058bd00)
2021/07/16 14:55:30 DEBUG : HEAD /toto/world.txt HTTP/1.1
Host: localstack:5000
User-Agent: rclone/v1.55.1

2021/07/16 14:55:30 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2021/07/16 14:55:30 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:55:30 DEBUG : HTTP RESPONSE (req 0xc00058bd00)
2021/07/16 14:55:30 DEBUG : HTTP/1.1 403 Forbidden
Access-Control-Allow-Headers: authorization,content-type,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Content-Type: text/html; charset=utf-8
Date: Fri, 16 Jul 2021 14:55:30 GMT
Server: BaseHTTP/0.3 Python/2.7.15
Content-Length: 0

2021/07/16 14:55:30 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2021/07/16 14:55:30 ERROR : Attempt 1/1 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 14:55:30 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 1 (retrying may help)
Elapsed time:         0.0s

2021/07/16 14:55:30 DEBUG : 2 go routines active
2021/07/16 14:55:30 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:

tried both --> same

BUT...
Interestingly:

  • if I copy a new file (no previous attempt) with --s3-no-head=true it works
  • if I copy again it fails:
$ rclone  copy ./world2.txt localstack:toto --s3-no-head=true
$ rclone  copy ./world2.txt localstack:toto --s3-no-head=true
2021/07/16 15:02:04 ERROR : Attempt 1/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 15:02:04 ERROR : Attempt 2/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 15:02:04 ERROR : Attempt 3/3 failed with 1 errors and: Forbidden: Forbidden
	status code: 403, request id: , host id: 
2021/07/16 15:02:04 Failed to copy: Forbidden: Forbidden
	status code: 403, request id: , host id:
  • deletefile does not work:
$ rclone deletefile localstack:toto/world2.txt --s3-no-head=true
2021/07/16 15:02:54 ERROR : Attempt 1/3 failed with 1 errors and: localstack:toto/world2.txt is a directory or doesn't exist
2021/07/16 15:02:54 ERROR : Attempt 2/3 failed with 1 errors and: localstack:toto/world2.txt is a directory or doesn't exist
2021/07/16 15:02:54 ERROR : Attempt 3/3 failed with 1 errors and: localstack:toto/world2.txt is a directory or doesn't exist
2021/07/16 15:02:54 Failed to deletefile: localstack:toto/world2.txt is a directory or doesn't exist
8

check if there is a folder named localstack:toto/world2.txt?

try to

  1. copy a folder, not a single file
  2. copy a single file using rclone copyto, not using the name world2.txt
9

seems like rclone ls localstack:bucket/ is doing a GET on the bucket:
GET /bucket?delimiter=&max-keys=1000&prefix=
and that is not accepted by localstack:

Unable to find forwarding rule for host "localhost:4566", path "GET /bucket?delimiter=&max-keys=1000&prefix=", target header "", auth header "", data ""

even a simple curl seems to be problematic: curl --get "http://localhost:4566/bucket":
--> same localstack error

What do you think ?

10

best to get localstack working first and then try using rclone.
perhaps post at their forum.

have you searched the internet, as there a bunch of posts, such as
https://github.com/localstack/localstack/issues/3453

11

best to get localstack working first

to be clear: localstack seems to work perfectly fine with the aws cli client for instance.

have you searched the internet, as there a bunch of posts, such as

I have opened on issue in localstack github, and the thread you're referring too is kind of old and should be merged into the localstack:latest I used.

I have no idea if localstack has a problem, or if the rclone is doing some non-supported requests.

12

well, if curl does not work...

post the entire rclone debug log, not a snippet

13

according to Has anyone tried localstack S3 with rclone ? · Issue #4311 · localstack/localstack · GitHub

Problem seems to be a malformed GET request on rclone:

curl "http://localhost:4566/bucket/?delimiter=&max-keys=1000&prefix="

<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>bucket</Name><Prefix></Prefix><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key>world.txt</Key><LastModified>2021-07-21T12:20:41.000Z</LastModified><ETag>&#34;b1946ac92492d2347c6235b4d2611184&#34;</ETag><Size>6</Size><StorageClass>STANDARD</StorageClass><Owner><ID>75aa57f09aa0c8caeab4f8c24e99d10f8e7faeebf76c078efc7c6caea54ba06a</ID><DisplayName>webfile</DisplayName></Owner></Contents></ListBucketResult>

You can see the official API docs the correct syntax is: GET /?delimiter=Delimiter&encoding-type=EncodingType&marker=Marker&max-keys=MaxKeys&prefix=Prefix HTTP/1.1

14

post the entire rclone.log with a command that causes the problem.
and add --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG --log-file=rclone.log

15

In the linked post above, I posted a minimal reproducible example, but here it is:

%rclone ls localstack:bucket/ --dump=bodies --retries=1 --low-level-retries=1 --log-level=DEBUG

<7>DEBUG : Using config file from "/home/karl/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "localstack:bucket/" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "localstack:bucket/"
<7>DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
<7>DEBUG : fs cache: renaming cache item "localstack:bucket/" to be canonical "localstack:bucket"
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : HTTP REQUEST (req 0xc00057be00)
<7>DEBUG : GET /bucket?delimiter=&max-keys=1000&prefix= HTTP/1.1
Host: localhost:4566
User-Agent: rclone/v1.55.1
Accept-Encoding: gzip

<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : HTTP RESPONSE (req 0xc00057be00)
<7>DEBUG : HTTP/1.1 404 
Connection: close
Access-Control-Allow-Headers: authorization,content-type,content-length,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent,x-amz-target,x-amz-acl,x-amz-version-id,x-localstack-target,x-amz-tagging,amz-sdk-invocation-id,amz-sdk-request
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-amz-version-id
Content-Type: text/html; charset=utf-8
Date: Wed, 21 Jul 2021 09:45:04 GMT
Server: hypercorn-h11

{"status": "running"}
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : 2 go routines active
Failed to ls: directory not found
16

localstack, like aws, have deprecated path style in favor of virtual style

so i would add this to the command and see what happens.
--s3-force-path-style=false

for example,

with --s3-force-path-style=true, rclone default behavior

DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "remote:bucket" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--log-file=rclone.log" "--s3-force-path-style=true"]
DEBUG : GET /bucket?delimiter=&encoding-type=url&max-keys=1000&prefix= HTTP/1.1
Host: s3.xyz.com

with --s3-force-path-style=false

DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "remote:bucket" "--dump=bodies" "--retries=1" "--low-level-retries=1" "--log-level=DEBUG" "--log-file=rclone.log" "--s3-force-path-style=false"]
DEBUG : GET /?delimiter=&encoding-type=url&max-keys=1000&prefix= HTTP/1.1
Host: bucket.s3.xyz.com
17

It really looks like rclone does not send the same request as aws cli does

listing localstack bucket

with aws cli

aws s3 --endpoint-url http://localhost:4566 --no-sign-request ls s3://bucket
2021-07-22 14:26:18          6 world.txt

localstack log:

2021-07-22 12:33:59,239:API: 127.0.0.1 - - [22/Jul/2021 12:33:59] "GET /bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url HTTP/1.1" 200 -

with rclone ls --s3-force-path-style=true

rclone ls localstack:bucket --s3-force-path-style=true --retries=1 --low-level-retries=1  --dump=bodies --log-level=DEBUG

<7>DEBUG : Using config file from "/home/karl/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "localstack:bucket" "--s3-force-path-style=true" "--retries=1" "--low-level-retries=1" "--dump=bodies" "--log-level=DEBUG"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "localstack:bucket"
<7>DEBUG : localstack: detected overridden config - adding "{ladgk}" suffix to name
<7>DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
<7>DEBUG : fs cache: renaming cache item "localstack:bucket" to be canonical "localstack{ladgk}:bucket"
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : HTTP REQUEST (req 0xc00062de00)
<7>DEBUG : GET /bucket?delimiter=&max-keys=1000&prefix= HTTP/1.1
Host: localhost:4566
User-Agent: rclone/v1.55.1
Accept-Encoding: gzip

<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : HTTP RESPONSE (req 0xc00062de00)
<7>DEBUG : HTTP/1.1 404 
Connection: close
Access-Control-Allow-Headers: authorization,content-type,content-length,content-md5,cache-control,x-amz-content-sha256,x-amz-date,x-amz-security-token,x-amz-user-agent,x-amz-target,x-amz-acl,x-amz-version-id,x-localstack-target,x-amz-tagging,amz-sdk-invocation-id,amz-sdk-request
Access-Control-Allow-Methods: HEAD,GET,PUT,POST,DELETE,OPTIONS,PATCH
Access-Control-Allow-Origin: *
Access-Control-Expose-Headers: x-amz-version-id
Content-Type: text/html; charset=utf-8
Date: Thu, 22 Jul 2021 12:35:21 GMT
Server: hypercorn-h11

{"status": "running"}
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : 2 go routines active
Failed to ls: directory not found

localstack log:

2021-07-22T12:36:37:INFO:localstack.services.edge: Unable to find forwarding rule for host "localhost:4566", path "GET /bucket?delimiter=&max-keys=1000&prefix=", target header "", auth header "", data ""

with rclone ls --s3-force-path-style=false

rclone ls localstack:bucket --s3-force-path-style=false --retries=1 --low-level-retries=1  --dump=bodies --log-level=DEBUG

<7>DEBUG : Using config file from "/home/karl/.config/rclone/rclone.conf"
<7>DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "ls" "localstack:bucket" "--s3-force-path-style=false" "--retries=1" "--low-level-retries=1" "--dump=bodies" "--log-level=DEBUG"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote "localstack:bucket"
<7>DEBUG : localstack: detected overridden config - adding "{0kRUr}" suffix to name
<7>DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
<7>DEBUG : fs cache: renaming cache item "localstack:bucket" to be canonical "localstack{0kRUr}:bucket"
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : HTTP REQUEST (req 0xc000727100)
<7>DEBUG : GET /?delimiter=&max-keys=1000&prefix= HTTP/1.1
Host: bucket.localhost:4566
User-Agent: rclone/v1.55.1
Accept-Encoding: gzip

<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : HTTP RESPONSE (req 0xc000727100)
<7>DEBUG : Error: dial tcp: lookup bucket.localhost on 8.8.8.8:53: no such host
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : 2 go routines active
Failed to ls: RequestError: send request failed
caused by: Get "http://bucket.localhost:4566/?delimiter=&max-keys=1000&prefix=": dial tcp: lookup bucket.localhost on 8.8.8.8:53: no such host

localstack log:

nothing
18

i think we are making progress by using --s3-force-path-style=false.

now it seems like a dns issue on the server, as rclone tries to lookup bucket.localhost.
have you searched the internet about this, as there seem to be other localstack and/or docker users with that problem.

1 Like
19

as rclone tries to lookup bucket.localhost.

Actually that seems to be the problem. Since localstack is running as a docker container, there is no way to resolve bucket.localhost. Why does rclone does that ? aws s3 ls is not doing it...

20

not sure why aws s3 works but not rclone.
i would search around the forum, about docker and dns issues.