S3 config copies input literally?

cranch · April 22, 2023, 9:24pm

I'm trying Rclone for the first time to back up my Google Drive to S3.

Using Rclone 1.62.2 on MacOS 13.3.1

I already have an S3 bucket in us-east-1.

The rclone S3 configuration under my ~/.config/rclone:

[aws-s3]
type = s3
provider = AWS
access_key_id = **reducted**
secret_access_key = **reducted**
region = us-east-1
location_constraint = ap-southeast-2
server_side_encryption = AES256
endpoint = us-east-1
acl = private

When I try to execute rclone with this configuration, I get:

❯ rclone -vvv lsd aws-s3:
2023/04/23 07:09:18 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "-vvv" "lsd" "aws-s3:"]
2023/04/23 07:09:18 DEBUG : Creating backend with remote "aws-s3:"
2023/04/23 07:09:18 DEBUG : Using config file from "/Users/.../.config/rclone/rclone.conf"
2023/04/23 07:09:18 DEBUG : Resolving service "s3" region "us-east-1"
2023/04/23 07:09:59 ERROR : : error listing: RequestError: send request failed
caused by: Get "https://us-east-1/": dial tcp: lookup us-east-1: no such host
2023/04/23 07:09:59 DEBUG : 4 go routines active
2023/04/23 07:09:59 Failed to lsd with 2 errors: last error was: RequestError: send request failed
caused by: Get "https://us-east-1/": dial tcp: lookup us-east-1: no such host

Before entering us-east-1 as the endpoint, I used 1 and the output was similar:

❯ rclone lsd aws-s3:
Enter configuration password:
password:
2023/04/23 06:55:00 ERROR : : error listing: RequestError: send request failed
caused by: Get "https://1/": dial tcp 0.0.0.1:443: connect: no route to host
2023/04/23 06:55:00 Failed to lsd with 2 errors: last error was: RequestError: send request failed
caused by: Get "https://1/": dial tcp 0.0.0.1:443: connect: no route to host

It appears that the part of the code which asks for the S3 endpoint takes the input literally, without converting the menu number input into the right value like s3.us-east-1.amazonaws.com

If I change my input to use an explicit hostname then it works:

[aws-s3]
type = s3
provider = AWS
access_key_id = **reducted**
secret_access_key = **reducted**
region = us-east-1
location_constraint = us-east-1
server_side_encryption = AES256
endpoint = s3.us-east-1.amazonaws.com
acl = private

And a test:

❯ rclone -vvv lsd aws-s3:
2023/04/23 07:17:48 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "-vvv" "lsd" "aws-s3:"]
2023/04/23 07:17:48 DEBUG : Creating backend with remote "aws-s3:"
2023/04/23 07:17:48 DEBUG : Using config file from " /Users/.../.config/rclone/rclone.conf"
2023/04/23 07:17:48 DEBUG : Resolving service "s3" region "us-east-1"
          -1 2015-06-03 15:54:48        -1 backups
...

Animosity022 · April 22, 2023, 10:03pm

Is your goal to modify the default endpoint?

region> 2

Option endpoint.
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Enter a value. Press Enter to leave empty.
endpoint>

cranch · April 22, 2023, 10:08pm

My goal is to set it. The instructions are to enter a number or leave blank for number 1.
When I did that, the program saved the literal 1 as the hostname and tried to resolve that via DNS.

Animosity022 · April 22, 2023, 10:09pm

It doesn't enter in anything if you leave it at the default.


[TESTS3]
type = s3
provider = AWS
region = us-east-2

cranch · April 22, 2023, 10:18pm

Correct for the first time, but once I entered a region (by picking a number), it wouldn't set it right.
Once I type 1 for The default endpoint it starts going haywire with the options for other questions.

Option region.
Region to connect to.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
   / The default endpoint - a good choice if you are unsure.
 1 | US Region, Northern Virginia, or Pacific Northwest.
   | Leave location constraint empty.
   \ (us-east-1)
   / US East (Ohio) Region.
 2 | Needs location constraint us-east-2.
   \ (us-east-2)
   / US West (Northern California) Region.
 3 | Needs location constraint us-west-1.
   \ (us-west-1)
   / US West (Oregon) Region.
 4 | Needs location constraint us-west-2.
   \ (us-west-2)
   / Canada (Central) Region.
 5 | Needs location constraint ca-central-1.
   \ (ca-central-1)
   / EU (Ireland) Region.
 6 | Needs location constraint EU or eu-west-1.
   \ (eu-west-1)
   / EU (London) Region.
 7 | Needs location constraint eu-west-2.
   \ (eu-west-2)
   / EU (Paris) Region.
 8 | Needs location constraint eu-west-3.
   \ (eu-west-3)
   / EU (Stockholm) Region.
 9 | Needs location constraint eu-north-1.
   \ (eu-north-1)
   / EU (Milan) Region.
10 | Needs location constraint eu-south-1.
   \ (eu-south-1)
   / EU (Frankfurt) Region.
11 | Needs location constraint eu-central-1.
   \ (eu-central-1)
   / Asia Pacific (Singapore) Region.
12 | Needs location constraint ap-southeast-1.
   \ (ap-southeast-1)
   / Asia Pacific (Sydney) Region.
13 | Needs location constraint ap-southeast-2.
   \ (ap-southeast-2)
   / Asia Pacific (Tokyo) Region.
14 | Needs location constraint ap-northeast-1.
   \ (ap-northeast-1)
   / Asia Pacific (Seoul).
15 | Needs location constraint ap-northeast-2.
   \ (ap-northeast-2)
   / Asia Pacific (Osaka-Local).
16 | Needs location constraint ap-northeast-3.
   \ (ap-northeast-3)
   / Asia Pacific (Mumbai).
17 | Needs location constraint ap-south-1.
   \ (ap-south-1)
   / Asia Pacific (Hong Kong) Region.
18 | Needs location constraint ap-east-1.
   \ (ap-east-1)
   / South America (Sao Paulo) Region.
19 | Needs location constraint sa-east-1.
   \ (sa-east-1)
   / Middle East (Bahrain) Region.
20 | Needs location constraint me-south-1.
   \ (me-south-1)
   / Africa (Cape Town) Region.
21 | Needs location constraint af-south-1.
   \ (af-south-1)
   / China (Beijing) Region.
22 | Needs location constraint cn-north-1.
   \ (cn-north-1)
   / China (Ningxia) Region.
23 | Needs location constraint cn-northwest-1.
   \ (cn-northwest-1)
   / AWS GovCloud (US-East) Region.
24 | Needs location constraint us-gov-east-1.
   \ (us-gov-east-1)
   / AWS GovCloud (US) Region.
25 | Needs location constraint us-gov-west-1.
   \ (us-gov-west-1)
region> 1

Option region.
region - the location where your bucket will be created and your data stored.
Choose a number from below, or type in your own string value.
Press Enter for the default (us-east-1).
 1 / Global CDN (All locations) Region
   \ (global)
 2 / Australia (All states)
   \ (au)
 3 / NSW (Australia) Region
   \ (au-nsw)
 4 / QLD (Australia) Region
   \ (au-qld)
 5 / VIC (Australia) Region
   \ (au-vic)
 6 / Perth (Australia) Region
   \ (au-wa)
 7 / Manila (Philippines) Region
   \ (ph)
 8 / Bangkok (Thailand) Region
   \ (th)
 9 / HK (Hong Kong) Region
   \ (hk)
10 / Ulaanbaatar (Mongolia) Region
   \ (mn)
11 / Bishkek (Kyrgyzstan) Region
   \ (kg)
12 / Jakarta (Indonesia) Region
   \ (id)
13 / Tokyo (Japan) Region
   \ (jp)
14 / SG (Singapore) Region
   \ (sg)
15 / Frankfurt (Germany) Region
   \ (de)
16 / USA (AnyCast) Region
   \ (us)
17 / New York (USA) Region
   \ (us-east-1)
18 / Freemont (USA) Region
   \ (us-west-1)
19 / Auckland (New Zealand) Region
   \ (nz)

Animosity022 · April 22, 2023, 10:22pm

Not sure what specifically you are typing as it works fine for me.

[TESTINGS3]
type = s3
provider = AWS
region = us-east-1

did another one with no issue.

cranch · April 22, 2023, 10:33pm

When I do as you suggest and don't type ANYTHNIG then it seems to give the default as you quote above.

But if I try to enter a number, things don't work.

Have you tried to enter any number in the region picker?

Animosity022 · April 23, 2023, 11:30am

Yes, I type in 1 and hit enter.

25 | Needs location constraint us-gov-west-1.
   \ (us-gov-west-1)
region> 1

Option endpoint.
Endpoint for S3 API.
Leave blank if using AWS to use the default endpoint for the region.
Enter a value. Press Enter to leave empty.
endpoint>

Option location_constraint.
Location constraint - must be set to match the Region.
Used when creating buckets only.
Choose a number from below, or type in your own value.
Press Enter to leave empty.

and get

[testings3]
type = s3
provider = AWS
region = us-east-1
location_constraint = us-east-2

I even added a constraint as I picked a number for that.

cranch · May 1, 2023, 9:04am

Thank you for taking the time to look at it, and apologise for the belated response - I was caught up with work.

I re-configured a test S3 destination and this time it worked. I'm not sure what made the difference.

system · May 31, 2023, 9:04am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.