iDrive e2 - can't access new bucket/region

dinosm · February 25, 2024, 10:15pm

What is the problem you are having with rclone?

I set up a new bucket in iDrive e2, region is London.
I cannot even list its contents with rclone, it just hangs at 'resolving s3 region'.
I have another bucket in Frankfurt, which works fine.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-94-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

iDrive e2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone lsd idriveuk: -vv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[idriveuk]
type = s3
provider = IDrive
region = eu-west-1
access_key_id = XXX
secret_access_key = XXX
endpoint = XXX
acl = private
bucket_acl = private
disable_checksum = true
disable_http2 = true

A log from the command that you were trying to run with the `-vv` flag

2024/02/25 22:06:25 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "lsd" "idriveuk:" "-vv"]
2024/02/25 22:06:25 DEBUG : Creating backend with remote "idriveuk:"
2024/02/25 22:06:25 DEBUG : Using config file from "/home/dinosm/.config/rclone/rclone.conf"
2024/02/25 22:06:25 DEBUG : Resolving service "s3" region "eu-west-1"

It just hangs there and doesn't go any further. I have tried other s3 regions, as well as region codes as given by iDrive e2 on their own website (e.g. gb-ldn), none of them works.
I have recreated the bucket, recreated the credential, no luck.

asdffdsa · February 25, 2024, 10:18pm

hi,
i have 20+ idrive remotes, none of them use region or disable_http2

asdffdsa · February 25, 2024, 10:21pm

and for a deeper look, try --dump=headers

dinosm · February 25, 2024, 10:21pm

If I don't put region in the config, it just hangs like so:

2024/02/25 22:19:10 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "lsd" "idriveuk:" "-vv"]
2024/02/25 22:19:10 DEBUG : Creating backend with remote "idriveuk:"
2024/02/25 22:19:10 DEBUG : Using config file from "/home/dinosm/.config/rclone/rclone.conf"
2024/02/25 22:19:10 DEBUG : Resolving service "s3" region "us-east-1"

Which led me to believe it is automatically selecting the wrong region, so I put a region there, but it didn't make a difference.

I put this there because I am having some issues streaming and read that http2 may be affecting this (both here when debugging a OneDrive issue and elsewhere).

EDIT: It hangs the same way even without disable-http2

Log with headers:

2024/02/25 22:23:51 DEBUG : rclone: Version "v1.65.1" starting with parameters ["rclone" "lsd" "idriveuk:" "-vv" "--dump=headers"]
2024/02/25 22:23:51 DEBUG : Creating backend with remote "idriveuk:"
2024/02/25 22:23:51 DEBUG : Using config file from "/home/dinosm/.config/rclone/rclone.conf"
2024/02/25 22:23:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/25 22:23:51 DEBUG : Resolving service "s3" region "us-east-1"
2024/02/25 22:23:51 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/02/25 22:23:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/02/25 22:23:51 DEBUG : HTTP REQUEST (req 0xc000aeeb00)
2024/02/25 22:23:51 DEBUG : GET / HTTP/1.1
Host: XXX
User-Agent: rclone/v1.65.1
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240225T222351Z
Accept-Encoding: gzip

2024/02/25 22:23:51 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

asdffdsa · February 25, 2024, 10:26pm

given that you got one idrive remote to work, strange that you cannot get a second remote working.
could be some basic human error.

can you post the redacted config for the working remote?

dinosm · February 25, 2024, 10:28pm

[idrive]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
region = eu-central-1
acl = private
endpoint = XXX
disable_checksum = true
disable_http2 = true

I also pasted the log with header dump in an edit in my previous post.

asdffdsa · February 25, 2024, 10:36pm

fwiw, i would not use region as it is not required and just one less thing to troubleshoot.
https://www.idrive.com/s3-storage-e2/rclone

also, if the endpoint is not correct, rclone will hang and eventually error out.

dinosm · February 25, 2024, 10:51pm

I've removed the region, and still does the same.

I've manually checked the credentials letter by letter, and same for the endpoint, they are all correct. I am stumped.

EDIT: I deleted the London bucket, the London region (and the credential then got auto-deleted too), then recreated them all from scratch. It gave me a different endpoint. Still same situation.

I noticed that rclone config gets the endpoint automatically, which must mean it can access iDrive using my credentials correctly enough to get the endpoint, so might this be a temporary iDrive issue?

asdffdsa · February 25, 2024, 10:56pm

[idrive_london]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
endpoint = p1e1.ldn.idrivee2-24.com

rclone lsd idrive_london: -vv
2024/02/25 17:55:29 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "lsd" "idrive_london:" "-vv"]
2024/02/25 17:55:29 DEBUG : Creating backend with remote "idrive_london:"
2024/02/25 17:55:29 DEBUG : Using config file from "c:\\data\\rclone\\rclone.conf"
2024/02/25 17:55:29 DEBUG : Resolving service "s3" region "us-east-1"
          -1 2024-02-25 17:53:11        -1 zork

Asad · February 27, 2024, 4:24am

ping the endpoint to check if it is accessible from your IP

asdffdsa · February 27, 2024, 12:59pm

need to make sure you are using the correct endpoint?
can you please post the real endpoint you are using, no need to redact?

at idrive website, when creating the access key, idrive will display the endpoint

rclone does not automatically get the endpoint, rclone config asks for the endpoint.

Option endpoint.
Endpoint for S3 API.
Required when using an S3 clone.
Enter a value. Press Enter to leave empty.
endpoint>

kapitainsky · February 27, 2024, 1:55pm

I have multiple buckets in multiple regions - no issue.

I have tried to replicate the problem and this is what I find:

my original setup - all is working OK

$ rclone config redacted iDrive:
[iDrive]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
endpoint = p7v1.ldn.idrivee2-40.com

$ rclone lsf iDrive: -vv
2024/02/27 13:52:09 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "lsf" "iDrive:" "-vv"]
2024/02/27 13:52:09 DEBUG : Creating backend with remote "iDrive:"
2024/02/27 13:52:09 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/02/27 13:52:09 DEBUG : Resolving service "s3" region "us-east-1"
test-crypt/
test-dal/
test-dal-lock/
test-lock/
test-uk/
2024/02/27 13:52:10 DEBUG : 7 go routines active

Wrong but valid endpoint (I used one from @asdffdsa test):

$ rclone config redacted iDrive:
[iDrive]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
endpoint = p1e1.ldn.idrivee2-24.com

$ rclone lsf iDrive: -vv
2024/02/27 13:48:40 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "lsf" "iDrive:" "-vv"]
2024/02/27 13:48:40 DEBUG : Creating backend with remote "iDrive:"
2024/02/27 13:48:40 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/02/27 13:48:40 DEBUG : Resolving service "s3" region "us-east-1"
2024/02/27 13:48:40 ERROR : : error listing: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.
	status code: 403, request id: 17B7BC63CE1C0BBF, host id:
2024/02/27 13:48:40 DEBUG : 7 go routines active
2024/02/27 13:48:40 Failed to lsf with 2 errors: last error was: error in ListJSON: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.
	status code: 403, request id: 17B7BC63CE1C0BBF, host id:

Wrong and invalid endpoint:

$ rclone config redacted iDrive:
[iDrive]
type = s3
provider = IDrive
access_key_id = XXX
secret_access_key = XXX
endpoint = blabla-p7v1.ldn.idrivee2-40.com

$ rclone lsf iDrive: -vv
2024/02/27 13:51:05 DEBUG : rclone: Version "v1.65.2" starting with parameters ["rclone" "lsf" "iDrive:" "-vv"]
2024/02/27 13:51:05 DEBUG : Creating backend with remote "iDrive:"
2024/02/27 13:51:05 DEBUG : Using config file from "/Users/kptsky/.config/rclone/rclone.conf"
2024/02/27 13:51:05 DEBUG : Resolving service "s3" region "us-east-1"

# it hangs here forever

Which let me believe that as test (3) behaves exactly like original OP problem then most likely problem is typo in the endpoint.

dinosm · February 27, 2024, 2:27pm

Here's my process for creating a new remote for the new iDrive bucket:

I choose '5' for S3 storage, then 12 for iDrive

Leave env_auth blank.

Enter my access key.

No one else has access so leave acl blank.

No I don't want to edit advanced config.

As you can see, at no point did it ask me to enter the endpoint, and it found it automatically. Letter by letter it looks correct.

kapitainsky · February 27, 2024, 2:41pm

Can you update your rclone to the latest version before we dive deeper?

asdffdsa · February 27, 2024, 2:42pm

can you ping that endpoint?

in any event, when createing the access key at idrive website, the correct endpoint is displayed
that is what i use so can you test that?

dinosm · February 27, 2024, 2:45pm

I actually did that yesterday after I noticed your version was .2 on the other thread


rclone v1.65.2
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-94-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.21.6
- go/linking: static
- go/tags: none

kapitainsky · February 27, 2024, 2:47pm

Thx. One issue less to worry about.

dinosm · February 27, 2024, 2:50pm

Ah there we go.

I can't ping the endpoint from the server.
I can however ping it from my local machine.

Server is in Germany, local machine in the UK, endpoint in the UK. Are these iDrive endpoints only accessible within the country they are located in?

asdffdsa · February 27, 2024, 2:51pm

no, as i have buckets in all regions.

kapitainsky · February 27, 2024, 2:51pm

Nope

Here you are ping from Denmark for example:

$ ping e5q1.ldn.idrivee2-17.com
PING e5q1.ldn.idrivee2-17.com (191.96.59.8) 56(84) bytes of data.
64 bytes from 191.96.59.8 (191.96.59.8): icmp_seq=1 ttl=53 time=9.63 ms
64 bytes from 191.96.59.8 (191.96.59.8): icmp_seq=2 ttl=53 time=9.94 ms
^C
--- e5q1.ldn.idrivee2-17.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 9.630/9.783/9.936/0.153 ms