Openstack keystone v3

What is the problem you are having with rclone?

Auth always fails though the user/key are known to be legit. We also have an older cluster (based off Juno using keystone v2) where the auth succeeds. The ultimate backend in this case is CEPH RGW but we use the OpenStack swift client for all object management very successfully.

What basic config item might be missing? Does rclone do the right thing with the keystone token? In our endpoint list, this site appears as:

https://rgw-s00.rbh.cloud/swift/v1/AUTH_$(project_id)s

I'm not sure rclone passes the tenant/project id correctly to storage.

What is your rclone version (output from rclone version)

rclone v1.52.0

  • os/arch: linux/amd64
  • go version: go1.14.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

centos x86_64

Which cloud storage system are you using? (eg Google Drive)

OpenStack Rocky (private hosted)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vvvvvvv --dump-bodies ls stp-rgw:database_backups_mongo

The rclone config contents with secrets removed.

[stp-rgw]
type = swift
user = platform_admin
key = <elided>
auth = https://keystone-s00.rbh.cloud/v3
tenant = platform
region = stp
storage_url = https://rgw-s00.rbh.cloud/swift/v1

A log from the command with the -vv flag

2020/06/15 14:51:11 NOTICE: --dump-bodies is obsolete - please use --dump bodies instead
2020/06/15 14:51:11 DEBUG : rclone: Version "v1.52.0" starting with parameters ["rclone" "-vvvvvvv" "--dump-bodies" "ls" "stp-rgw:database_backups_mongo"]
2020/06/15 14:51:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/06/15 14:51:11 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2020/06/15 14:51:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/15 14:51:11 DEBUG : HTTP REQUEST (req 0xc0003ff000)
2020/06/15 14:51:11 DEBUG : POST /v3/auth/tokens HTTP/1.1
Host: keystone-s00.rbh.cloud
User-Agent: rclone/v1.52.0
Content-Length: 194
Content-Type: application/json
Accept-Encoding: gzip

{"auth":{"identity":{"methods":["password"],"password":{"user":{"name":"platform_admin","password":"<elided>"}}},"scope":{"project":{"name":"platform","domain":{"name":"Default"}}}}}
2020/06/15 14:51:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/15 14:51:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/15 14:51:11 DEBUG : HTTP RESPONSE (req 0xc0003ff000)
2020/06/15 14:51:11 DEBUG : HTTP/1.1 400 Bad Request
Content-Length: 232
Content-Type: application/json
Date: Mon, 15 Jun 2020 19:51:11 GMT
Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
Vary: X-Auth-Token
X-Openstack-Request-Id: req-57597fbe-e5b2-44f8-8834-760e5a24ed20

{"error": {"message": "Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.", "code": 400, "title": "Bad Request"}}
2020/06/15 14:51:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/15 14:51:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/15 14:51:11 DEBUG : HTTP REQUEST (req 0xc00059c200)
2020/06/15 14:51:11 DEBUG : POST /v3/auth/tokens HTTP/1.1
Host: keystone-s00.rbh.cloud
User-Agent: rclone/v1.52.0
Content-Length: 194
Content-Type: application/json
Accept-Encoding: gzip

{"auth":{"identity":{"methods":["password"],"password":{"user":{"name":"platform_admin","password":"<elided>"}}},"scope":{"project":{"name":"platform","domain":{"name":"Default"}}}}}
2020/06/15 14:51:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/06/15 14:51:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/15 14:51:11 DEBUG : HTTP RESPONSE (req 0xc00059c200)
2020/06/15 14:51:11 DEBUG : HTTP/1.1 400 Bad Request
Content-Length: 232
Content-Type: application/json
Date: Mon, 15 Jun 2020 19:51:11 GMT
Server: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5
Vary: X-Auth-Token
X-Openstack-Request-Id: req-9da88afc-8b09-450f-9eee-fac773fa6132

{"error": {"message": "Expecting to find domain in user. The server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.", "code": 400, "title": "Bad Request"}}
2020/06/15 14:51:11 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/06/15 14:51:11 Failed to create file system for "stp-rgw:database_backups_mongo": Bad Request

Rclone's request to get the token fails with this error

I think that indicates the domain or maybe tenant_domain is missing from the rclone config? Not really sure!

I just looked at a v3 config I have - I have this which is very similar to yours. It was also for connecting to CEPH via Openstack...

[swiftv3]
type = swift
user = USER
key = KEY
auth = https://example.com:5000/v3.0
tenant = USER
region = REGION

The other thing you could try is if you have a working set of openstack environment variables then tell rclone to use them with env_auth=true and leave the rest of the config blank.

OpenStack auth is so confusing!

As things typically work -- things started working for me after posting this. My config ends up:

[stp-rgw]
type = swift 
domain = default 
key = <elided>
region = stp
storage_url = https://rgw-s00.rbh.cloud/swift/v1/AUTH_<guid>
tenant = platform
tenant_domain = default 
user = platform_admin
auth = https://keystone-s00.rbh.cloud/v3

I think what got it working was adding the domain = default and tenant_domain = default options. This got me past the auth issue to where it received a token and then I needed to add the project GUID to the storage link. Voila!

The older cluster needed the auth_version = 2 and that definition works with the defaults values as well. Perfect. This is a wonderful solution!

1 Like

Great - glad you got it working :slight_smile:

OpenStack Swift auth is very complicated and can be set up in many different ways by the providers!

I'm not sure you should need to set storage_url rclone should get that back when it gets its list of endpoints. You may be doing something unusual though where you need to set that.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.