Swift, env_auth = true, env OS_ACCESS_TOKEN, HTTP Error: 404: 404 NOT FOUND

Help and Support
1

What is the problem you are having with rclone?

Failed to create file system for "incd-covid-oidc:": HTTP Error: 404: 404 NOT FOUND

What is your rclone version (output from rclone version)

rclone v1.54.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 20.04 LTS

Which cloud storage system are you using? (eg Google Drive)

Openstack Swift (provided by ceph radogw), community cloud (I am the sys admin)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -vv --dump auth lsd incd-covid-oidc:

The rclone config contents with secrets removed.

[incd-covid-oidc]
type = swift
env_auth = true

The following environment variables are set:

OS_ACCESS_TOKEN=e...
OS_AUTH_TYPE=v3oidcaccesstoken
OS_AUTH_URL=https://stratus.ncg.ingrid.pt:5000/v3
OS_IDENTITY_PROVIDER=egi.eu
OS_PROJECT_ID=
OS_PROTOCOL=openid

A log from the command with the -vv flag

rclone -vv --dump auth lsd incd-covid-oidc:
<7>DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "-vv" "--dump" "auth" "lsd" "incd-covid-oidc:"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Using config file from "/home/david/.config/rclone/rclone.conf"
<7>DEBUG : Creating backend with remote "incd-covid-oidc:"
<7>DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : HTTP REQUEST (req 0xc000123c00)
<7>DEBUG : POST /v3/auth/tokens HTTP/1.1
Host: stratus.ncg.ingrid.pt:5000
User-Agent: rclone/v1.54.0
Content-Length: 61
Content-Type: application/json
Accept-Encoding: gzip

<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : HTTP RESPONSE (req 0xc000123c00)
<7>DEBUG : HTTP/1.1 404 NOT FOUND
Content-Length: 79
Content-Type: application/json
Date: Fri, 26 Feb 2021 12:49:47 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
Vary: X-Auth-Token
X-Openstack-Request-Id: req-6776150e-f98d-44c9-8a5d-4e1eafea11f9

<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Failed to create file system for "incd-covid-oidc:": HTTP Error: 404: 404 NOT FOUND

And

rclone -vv --dump bodies lsd incd-covid-oidc:
<7>DEBUG : rclone: Version "v1.54.0" starting with parameters ["rclone" "-vv" "--dump" "bodies" "lsd" "incd-covid-oidc:"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Using config file from "/home/david/.config/rclone/rclone.conf"
<7>DEBUG : Creating backend with remote "incd-covid-oidc:"
<7>DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : HTTP REQUEST (req 0xc000176e00)
<7>DEBUG : POST /v3/auth/tokens HTTP/1.1
Host: stratus.ncg.ingrid.pt:5000
User-Agent: rclone/v1.54.0
Content-Length: 61
Content-Type: application/json
Accept-Encoding: gzip

{"auth":{"identity":{"methods":["token"],"token":{"id":""}}}}
<7>DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
<7>DEBUG : HTTP RESPONSE (req 0xc000176e00)
<7>DEBUG : HTTP/1.1 404 NOT FOUND
Content-Length: 79
Content-Type: application/json
Date: Fri, 26 Feb 2021 12:50:36 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5
Vary: X-Auth-Token
X-Openstack-Request-Id: req-ee8d220f-6c4e-4cc2-b9e8-cfb5508e340c

{"error":{"code":404,"message":"No token in the request","title":"Not Found"}}
<7>DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Failed to create file system for "incd-covid-oidc:": HTTP Error: 404: 404 NOT FOUND

Further information

I have searched the github issues and this forum, and tried a few more options on
the rclone.conf such as access_key_id and secret_access_key. The error is always the same

And of course I might be missing something very simple.

Openstack client with these variables (same terminal)

openstack object list somedir
+--------------------------------+
| Name                           |
+--------------------------------+
| rclone-v1.54.0-linux-amd64.deb |
+--------------------------------+

rclone with remote to same swift but another project and using keystone3 local user, works perfectly:

[incd-swift-lip]
type = swift
env_auth = false
user = myusername
key = xxx
auth = https://stratus.ncg.ingrid.pt:5000/v3
domain = Default
tenant = lip
tenant_domain = Default
storage_url = https://stratus-stor.ncg.ingrid.pt:8080/swift/v1/AUTH_<projectID>
endpoint_type = public

rclone lsd incd-swift-lip:
    24368884 2021-02-26 12:59:51         1 test
    33701454 2021-02-26 12:59:51         2 tt2
    53678763 2021-02-26 12:59:51      6072 udenv2
2

What does this do? I suspect this is the reason it isn't working.

I'm not familiar with identity providers so I expect rclone is doing something incorrect here.

The only supported enviroment variables are here

So I think to make this work would need a patch to ncw/swift

3

My guess would be OS_AUTH_URL is wrong because of this section:

Or maybe the server doesn't understand the ":5000" part of the Host: header?

4

Just noticed the error the server returned - not sure what that means...

5

I don't know if this is a bug, a request for a new feature or simply some misconfiguration on my part.
Openstack is configured both with local keystone and with keystone federations, that has one IdP configured and uses the OpenID connect protocol, as such all authentication and authorization in swift is through keystone, that "decides" which flows of authn/authz to use

Below I will paste some of the contents of openstack cli with debug option which may shed some light on this matter, I can provide the full debug later on if needed:

openstack --debug object list somedir

options: Namespace(access_token='***', access_token_endpoint='', access_token_type='', application_credential_id='', application_credential_name='', 
application_credential_secret='***', auth_methods='', auth_type='v3oidcaccesstoken',
auth_url='https://stratus.ncg.ingrid.pt:5000/v3', cacert=None, cert='', client_id='',
 client_secret='***', cloud='', code='', debug=True, default_domain='default', 
default_domain_id='', default_domain_name='', deferred_help=False, 
discovery_endpoint='', domain_id='', domain_name='', endpoint='', 
identity_provider='egi.eu', insecure=None, interface='public', key='', log_file=None, 
openid_scope='', os_beta_command=False, os_compute_api_version='', 
os_identity_api_version='', os_image_api_version='', os_network_api_version='', 
os_object_api_version='', os_project_id=None, os_project_name=None, 
os_volume_api_version='', passcode='', password='***', project_domain_id='', 
project_domain_name='', project_id='xxxx', 
project_name='', protocol='openid', redirect_uri='', region_name='', 
remote_project_domain_id='', remote_project_domain_name='', remote_project_id='', 
remote_project_name='', service_provider='', system_scope='', timing=False, 
token='***', trust_id='', user_domain_id='', user_domain_name='', user_id='', 
username='', verbose_level=3, verify=None)

Auth plugin v3oidcaccesstoken selected

Notably the following are set, which may indicate the variables which have to be set in the conf of the remote

access_token
application_credential_secret
auth_type='v3oidcaccesstoken'
auth_url='https://stratus.ncg.ingrid.pt:5000/v3'
client_secret
identity_provider
password - this one I don't know from where it comes, if the access_token is used for it
project_id
protocol
token

Further below I have the REQ and RESP BODY:

REQ: curl -g -i -X POST https://stratus.ncg.ingrid.pt:5000/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth
-H "Authorization: {SHA256}...."
-H "User-Agent: openstacksdk/0.53.0 keystoneauth1/4.3.0 python-requests/2.25.1 CPython/3.8.5"
Starting new HTTPS connection (1): stratus.ncg.ingrid.pt:5000
https://stratus.ncg.ingrid.pt:5000 "POST /v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth HTTP/1.1" 201 633

RESP: [201] Content-Length: 633 Content-Type: application/json Date: Fri, 26 Feb 2021 15:03:09 GMT Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips PHP/5.4.16 mod_wsgi/3.4 Python/2.7.5 Vary: X-Auth-Token X-Subject-Token: {SHA256}... x-openstack-request-id: req-...

RESP BODY: {"token": {"issued_at": "2021-02-26T15:03:09.000000Z", "audit_ids": ["..."], "methods": ["openid"], "expires_at": "2021-02-26T16:03:09.000000Z", "user": {"OS-FEDERATION": {"identity_provider": {"id": "egi.eu"}, "protocol": {"id": "openid"}, "groups": [{"id": "..."}, {"id": "..."}, {"id": "..."}, {"id": "..."}]}, "domain": {"id": "Federated", "name": "Federated"}, "id": "...", "name": "...@egi.eu_egiID"}}}

POST call to https://stratus.ncg.ingrid.pt:5000/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth used request id req-...

And further below:

Making authentication request to https://stratus.ncg.ingrid.pt:5000/v3/auth/tokens
https://stratus.ncg.ingrid.pt:5000 "POST /v3/auth/tokens HTTP/1.1" 201 6120
6

So it looks like the first request to

https://stratus.ncg.ingrid.pt:5000/v3/OS-FEDERATION/identity_providers/egi.eu/protocols/openid/auth

Is used to get something extra to post to

https://stratus.ncg.ingrid.pt:5000/v3/auth/tokens

the Auth URL.

Can you see what it is? Probably a token.

If you can find out what it is you can try putting this as the API key in rclone.

This isn't a proper solution as it looks like rclone need to be doing a whole set of extra Auth somehow.

Failing that you can find the storage URL and token using be the openstack tools and give those to rclone - that should work. Again not ideal.

7

all, thanks for the tips and here it goes how it works.

I needed to do a token issue from the access_token - all with openstack cli
on one hand I had

OS_ACCESS_TOKEN=e...
OS_AUTH_TYPE=v3oidcaccesstoken
OS_AUTH_URL=https://stratus.ncg.ingrid.pt:5000/v3
OS_IDENTITY_PROVIDER=egi.eu
OS_PROJECT_ID=<the projectID>
OS_PROTOCOL=openid

Now I can do

openstack token issue

This gives me a keystone unscoped token, now I need to set the following env variables, I do that in another "clean" terminal:

OS_AUTH_TOKEN=<Here the unscoped token I got in the previous step>
OS_AUTH_TYPE=v3token
OS_AUTH_URL=https://stratus.ncg.ingrid.pt:5000/v3
OS_PROJECT_ID=<the projectID>
OS_STORAGE_URL=https://stratus-stor.ncg.ingrid.pt:8080/swift/v1/AUTH_<the projectID>

Now I can use the rclone remote with just:

[incd-covid-oidc]
type = swift
env_auth = true

And it works as expected:

rclone lsd incd-covid-oidc:
    14296226 2021-03-01 11:10:28         1 somedir

rclone ls incd-covid-oidc:somedir
 14296226 rclone-v1.54.0-linux-amd64.deb

A colleague of mine much more knowledgeable than me in federated identity and auth protocols pointed out that https://github.com/ncw/swift/blob/master/auth_v3.go would need implmentation of oidc auth.

the ticket can be closed, and it come to the point of asking if you can or plan to implement this in the code, and of course I can involve other colleagues to help in the discussion if need be.

8

Well done - that looked moderately involved!

Yes you are right, that is the missing piece...

I've not had much time to look at the swift library in recent years so I'd love some help with this. Do you think one of your colleagues could contribute a fix for this?

9

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.