Rclone config api without credentials

[lugoues]

What is the problem you are having with rclone?

Hello all.

I am attempting to get rclone to work with GCP/GKE and ADC and to pull authorization from the environment, which I know rclone supports, and I've validated. The issue I'm having is that Seldon, which is responsible for loading the config, is using the rclone API and it seems the API has no concept of adding an empty config (just type, no other fields, no credentials). Have I overlooked something, is this possible?

Seldon's workflow incase it helps:

1. create a secret in k8s,
2. seldon agent fetches that secret
3. seldon agent calls rclone's remote API to inject the new config into rclone using config/create

I've replicated the same steps using rclone's console app to attempt a config/create call myself and I see the same issue. Rclone spits out that I need to auth in a browser and doesn't save the config to disk. I've manually written the file myself and rclone works as expected so I think it's just related to the config api.

I've looked through Seldon's code, and it seems to be ok.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

• os/version: redhat 9.1 (64 bit)
• os/kernel: 5.10.0-21-cloud-amd64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.19.4
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

google cloud storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rc config/create name="gs" type="google cloud storage" parameters='{}' opt='{"nonInteractive": true}'
# and/or
rclone rc config/create name="gs" type="google cloud storage" parameters='{}' opt='{}'

A log from the command with the -vv flag

rclone rc config/create name="gs" type="google cloud storage" parameters='{}' opt='{"nonIneractive": true}' -vvvvv
2023/03/04 00:46:21 DEBUG : Configuration directory could not be created and will not be used: mkdir /.config: permission denied
2023/03/04 00:46:21 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "rc" "config/create" "name=gs" "type=google cloud storage" "parameters={}" "opt={\"nonInteractive\": true}" "-vvvvv"]
2023/03/04 00:46:21 DEBUG : 4 go routines active
2023/03/04 00:46:21 Failed to rc: failed to read rc response: 500 Internal Server Error: {
        "error": "config failed to refresh token: failed to start auth webserver: listen tcp 127.0.0.1:53682: bind: address already in use",
        "input": {
                "name": "gs",
                "opt": "{\"nonInteractive\": true}",
                "parameters": "{}",
                "type": "google cloud storage"
        },
        "path": "config/create",
        "status": 500
}

[lugoues]

The answer to my question is no, it's not currently possible. I've opened a PR #6804 which will add this functionality to the google cloud storage provider.

[ncw (Nick Craig-Wood)]

Thank you for the PR

I'll just note that you can use the connection string style remote now so :gcs,env_auth: with no need for config file at all. (You might need something else in there, I don't know).

[lugoues]

Oh cool.

I use rclone through Seldon Core and it has its own remote config it uses to configure rclone through rclone's api. I think that actually may be another problem, rclone can't configure a remote that has no parameters.

Meaning you can't create the following through the api.

[gs]
type = google cloud stroage

[ncw (Nick Craig-Wood)]

In general what you wrote will work

rclone rc config/create --loopback name=ladflksfjlksjflkdj type="swift" parameters="{}"

But if you try type="google cloud storage" then it will be trying to open the oauth webserver without your env_auth patch. There are ways around this which I can explain if you need to know, but I think env_auth has probably sorted your problem!

[lugoues]

Yep env_auth works It seems like a more straight forward method too.