Strange behaviour with connections strings

Help and Support
1

What is the problem you are having with rclone?

I'm connecting to our own Ceph cluster using rclone while trying to create the remote on the fly using connections strings. If I specify the endpoint with a prefixed https://, rclone complains about a missing secret_access_key. If I remove the leading https://, it works. If I shift the endpoint parameter to a different place in the connection string, I get an error about the server misbehaving.
I understand that the prefixed https:// might be problematic, but it seems to me rclone's handling of the problem is also somewhat inconsistent and a better error message would possibly be helpful.

Run the command 'rclone version' and share the full output of the command.

rclone v1.56.2

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.15.0-58-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Ceph S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls :s3,access_key_id=XXX,endpoint="https://s3.ourdomain.com",secret_access_key=YYY:bucketName

The rclone config contents with secrets removed.

N/A using on-the-fly remotes

A log from the command with the -vv flag

<7>DEBUG : rclone: Version "v1.56.2" starting with parameters ["rclone" "ls" ":s3,access_key_id=XXX,endpoint=https://s3.ourdomain.com,secret_access_key=YYY:bucketName" "-vv"]
<7>DEBUG : rclone: systemd logging support activated
<7>DEBUG : Creating backend with remote ":s3,access_key_id=XXX,endpoint=https://s3.ourdomain.com,secret_access_key=YYY:bucketName"
<7>DEBUG : :s3: detected overridden config - adding "{znKRX}" suffix to name
<5>NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
Failed to create file system for ":s3,access_key_id=XXX,endpoint=https://s3.ourdomain.com,secret_access_key=YYY:bucketName": secret_access_key not found

If I change the endpoint parameter to

endpoint=s3.ourdomain.com

it works.

If I shift the original endpoint parameter to before the access_key_id param or after the secret_access_key param, I get a different error:

<5>NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
Failed to ls: RequestError: send request failed
caused by: Get "https://https/s3.ourdomain.com%2Caccess_key_id%3DXXX%2Csecret_access_key%3DYYY%3AbucketName?delimiter=&max-keys=1000&prefix=": dial tcp: lookup https on 127.0.0.53:53: server misbehaving
2

hi,

this worked for me, specify the provider

rclone lsd -vv ":s3,provider=Wasabi,access_key_id=redacted,endpoint='https://s3.us-east-2.wasabisys.com',secret_access_key=redacted":
DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "lsd" "-vv" ":s3,provider=Wasabi,access_key_id=redacted,endpoint='https://s3.us-east-2.wasabisys.com',secret_access_key=redacted:"]
DEBUG : Creating backend with remote ":s3,provider=Wasabi,access_key_id=redacted,endpoint='https://s3.us-east-2.wasabisys.com',secret_access_key=redacted:"
DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
DEBUG : :s3: detected overridden config - adding "{32FrO}" suffix to name
DEBUG : fs cache: renaming cache item ":s3,provider=Wasabi,access_key_id=redacted,endpoint='https://s3.us-east-2.wasabisys.com',secret_access_key=redacted:" to be canonical ":s3{32FrO}:"
          -1 2022-02-09 09:49:42        -1 zork
3

This is a shell quoting problem. You need

rclone ls ':s3,access_key_id=XXX,endpoint="https://s3.ourdomain.com",secret_access_key=YYY:bucketName'

The shell doesn't pass the " on to rclone unless you put them in '

4

hi,

just curious, as connection strings are oh so confusing, second only to filters.

i thought the issue was that the OP did not use provider=Ceph

the command i posted, works both .sh and shell/terminal.
and in terms of single/double quotes, follows the documentation at https://rclone.org/docs/#connection-strings

5

He he!

That probably is an issue but the main reason it didn't work was the quoting.

Your correct example had the "quotes" around it unlike the OP.

This is the hint that the shell has removed the ' before they got to rclone and that the : in the URL http:// has been taken as the end of connection string.

Shell quoting is very confusing. That's the main reason for the debug output where rclone shows you the command line it was run with. This is after the shell has removed the quotes.

6

Thank you @ncw and @asdffdsa for your prompt responses. If I had read the documentation more carefully and looked at the debug output more carefully, I should have been able to figure this out, apologies!
Your help is much appreciated.

7

And by the way, what difference does it make whether I supply the parameter provider=Ceph or not?

8

It turns off features Ceph doesn't understand so its use is recommended.

9

Ok makes sense, thanks a lot!

10

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.