Rclone failing with validation errors on Tencent COS

@ncw

Hm, no, path_style in the config doesn't make a difference. But if I look at the S3 documentation Tencent uses the opposite - virtual hosting.

The problem (or rather the expected behavior, but problem with the previous config we had) is that rclone 1.51 adds the bucket name itself, when this is passed on the commandline, so you end with BUCKET.BUCKET in the URL.

So to summarize:

  • Proper configuration of rclone for s3 with virtual hosting would not include the bucket name in the endpoint address
  • Instead the bucket name needs to be passed on the commandline (like it is the case for other providers, too)
  • In this configuration rclone ls will list every bucket the access id has access to, but commands such as sync won't accidentally overwrite any data because it would complain with the validation error above

@ncw Does that summarize it correctly?

my config file is password protected.
so i would not be able to paste .ini into it correct?

So does AWS for all new buckets.

That is probably to be expected. The endpoint should be without the bucket name - that is how it works with AWS too.

I could do a one_bucket_only config for s3 backends most likely. No-one has asked for it so far though :wink:

Corrrect

Correct

Rclone won't go overwriting stuff in other buckets if you passed the bucket in on the command line.

I quite like the alias idea for keeping it neat.

If you don't want rclone to access those other buckets then you should look into limited permission credentials which would allow rclone only to access one bucket (you'll still need the bucket name!). I don't know how to do that on tencent, but it is pretty easy on AWS.

Yes you'd need to go through rclone config or decrypt the file for a moment then re-encrypt it.

That is probably to be expected. The endpoint should be without the bucket name - that is how it works with AWS too.

Got it, thanks for the explanation.

I could do a one_bucket_only config for s3 backends most likely. No-one has asked for it so far though :wink:

Well, as a matter of fact the previous rclone version had this feature, even if it was by accident :smiley:

Because rclone lsd remote:. with v1.46 and the old configuration (endpoint including bucket name) returns the exact some output as rclone lsd remote:BUCKETNAME with v1.51 and the correct configuration (endpoint without bucket name).

So I assume the change is that rclone nowadays construct the virtual hosting endpoint itself, while previously it didn't? Well, independent of the reason it worked before: you helped me find a configuration that works. Thanks for that.

I'll try the alias functionality, but to be honest, I do not see much value in that approach. I was worried about accidental misuse mostly and not so much about doing something with malicious intent (which can really only be solved on the cloud providers access control level), but as you taught me that is not really possible with the correct configuration.

For other cloud providers we are passing the container-name, so I assume there is no reason in not doing that for Tencent.

(Oh, and just for the sake of completeness: the configuration also works with the old version, which makes migration easy :slight_smile:

1 Like

:slight_smile:

Actually it is the AWS SDK which does this. I'm not sure exactly why it has changed but it is maybe the SDK...

Grand!

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.