Unable to mkdir to create bucket on S3 over Crypted Remote?

Help and Support
1

I'm attempting to create an S3 bucket over a crypted remote, but it doesn't seem to work. I receive a "METHOD NOT ALLOWED" response. I'm able to create an S3 bucket over the unencrypted remote that the Crypt points to, so I figured I have the correct permissions. I've tested this on version 1.39-034-gacd55a8f and 1.38-234-g4c0e2f9b, on Windows 10. My remote points to the "UnencryptedRemote:". I attempted to create the bucket over the unencrypted remote, but then it seems I can't actually move anything over the Crypted connection.

The one thing that did work, was creating the bucket, and then pointing the Crypt directly to that bucket, then moving/copying files in. I can make this work, but will require a bit of scripting updates with my current script (It's automatically creating buckets, so it will also need to automatically create encrypted remotes for each bucket). Does crypt not work with the Root of an S3 remote?

Example with crypt (Latest beta version):

rclone --config "\some\config\location\rclone.conf" mkdir AWS-Encrypted:New-Remote-d41d8 -vv

Output:

2018/01/18 08:45:03 DEBUG : Using config file from "\some\config\location\rclone.conf"
2018/01/18 08:45:03 DEBUG : rclone: Version "v1.39-034-gacd55a8f" starting with parameters ["C:\rclonetest\rclone.exe" "--config" "\some\config\location\rclone.conf" "mkdir" "AWS-Encrypted:New-Remote-d41d8" "-vv"]
2018/01/18 08:45:03 INFO : Encrypted drive 'AWS-Encrypted:New-Remote-d41d8': Modify window is 1ns
2018/01/18 08:45:03 DEBUG : Encrypted drive 'AWS-Encrypted:New-Remote-d41d8': Making directory
2018/01/18 08:45:03 ERROR : Attempt 1/3 failed with 1 errors and: MethodNotAllowed: Method Not Allowed
status code: 405, request id: B248B543756BB9EB, host id: lj6Aj6NdqWFBwetjOPkOxY49m4NZfcXaeLDd80g2EbcYEZVGhGw40oVP4ldkVh5obE+tZDm5Xyo=
2018/01/18 08:45:03 DEBUG : Encrypted drive 'AWS-Encrypted:New-Remote-d41d8': Making directory
2018/01/18 08:45:03 ERROR : Attempt 2/3 failed with 1 errors and: MethodNotAllowed: Method Not Allowed
status code: 405, request id: 90AC009321C05FCB, host id: BwzoBiQnH32JuP1XvXsOTf7tg+Ym0Z8pVjQeW5kkcB9jB9iRsGlxUl3JRP2y4yvlNHbLL5NBaDg=
2018/01/18 08:45:03 DEBUG : Encrypted drive 'AWS-Encrypted:New-Remote-d41d8': Making directory
2018/01/18 08:45:03 ERROR : Attempt 3/3 failed with 1 errors and: MethodNotAllowed: Method Not Allowed
status code: 405, request id: 7EC9BB639DBA985E, host id: Ywqd1A84SM51frbI4fN67t80/4HfhcKCcWl3ySrwywL1OzyA+tWmI2KNHmblxXeSlpf99tX9SVM=
2018/01/18 08:45:04 Failed to mkdir: MethodNotAllowed: Method Not Allowed
status code: 405, request id: 7EC9BB639DBA985E, host id: Ywqd1A84SM51frbI4fN67t80/4HfhcKCcWl3ySrwywL1OzyA+tWmI2KNHmblxXeSlpf99tX9SVM=

When I don't use the Crypted remote, I'm able to create it just fine:

rclone.exe --config "\some\config\location\rclone.conf" mkdir AWS-Unencrypted:New-Remote-d41d8 -vv

Output:

2018/01/18 08:49:24 DEBUG : Using config file from "\some\config\location\rclone.conf"
2018/01/18 08:49:24 DEBUG : rclone: Version "v1.39-034-gacd55a8f" starting with parameters ["C:\rclonetest\rclone.exe" "--config" "\some\config\location\rclone.conf" "mkdir" "AWS-Unencrypted:New-Remote-d41d8" "-vv"]
2018/01/18 08:49:24 INFO : S3 bucket New-Remote-d41d8: Modify window is 1ns
2018/01/18 08:49:24 DEBUG : S3 bucket New-Remote-d41d8: Making directory
2018/01/18 08:49:25 DEBUG : Go routines at exit 4
2018/01/18 08:49:25 DEBUG : rclone: Version "v1.39-034-gacd55a8f" finishing with parameters ["C:\rclonetest\rclone.exe" "--config" "\some\config\location\rclone.conf" "mkdir" "AWS-Unencrypted:New-Remote-d41d8" "-vv"]

2

Here is what it says in the docs...

This means that if you
are using a bucket based remote (eg S3, B2, swift) then you should
probably put the bucket in the remote s3:bucket. If you just use
s3: then rclone will make encrypted bucket names too (if using file
name encryption) which may or may not be what you want.

Indicating that it should work but isn't recommended...

However I think that is probably wrong and it doesn't work - you'll need to point to a bucket. The reason for that is that the bare remote s3: works just well enough to list the buckets but you can't actually use the files within...

I may fix this at some point (it is complicated) but for now I think the answer is no, crypt does not work with the root of an s3 remote.

3

Yeah I saw that it isn’t recommended, according to the documentation and also when configuring the remote. This is probably why. Thanks for the response!

4

@ncw is there a recommended way to check if an S3 bucket exists before attempting to copy anything to it? Or is assuming an empty response from

rclone lsd EncryptedRemote:

a worthy method?

5

That should give an error if the bucket doesn’t exist I would have thought.