AWS S3 has released "Directory bucket" for a while, and it seems provides much better performance than general-purpose buckets. However, uploading data to it is only supported via AWS SDK/CLI so far.
Does rclone have any plan to support it?
I skimmed the docs on directory buckets - I think if you create one then rclone should be able to upload to it - as far as rclone is concerned it is a bucket with a funny name.
The only specific bit of API for them I found was ListDirectoryBuckets which oddly is not supported by the AWS go SDK v2 yet (I'll make a feature request now - see issue #2779).
Is this not it?
https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/service/s3#Client.ListDirectoryBuckets
Looks like it is! How did I miss that? Thanks
I actually tested it with both "rclone ls" and "rclone copy".
Then "rclone ls" resulted in "Failed to ls: directory not found"
And the "rclone copy" resulted in "Failed to copy: AccessDenied: The requested bucket name ends with a reserved suffix"
I'm not sure about the implementation, does "rclone copy" try creating bucket if does not exist? Since the error message looks like something happened only in bucket creation.
UPDATE:
I also found a flag "--s3-no-check-bucket", then it resulted in another error: "Failed to copy: NoSuchBucket: The specified bucket does not exist"
Can you post the result of rclone ls -vv --dump bodies --retries 1 remote:bucket please?
2024/09/12 17:55:36 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["rclone" "ls" "-vv" "--dump" "bodies" "--retries" "1" "s3:frank-training--use1-az6--x-s3"]
2024/09/12 17:55:36 DEBUG : Creating backend with remote "s3:frank-training--use1-az6--x-s3"
2024/09/12 17:55:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/12 17:55:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/12 17:55:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : HTTP REQUEST (req 0xc000a3a480)
2024/09/12 17:55:36 DEBUG : POST / HTTP/1.1
Host: sts.us-east-1.amazonaws.com
User-Agent: rclone/
Content-Length: 1189
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Accept-Encoding: gzip
Action=AssumeRoleWithWebIdentity&RoleArn=arn%3Aaws%3Aiam%3A%3A846209230485%3Arole%2FCloudflareBenchmarkS3Access&RoleSessionName=1726131336747799851&Version=2011-06-15&WebIdentityToken=[MASKED]
2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : HTTP RESPONSE (req 0xc000a3a480)
2024/09/12 17:55:36 DEBUG : HTTP/1.1 200 OK
Content-Length: 2299
Content-Type: text/xml
Date: Thu, 12 Sep 2024 08:55:35 GMT
X-Amzn-Requestid: 9b759876-4172-483a-b710-d650b218a81b
2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : HTTP RESPONSE (req 0xc00063e5a0)
2024/09/12 17:55:36 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 12 Sep 2024 08:55:35 GMT
Server: AmazonS3
X-Amz-Id-2: i0wjQ/4ptbCfZNqTwke2uA+VhJ07BzNMMHSnp+rdjsvvskUSYe3fxATEjzwvs1zl7yNpINwL0ss=
X-Amz-Request-Id: QT37YQBKCDS8ESYY
140
<?xml version="1.0" encoding="UTF-8"?>NoSuchBucketThe specified bucket does not existfrank-training--use1-az6--x-s3QT37YQBKCDS8ESYYi0wjQ/4ptbCfZNqTwke2uA+VhJ07BzNMMHSnp+rdjsvvskUSYe3fxATEjzwvs1zl7yNpINwL0ss=
0
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : 6 go routines active
2024/09/12 17:55:36 Failed to ls: directory not found
2024/09/12 17:55:36 DEBUG : rclone: Version "v1.60.1-DEV" starting with parameters ["rclone" "ls" "-vv" "--dump" "bodies" "--retries" "1" "s3:frank-training--use1-az6--x-s3"]
2024/09/12 17:55:36 DEBUG : Creating backend with remote "s3:frank-training--use1-az6--x-s3"
2024/09/12 17:55:36 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/12 17:55:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/12 17:55:36 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : HTTP REQUEST (req 0xc000a3a480)
2024/09/12 17:55:36 DEBUG : POST / HTTP/1.1
Host: sts.us-east-1.amazonaws.com
User-Agent: rclone/
Content-Length: 1189
Content-Type: application/x-www-form-urlencoded; charset=utf-8
Accept-Encoding: gzip
Action=AssumeRoleWithWebIdentity&RoleArn=arn%3Aaws%3Aiam%3A%3A846209230485%3Arole%2FCloudflareBenchmarkS3Access&RoleSessionName=1726131336747799851&Version=2011-06-15&WebIdentityToken=[MASKED]2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : HTTP RESPONSE (req 0xc000a3a480)
2024/09/12 17:55:36 DEBUG : HTTP/1.1 200 OK
Content-Length: 2299
Content-Type: text/xml
Date: Thu, 12 Sep 2024 08:55:35 GMT
X-Amzn-Requestid: 9b759876-4172-483a-b710-d650b218a81b
2024/09/12 17:55:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : HTTP RESPONSE (req 0xc00063e5a0)
2024/09/12 17:55:36 DEBUG : HTTP/1.1 404 Not Found
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 12 Sep 2024 08:55:35 GMT
Server: AmazonS3
X-Amz-Id-2: i0wjQ/4ptbCfZNqTwke2uA+VhJ07BzNMMHSnp+rdjsvvskUSYe3fxATEjzwvs1zl7yNpINwL0ss=
X-Amz-Request-Id: QT37YQBKCDS8ESYY
140
<?xml version="1.0" encoding="UTF-8"?>NoSuchBucketThe specified bucket does not existfrank-training--use1-az6--x-s3QT37YQBKCDS8ESYYi0wjQ/4ptbCfZNqTwke2uA+VhJ07BzNMMHSnp+rdjsvvskUSYe3fxATEjzwvs1zl7yNpINwL0ss=
0
2024/09/12 17:55:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/12 17:55:36 DEBUG : 6 go routines active
2024/09/12 17:55:36 Failed to ls: directory not found
hi,
that is a old, dev version of rclone using the old, end-of-life, aws sdk 1.0
should rclone selfupdate, post output of rclone version and re-run the previous command.
1 Like
Ohh!! Sorry I installed rclone by "apt install" and I completely missed that. After upgrading to 1.68.0, the "rclone ls" works perfectly. However when doing "rclone copy --s3-disable-checksum", I got this error:
Failed to copy: operation error S3: PutObject, https response error StatusCode: 501, RequestID: 01c6612be4000191e8d3edb404074bc25946d5fd, HostID: yfx6T, api error NotImplemented: This bucket does not support Content Md5 header.
1 Like
hmm, not sure why you got that error.
can you copy a single file and post a full debug log.
Sure, here is the log when I tried with a single local file:
2024/09/13 13:04:39 NOTICE: Automatically setting -vv as --dump is enabled
2024/09/13 13:04:39 DEBUG : rclone: Version "v1.68.0" starting with parameters ["rclone" "copy" "testdata" "s3:r2-benchmark--use1-az4--x-s3" "--retries" "1" "--dump" "bodies"]
2024/09/13 13:04:39 DEBUG : Creating backend with remote "testdata"
2024/09/13 13:04:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2024/09/13 13:04:39 DEBUG : fs cache: renaming cache item "testdata" to be canonical "/testdata"
2024/09/13 13:04:39 DEBUG : Creating backend with remote "s3:r2-benchmark--use1-az4--x-s3"
2024/09/13 13:04:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/13 13:04:39 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : HTTP REQUEST (req 0xc0000ccdc0)
2024/09/13 13:04:39 DEBUG : GET /?session= HTTP/1.1
Host: r2-benchmark--use1-az4--x-s3.s3express-use1-az4.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 901dd5c0-7171-4587-bcaf-d419a4914b68
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240913T040439Z
X-Amz-Security-Token: [MASKED]
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 DEBUG : HTTP RESPONSE (req 0xc0000ccdc0)
2024/09/13 13:04:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 1719
Content-Type: application/xml
Date: Fri, 13 Sep 2024 04:04:39 GMT
Server: AmazonS3
X-Amz-Id-2: RkfjS1
X-Amz-Request-Id: 01ee1d8a14000191e98df19f040761c9ba2cd248
[MASKED][MASKED][MASKED]2024-09-13T04:09:39Z
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : HTTP REQUEST (req 0xc0005aa500)
2024/09/13 13:04:39 DEBUG : GET /?delimiter=%2F&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: r2-benchmark--use1-az4--x-s3.s3express-use1-az4.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.0
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: d33763b2-eca9-4478-8399-5fdd204dce0e
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20240913T040439Z
X-Amz-S3session-Token: [MASKED]
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 DEBUG : HTTP RESPONSE (req 0xc0005aa500)
2024/09/13 13:04:39 DEBUG : HTTP/1.1 200 OK
Content-Length: 307
Content-Type: application/xml
Date: Fri, 13 Sep 2024 04:04:39 GMT
Server: AmazonS3
X-Amz-Bucket-Region: us-east-1
X-Amz-Id-2: IB3CeAz8xyErHIOz4
X-Amz-Request-Id: 0188ce48c5000191e98df1c804079a6f383217d6
r2-benchmark--use1-az4--x-s301000urlfalse/
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 DEBUG : test.txt: Need to transfer - File not found at Destination
2024/09/13 13:04:39 DEBUG : S3 bucket r2-benchmark--use1-az4--x-s3: Waiting for checks to finish
2024/09/13 13:04:39 DEBUG : S3 bucket r2-benchmark--use1-az4--x-s3: Waiting for transfers to finish
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : HTTP REQUEST (req 0xc0005aac80)
2024/09/13 13:04:39 DEBUG : PUT /test.txt?x-id=PutObject HTTP/1.1
Host: r2-benchmark--use1-az4--x-s3.s3express-use1-az4.us-east-1.amazonaws.com
User-Agent: rclone/v1.68.0
Content-Length: 5
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 15002fcf-4940-4aba-9dc1-e061f3b30e82
Amz-Sdk-Request: attempt=1; max=1
Authorization: XXXX
Content-Md5: 2Oj8otwPiW/Xy0ywAxuiSQ==
Content-Type: text/plain; charset=utf-8
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20240913T040439Z
X-Amz-Meta-Mtime: 1726200035.866676262
X-Amz-S3session-Token: [MASKED]
test
2024/09/13 13:04:39 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 DEBUG : HTTP RESPONSE (req 0xc0005aac80)
2024/09/13 13:04:39 DEBUG : HTTP/1.1 501 Not Implemented
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 13 Sep 2024 04:04:39 GMT
Server: AmazonS3
X-Amz-Id-2: dfXbryY
X-Amz-Request-Id: 01d0bc8602000191e98df1cd0407ad773b4ab921
eb
<?xml version="1.0" encoding="UTF-8"?>NotImplementedThis bucket does not support Content Md5 header.01d0bc8602000191e98df1cd0407ad773b4ab921dfXbryY
0
2024/09/13 13:04:39 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2024/09/13 13:04:39 ERROR : test.txt: Failed to copy: operation error S3: PutObject, https response error StatusCode: 501, RequestID: 01d0bc8602000191e98df1cd0407ad773b4ab921, HostID: dfXbryY, api error NotImplemented: This bucket does not support Content Md5 header.
2024/09/13 13:04:39 ERROR : Attempt 1/1 failed with 1 errors and: operation error S3: PutObject, https response error StatusCode: 501, RequestID: 01d0bc8602000191e98df1cd0407ad773b4ab921, HostID: dfXbryY, api error NotImplemented: This bucket does not support Content Md5 header.
2024/09/13 13:04:39 INFO :
Transferred: 5 B / 5 B, 100%, 0 B/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 0.1s
2024/09/13 13:04:39 DEBUG : 9 go routines active
2024/09/13 13:04:39 NOTICE: Failed to copy: operation error S3: PutObject, https response error StatusCode: 501, RequestID: 01d0bc8602000191e98df1cd0407ad773b4ab921, HostID: dfXbryY, api error NotImplemented: This bucket does not support Content Md5 header.
Hmm....
I found this in the docs
Data integrity with Content-MD5
- General purpose bucket - To ensure that data is not corrupted traversing the network, use the
Content-MD5header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, Amazon S3 returns an error. Alternatively, when the object's ETag is its MD5 digest, you can calculate the MD5 while putting the object to Amazon S3 and compare the returned ETag to the calculated MD5 value.- Directory bucket - This functionality is not supported for directory buckets.
So looks like Content-MD5 is not supported...
Can you try this with the --s3-directory-bucket flag, or put directory_bucket = true into your config.
v1.69.0-beta.8308.3118a01dc.fix-s3-directory-bucket on branch fix-s3-directory-bucket (uploaded in 15-30 mins)
There might be other things which need changing.
Ideally rclone would detect a directory bucket is in use and set this automatically, but I don't know how to do that!
Was that with the --s3-directory-bucket flag? When I use that flag I don't get the Content-Md5 header any more.
If it was, can you show your complete command line please
Sorry I misread the new flag earlier. After adding it, I found that I need to also give --ignore-checksum together to make it work. It'll be nice if this flag can be automatically included when --s3-directory-bucket is given.
rclone would detect a directory bucket is in use and set this automatically,
I guess the only way without server interaction is to rely on the bucket suffix "--x-s3". But I think the current flag is totally ok for us.
How soon we can expect the release of this fix?
Thanks for having another go.
I looked into the checksum thing and there were a few more things I needed to fix.
Please give this a go - you shouldn't need --ignore-checksum or -s3-no-check-bucket any more.
v1.69.0-beta.8311.e0e5195b2.fix-s3-directory-bucket on branch fix-s3-directory-bucket (uploaded in 15-30 mins)
This now passes the integration tests on a directory bucket
I decided to go with the flag for the moment rather than auto detect so people will make a deliberate choice to use directory buckets.
Once you've give the above a test I'll merge it to the beta then it will be released in a stable release in v1.69 which is scheduled for 20th October.
Thank you! I've tested it and it works perfectly!
Thanks for testing 
I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.69
Let me know if you find any other problems.
In my testing I found quite a few oddities which I think are bugs in the service - hopefully AWS will fix these shortly!
1 Like
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.