SpectraLogic provider in v1.72.1 not working

stephenmcconnachie · December 12, 2025, 4:41pm

What is the problem you are having with rclone?

The SpectraLogic provider (under the S3 category) in latest version 1.72.1 - is not working with our SpectraLogic BlackPearl gateway.

It may be caused by the AWS SDK v2, which I think adds ?x-id= to a GET request. I used –dump to get the headers, and pasted that below for reference. But this is how the GET looks:

GET /?x-id=ListBuckets HTTP/1.1

But it may be that the ListBuckets is not supported in SpectraLogic Black Pearl API - instead the command there is GetBuckets, possibly.

This is the Spectra Logic Black Pearl API documentation for reference - this link being the Get Buckets command:

https://developer.spectralogic.com/doc/ds3api/5.4/DS3%20API%20Reference.htm#DS3APIReferenceFlare/Get_Buckets__Get_Service.htm#commonbucketoperations_1227021396_1062266?TocPath=Amazon%2520S3%2520Operations%257CAmazon%2520S3%2520Bucket%2520Operations%257C_____4

I installed the legacy 1.62 version of rclone and used the generic Other provider, and that works with the SpectraLogic Black Pearl API.

But I want to be using the latest version, as I want to use the desktop UI from GitHub - rclone-ui/rclone-ui: The cross-platform desktop GUI for rclone & S3. - which does not work with the legacy rclone 1.62 version.

So I would like to look at getting the SpectraLogic provider to work in 1.72, if possible.

Run the command 'rclone version' and share the full output of the command.

rclone v1.72.1
os/version: ubuntu 24.04 (64 bit)
os/kernel: 6.11.0-24-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.25.5
go/linking: static
go/tags: noned "image.png"

Which cloud storage system are you using? (eg Google Drive)

It’s an on-premises SpectraLogic Black Pearl S3 gateway, which lets us interact with Spectra Logic data tape libraries using Python SDK and other methods.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone lsd SpectraLogic_BP2:

The rclone config contents with secrets removed.

[SpectraLogic_BP2]
type = s3
provider = SpectraLogic
access_key_id = redacted
secret_access_key = redacted
endpoint = http://172.18.7.102

A log from the command with the `-vv` flag

rclone -vv lsd SpectraLogic_BP2:
2025/12/12 16:26:18 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "-vv" "lsd" "SpectraLogic_BP2:"]
2025/12/12 16:26:18 DEBUG : Creating backend with remote "SpectraLogic_BP2:"
2025/12/12 16:26:18 DEBUG : Using config file from "/home/mcconnachies/.config/rclone/rclone.conf"
2025/12/12 16:26:18 ERROR : error listing: operation error S3: ListBuckets, https response error StatusCode: 400, RequestID: 158697, HostID: , api error InvalidUri: InvalidUri[400]: Request was malformed: No handlers exist that can handle the request. It looks like you meant to use one of these handlers: {com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketsRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketRequestHandler=Query Parameters Required: [], Optional: [delimiter, marker, maxKeys, prefix, versions], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadObjectRequestHandler=Query Parameters Required: [], Optional: [version_id], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadBucketRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.ListMultiPartUploadPartsRequestHandler=Query Parameters Required: [uploadId], Optional: [maxParts, partNumberMarker]}
2025/12/12 16:26:18 DEBUG : 4 go routines active
2025/12/12 16:26:18 NOTICE: Failed to lsd with 2 errors: last error was: operation error S3: ListBuckets, https response error StatusCode: 400, RequestID: 158697, HostID: , api error InvalidUri: InvalidUri[400]: Request was malformed: No handlers exist that can handle the request. It looks like you meant to use one of these handlers: {com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketsRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketRequestHandler=Query Parameters Required: [], Optional: [delimiter, marker, maxKeys, prefix, versions], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadObjectRequestHandler=Query Parameters Required: [], Optional: [version_id], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadBucketRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.ListMultiPartUploadPartsRequestHandler=Query Parameters Required: [uploadId], Optional: [maxParts, partNumberMarker]}

A log from the command with `–dump headers`

rclone lsd SpectraLogic_BP2: --dump headers 2>&1 | head -60
2025/12/12 16:26:54 NOTICE: Automatically setting -vv as --dump is enabled
2025/12/12 16:26:54 DEBUG : rclone: Version "v1.72.1" starting with parameters ["rclone" "lsd" "SpectraLogic_BP2:" "--dump" "headers"]
2025/12/12 16:26:54 DEBUG : Creating backend with remote "SpectraLogic_BP2:"
2025/12/12 16:26:54 DEBUG : Using config file from "/home/mcconnachies/.config/rclone/rclone.conf"
2025/12/12 16:26:54 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/12/12 16:26:54 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2025/12/12 16:26:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/12/12 16:26:54 DEBUG : HTTP REQUEST (req 0xc0003d2c80)
2025/12/12 16:26:54 DEBUG : GET /?x-id=ListBuckets HTTP/1.1
Host: 172.18.7.102
User-Agent: rclone/v1.72.1
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 87073b14-0d09-4592-8673-4bfa5a6f767c
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251212T162654Z

2025/12/12 16:26:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/12/12 16:26:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/12/12 16:26:54 DEBUG : HTTP RESPONSE (req 0xc0003d2c80)
2025/12/12 16:26:54 DEBUG : HTTP/1.1 400 Bad Request
Connection: close
Content-Length: 1012
Content-Language: en-US
Content-Type: text/xml;charset=UTF-8
Date: Fri, 12 Dec 2025 16:26:54 GMT
X-Amz-Request-Id: 158699

2025/12/12 16:26:54 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/12/12 16:26:54 ERROR : error listing: operation error S3: ListBuckets, https response error StatusCode: 400, RequestID: 158699, HostID: , api error InvalidUri: InvalidUri[400]: Request was malformed: No handlers exist that can handle the request. It looks like you meant to use one of these handlers: {com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketsRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketRequestHandler=Query Parameters Required: [], Optional: [delimiter, marker, maxKeys, prefix, versions], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadObjectRequestHandler=Query Parameters Required: [], Optional: [version_id], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadBucketRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.ListMultiPartUploadPartsRequestHandler=Query Parameters Required: [uploadId], Optional: [maxParts, partNumberMarker]}
2025/12/12 16:26:54 DEBUG : 4 go routines active
2025/12/12 16:26:54 NOTICE: Failed to lsd with 2 errors: last error was: operation error S3: ListBuckets, https response error StatusCode: 400, RequestID: 158699, HostID: , api error InvalidUri: InvalidUri[400]: Request was malformed: No handlers exist that can handle the request. It looks like you meant to use one of these handlers: {com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketsRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.GetBucketRequestHandler=Query Parameters Required: [], Optional: [delimiter, marker, maxKeys, prefix, versions], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadObjectRequestHandler=Query Parameters Required: [], Optional: [version_id], com.spectralogic.s3.server.handler.reqhandler.amazons3.HeadBucketRequestHandler=Query Parameters Required: [], Optional: [], com.spectralogic.s3.server.handler.reqhandler.amazons3.ListMultiPartUploadPartsRequestHandler=Query Parameters Required: [uploadId], Optional: [maxParts, partNumberMarker]}

ncw · December 18, 2025, 12:02pm

If the x-id is the problem you could try this --s3-use-x-id=false

  --s3-use-x-id Tristate                                Set if rclone should add x-id URL parameters (default unset)

If that works we should add it to the quirks for the SpectraLogic provider.

stephenmcconnachie · December 18, 2025, 1:00pm

Thank you!

I have been using the previous version with the generic Other S3 provider and that interacts successfully.

But in the end I have explored other options for navigating the Black Pearl, not for this issue but because I can’t find a way to prevent rclone from trying to list all files in a bucket / folder. We have a few very flat buckets with over 500,000 files, so listing all is not workable.

Is there a method of limiting the list to a maximum number? I can;t find one in the documentation but I may have missed it?

ncw · December 19, 2025, 10:10pm

Did the --s3-use-x-id=false trick work?

Recent rclones will list the bucket as data is received rather that waiting for the whole bucket listing.

So if you can get v1.72.1 working that will probably fix your problem - you just head the output rclone for however many lines you want.

system · January 18, 2026, 10:10pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.