What is the problem you are having with rclone?
Rclone with the s3 protocol does not handle malformed xml responses well. This is a reproducer so that everybody can see the behavior.
Run the command 'rclone version' and share the full output of the command.
rclone v1.60.1-DEV
- os/version: debian trixie/sid (64 bit)
- os/kernel: 6.10.7-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.22.6
- go/linking: dynamic
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS (reproducer); in fact, I use my own setup which classifies as provider=Other in rclone.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Preparation: Upload an object with ^C (ascii 3) in the object key (to enter that in a terminal application, use ctrl+v ctrl+c):
rclone copy empty s3:rclone-demo-bug/key^C
Normal listing uses URL-encoding and works:
rclone lsf --dump bodies s3:rclone-demo-bug
[...]
DEBUG : HTTP RESPONSE (req 0xc00083b320)
DEBUG : HTTP/1.1 200 OK
[...]
<Key>key%03/empty</Key>
[...]
However listing with url encoding switched off (which is the default state for the Other provider) does not result in a user visible error, but in a long retry loop:
rclone lsl --s3-list-url-encode=false -vv --dump bodies s3:rclone-demo-bug
[...]
DEBUG : HTTP REQUEST (req 0xc0009e8ea0)
DEBUG : GET /?delimiter=&list-type=2&max-keys=1000&prefix= HTTP/1.1
[...]
DEBUG : HTTP RESPONSE (req 0xc0009e8ea0)
DEBUG : HTTP/1.1 200 OK
[...]
<Key>key/empty</Key>
[...]
Everything about this is already known (compare upstream XML decoding bug.. command rclone info has broken s3 remote · Issue #3345 · rclone/rclone · GitHub) which is my justification to only show a small sensible excerpt of the output.
The bad thing about this is that rclone does not accept the error from the XML decoder as fatal, but retries the HTTP request 10 times for 50 seconds which gives a bad user experience.
Only after all these senseless requests prints the error message (with -vv, invisible to the normal user):
DEBUG : S3 bucket rclone-demo-bug: Retrying listing because of characters which can't be XML encoded
DEBUG : pacer: low level retry 1/2 (error SerializationError: failed to decode REST XML response
status code: 200, request id: RNKV2JPK36A5P8KY
caused by: XML syntax error on line 2: illegal character code U+0003)
DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
DEBUG : HTTP REQUEST (req 0xc000420000)
DEBUG : GET /?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix= HTTP/1.1
Host: rclone-demo-bug.s3.eu-central-1.amazonaws.com
[...]
... and then finally repeats the request with encoding-type=url added, which to the user looks like AWS would respond really slow to the list request.
It would give a much better user experience to error out early on malformed xml, teach the user how to fix their config and have a quickly responding backend from there onward.
The rclone config contents with secrets removed.
type = s3
provider = AWS