Copying from IBM COS throws deserialization error

Help and Support
1

What is the problem you are having with rclone?

I want to copy files from IBM COS to AWS S3.

Run the command 'rclone version' and share the full output of the command.

rclone v1.71.2
- os/version: fedora 40 (64 bit)
- os/kernel: 6.14.5-100.fc40.x86_64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.25.3
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy 'ibm-s3:base-gb-lfc/' 's3:bucket/' --dry-run -P --fast-list --check-first -vvv

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[ibm-s3]
type = s3
provider = IBMCOS
access_key_id = XXX
secret_access_key = XXX
endpoint = s3.eu-gb.cloud-object-storage.appdomain.cloud
acl = private

A log from the command that you were trying to run with the -vv flag

2025/10/24 13:48:00 DEBUG : rclone: Version "v1.71.2" starting with parameters ["rclone" "copy" "ibm-s3:base-gb-lfc/" "s3:bucket/" "--dry-run" "-P" "--fast-list" "--check-first" "-vvv"]
2025/10/24 13:48:00 DEBUG : Creating backend with remote "ibm-s3:base-gb-lfc/"
2025/10/24 13:48:00 DEBUG : Using config file from "/home/raphael/.config/rclone/rclone.conf"
2025/10/24 13:48:00 DEBUG : fs cache: renaming cache item "liverpool-ibm-s3:base-gb-lfc/" to be canonical "ibm-s3:base-gb-lfc"
2025/10/24 13:48:00 DEBUG : Creating backend with remote "s3:bucket/"
2025/10/24 13:48:00 DEBUG : fs cache: renaming cache item "s3:bucket/" to be canonical "s3:bucket/"
2025/10/24 13:48:00 INFO  : S3 bucket <bucket> path .: Running all checks before starting transfers
2025/10/24 13:48:26 DEBUG : S3 bucket base-gb-lfc: Retrying listing because of characters which can't be XML encoded
2025/10/24 13:48:26 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjects, https response error StatusCode: 200, RequestID: 1aa5f547-7f23-40ca-8a10-43637b9137c9, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F)
2025/10/24 13:48:26 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2025/10/24 13:48:26 DEBUG : pacer: Reducing sleep to 0s
2025/10/24 13:49:06 ERROR : S3 bucket base-gb-lfc: error reading source root directory: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: b65729b0-bce8-4a63-9f83-cc20418730ef, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
2025/10/24 13:49:06 DEBUG : S3 bucket <redacted> path .: Waiting for checks to finish
2025/10/24 13:49:06 INFO  : S3 bucket <redacted> path .: Checks finished, now starting transfers
2025/10/24 13:49:06 DEBUG : S3 bucket <redacted> path .: Waiting for transfers to finish
2025/10/24 13:49:06 ERROR : Attempt 1/3 failed with 1 errors and: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: b65729b0-bce8-4a63-9f83-cc20418730ef, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
2025/10/24 13:49:06 INFO  : S3 bucket <redacted> path .: Running all checks before starting transfers
2025/10/24 13:49:32 DEBUG : S3 bucket base-gb-lfc: Retrying listing because of characters which can't be XML encoded
2025/10/24 13:49:32 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjects, https response error StatusCode: 200, RequestID: 05601478-a483-44c2-bc1d-4cdb7c81347a, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F)
2025/10/24 13:49:32 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2025/10/24 13:49:32 DEBUG : pacer: Reducing sleep to 0s
2025/10/24 13:50:15 ERROR : S3 bucket base-gb-lfc: error reading source root directory: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: 28ef3e38-9d6e-49a4-8017-a6a838b89fd8, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
2025/10/24 13:50:15 DEBUG : S3 bucket <redacted> path .: Waiting for checks to finish
2025/10/24 13:50:15 INFO  : S3 bucket <redacted> path .: Checks finished, now starting transfers
2025/10/24 13:50:15 DEBUG : S3 bucket <redacted> path .: Waiting for transfers to finish
2025/10/24 13:50:15 ERROR : Attempt 2/3 failed with 1 errors and: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: 28ef3e38-9d6e-49a4-8017-a6a838b89fd8, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
2025/10/24 13:50:15 INFO  : S3 bucket <redacted> path .: Running all checks before starting transfers
2025/10/24 13:50:41 DEBUG : S3 bucket base-gb-lfc: Retrying listing because of characters which can't be XML encoded
2025/10/24 13:50:41 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjects, https response error StatusCode: 200, RequestID: 32fdeaa4-f479-47b7-8a61-651cf1492bec, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F)
2025/10/24 13:50:41 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2025/10/24 13:50:41 DEBUG : pacer: Reducing sleep to 0s
2025/10/24 13:51:26 ERROR : S3 bucket base-gb-lfc: error reading source root directory: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: cf3e31f4-6eaf-4ae4-a461-ad7efcbed601, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
[...]
2025/10/24 13:51:26 ERROR : Attempt 3/3 failed with 1 errors and: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: cf3e31f4-6eaf-4ae4-a461-ad7efcbed601, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 0 / 0, -, Listed 1829676
Elapsed time:      3m26.1s
2025/10/24 13:51:26 NOTICE: 
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (retrying may help)
Checks:                 0 / 0, -, Listed 1829676
Elapsed time:      3m26.1s

2025/10/24 13:51:26 DEBUG : 8 go routines active
2025/10/24 13:51:26 NOTICE: Failed to copy: operation error S3: ListObjects, https response error StatusCode: 200, RequestID: cf3e31f4-6eaf-4ae4-a461-ad7efcbed601, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F

I get the same issue with `rclone size ‘ibm-s3:base-gb-lfc/‘

Thanks for the help :folded_hands:

2

welcome to the forum,

can you reproduce using rclone ls or rclone lsd ?

for a deeper look, can use -vv --retries=1 with a dump flag such as --dump=headers

3

Yes I’m able to reproduce with rclone ls:

[...]
2025/10/24 14:07:48 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2025/10/24 14:07:49 DEBUG : S3 bucket base-gb-lfc: Retrying listing because of characters which can't be XML encoded
2025/10/24 14:07:49 DEBUG : pacer: low level retry 1/2 (error operation error S3: ListObjects, https response error StatusCode: 200, RequestID: be505451-8c4e-48ea-81a7-916609a8877b, HostID: , deserialization failed, failed to decode response body, XML syntax error on line 1: illegal character code U+001F)
2025/10/24 14:07:49 DEBUG : pacer: Rate limited, increasing sleep to 10ms
2025/10/24 14:07:49 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2025/10/24 14:07:49 DEBUG : HTTP REQUEST (req 0xc0002d9040)
2025/10/24 14:07:49 DEBUG : GET /base-gb-lfc?delimiter=&encoding-type=url&marker=Partnerships%2FNike%2FNike%20Kit%20Shoot%2022-23%2FHome%2FCommercial%2FBobby%20Clark%2FCommercial%20First%20Grade_Jpgs%2F2023_01_2524113.jpg&max-keys=1000&prefix= HTTP/1.1
Host: s3.eu-gb.cloud-object-storage.appdomain.cloud
User-Agent: rclone/v1.71.2
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: 2b323478-d9d7-4c97-a5a4-2401e115b02f
Amz-Sdk-Request: attempt=1; max=10
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20251024T120749Z

It doesn’t happen on all requests, eventually during the listing one request fails and it stops the entire process.

Do you need something else to troubleshoot ?

4

i am just trying to collect the needed information so more experienced forum members have the info they need.

maybe post the output of
rclone ls remote: --vv --retries=1 --low-level-retries=1 --dump=bodies

5

Okay I might have found what’s going on. There’s a unicode char in a path that rclone cannot decode/disallow.

I can’t even paste it here. What options do I have ? Is this something you want to fix ?

6
7

Try this flag

  --s3-list-url-encode Tristate   Whether to url encode listings: true/false/unset (default unset)

Try --s3-list-url-encode=true and --s3-list-url-encode=false

It might be the default is wrong for IBMCOS

8

Unfortunately no luck, I tried both. For now I excluded the only path that’s problematic and copy succeeded.

9

If neither of the flags worked then this is probably a bug in the S3 implementation of IBMCOS. Can you download the file with their web interface? If not you could report a bug.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.