List --s3-versions files looping bug

What is the problem you are having with rclone?

rclone ls --s3-versions is going into an infinite loop repeating the same files.

the problem seems to be with unversioned files. are relisted infinitely. Only killing the process to stop.

Is there any way to list only versioned files?

Run the command 'rclone version' and share the full output of the command.

rclone v1.64.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.2.0-1018-gcp (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

google cloud storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone ls --s3-versions remote:archive/2022/

The rclone config contents with secrets removed.

type = s3
provider = GCS
access_key_id = 
secret_access_key = 
endpoint =

A log from the command with the -vv flag

2023/11/20 12:43:56 DEBUG : rclone: Version "v1.64.2" starting with parameters ["rclone" "ls" "--s3-versions" "remote:archive/2022/7" "-vv"]
2023/11/20 12:43:56 DEBUG : Creating backend with remote "remote:archive/2022/7"
2023/11/20 12:43:56 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/11/20 12:43:56 DEBUG : remote: detected overridden config - adding "{pO73u}" suffix to name
2023/11/20 12:43:56 DEBUG : Resolving service "s3" region "us-east-1"
2023/11/20 12:43:56 DEBUG : fs cache: renaming cache item "remote:archive/2022/7" to be canonical "remote{pO73u}:nd-srv/pacs-data/archive/2022/7"

the loop log. rename .log to .rar file
log.rar.log (12.6 KB)

This is usually caused by --s3-list-version being set wrong for the provider. Can you try different versions for that and see if it makes a difference?

I tried --s3-list-version=1 and --s3-list-version=2. same problem.

if remove --s3-versions works fine.

Can you do a log with -vv --dump bodies that should show what the problem is.


log.rar.txt (82.7 KB)

rename "log.rar.txt" to "log.txt.rar"

Thank you.

It has definitely got stuck in a loop - you can see it fetching the same thing over and over again

GET /?delimiter=&encoding-type=url&max-keys=1000&prefix=pacs-data%2Farchive%2F2022%2F8%2F&versions= HTTP/1.1

Which is getting a prefix of pacs-data/archive/2022/8/ when I decode it.

But why is the question?

Looking at the list response we see

<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns=''>

...snip lots of similar blocks...


Something strange has gone on here for definite.

According to the AWS docs, this is what the response should look like

<?xml version="1.0" encoding="UTF-8"?>

In particular note that the response should be a ListVersionsResult but gcs has given us a ListBucketResult

This is why rclone is looping because it is expecting to find NextKeyMarker and NextVersionIdMarker but it isn't so it is starting the listing over again each time.

However that doesn't explain why gcs is giving us the wrong listing result.

A bit of searching finds a comment which indicates gcs is being incompatible here. There is some more discussion on the drupal forum about this.

I can't find a defininitive statement in the GCS docs about this, but looking at the docs for the API call it just doesn't look compatible with S3.

Rclone should definitely notice the response is wrong and give an error which would have saved your looping problem.

I have fixed that here:

v1.65.0-beta.7523.15747cc9f.fix-s3-gcs-looping on branch fix-s3-gcs-looping (uploaded in 15-30 mins)

However I think the real problem here is that the GCS S3 interface isn't fully AWS S3 compatible. I couldn't find any google issues about this so it would probably be worth making one if you agree with my diagnosis.

I can't make a workaround for this - the AWS S3 SDK is very inflexible unfortunately.

1 Like


2023/11/21 14:03:56 Failed to ls: s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker

by now we solved our problem using the google api... (slow and painful)

It's not something we need to use often. In the future we can think and see how to work with versioning using the Google API.

Glad that worked :slight_smile:

Sorry rclone couldn't hack it for you this time!

Do you think it is worth reporting a Google bug about this? I guess if I do that, then at least I can link to it in the rclone docs!

I have reported lots of bugs but Google rarely seem do anything about them alas.


no problem. Thank you for the assistance!

where is the right place to report this to google? I never went back.

I did it here. Not sure is the right place.

That is perfect. I added some more detail to the bug report!

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.65

I put a note in the docs also.

1 Like