Thank you.
It has definitely got stuck in a loop - you can see it fetching the same thing over and over again
GET /?delimiter=&encoding-type=url&max-keys=1000&prefix=pacs-data%2Farchive%2F2022%2F8%2F&versions= HTTP/1.1
Which is getting a prefix of pacs-data/archive/2022/8/ when I decode it.
But why is the question?
Looking at the list response we see
<?xml version='1.0' encoding='UTF-8'?>
<ListBucketResult xmlns='http://doc.s3.amazonaws.com/2006-03-01'>
<Name>nd-srv</Name>
<Prefix>pacs-data/archive/2022/8/</Prefix>
<Marker>
</Marker>
<GenerationMarker>
</GenerationMarker>
<NextMarker>pacs-data/archive/2022/8/1/10/11992158/8EC0C0C4/77407295</NextMarker>
<NextGenerationMarker>1683228165974407</NextGenerationMarker>
<MaxKeys>1000</MaxKeys>
<IsTruncated>true</IsTruncated>
<Encoding-Type>url</Encoding-Type>
<Version>
<Key>pacs-data/archive/2022/8/1/10/08E75EBF/F9A5759E/271EE0F6</Key>
<Generation>1683224794134204</Generation>
<MetaGeneration>1</MetaGeneration>
<IsLatest>true</IsLatest>
<LastModified>2023-05-04T18:26:34.136Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"ea549ded855170e0670a00e5d2e13a0a"</ETag>
<Size>54366</Size>
</Version>
...snip lots of similar blocks...
<Version>
<Key>pacs-data/archive/2022/8/1/10/11992158/8EC0C0C4/77407295</Key>
<Generation>1683228165974407</Generation>
<MetaGeneration>1</MetaGeneration>
<IsLatest>true</IsLatest>
<LastModified>2023-05-04T19:22:45.975Z</LastModified>
<DeletedTime>
</DeletedTime>
<ETag>"361b28d02accd86fef5cf92791795699"</ETag>
<Size>243496</Size>
</Version>
</ListBucketResult>
Something strange has gone on here for definite.
According to the AWS docs, this is what the response should look like
<?xml version="1.0" encoding="UTF-8"?>
<ListVersionsResult>
<IsTruncated>boolean</IsTruncated>
<KeyMarker>string</KeyMarker>
<VersionIdMarker>string</VersionIdMarker>
<NextKeyMarker>string</NextKeyMarker>
<NextVersionIdMarker>string</NextVersionIdMarker>
<Version>
<ChecksumAlgorithm>string</ChecksumAlgorithm>
...
<ETag>string</ETag>
<IsLatest>boolean</IsLatest>
<Key>string</Key>
<LastModified>timestamp</LastModified>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
<RestoreStatus>
<IsRestoreInProgress>boolean</IsRestoreInProgress>
<RestoreExpiryDate>timestamp</RestoreExpiryDate>
</RestoreStatus>
<Size>long</Size>
<StorageClass>string</StorageClass>
<VersionId>string</VersionId>
</Version>
...
<DeleteMarker>
<IsLatest>boolean</IsLatest>
<Key>string</Key>
<LastModified>timestamp</LastModified>
<Owner>
<DisplayName>string</DisplayName>
<ID>string</ID>
</Owner>
<VersionId>string</VersionId>
</DeleteMarker>
...
<Name>string</Name>
<Prefix>string</Prefix>
<Delimiter>string</Delimiter>
<MaxKeys>integer</MaxKeys>
<CommonPrefixes>
<Prefix>string</Prefix>
</CommonPrefixes>
...
<EncodingType>string</EncodingType>
</ListVersionsResult>
In particular note that the response should be a ListVersionsResult but gcs has given us a ListBucketResult
This is why rclone is looping because it is expecting to find NextKeyMarker and NextVersionIdMarker but it isn't so it is starting the listing over again each time.
However that doesn't explain why gcs is giving us the wrong listing result.
A bit of searching finds a comment which indicates gcs is being incompatible here. There is some more discussion on the drupal forum about this.
I can't find a defininitive statement in the GCS docs about this, but looking at the docs for the API call it just doesn't look compatible with S3.
Rclone should definitely notice the response is wrong and give an error which would have saved your looping problem.
I have fixed that here:
v1.65.0-beta.7523.15747cc9f.fix-s3-gcs-looping on branch fix-s3-gcs-looping (uploaded in 15-30 mins)
However I think the real problem here is that the GCS S3 interface isn't fully AWS S3 compatible. I couldn't find any google issues about this so it would probably be worth making one if you agree with my diagnosis.
I can't make a workaround for this - the AWS S3 SDK is very inflexible unfortunately.
