Sync from AWS S3 to local hangs

Help and Support
1

What is the problem you are having with rclone?

I'm trying to mirror the Zimbra repositories, for which they provide a public endpoint from S3. Their own documentation have example incantations using the aws CLI, like:

aws s3 sync s3://repo.zimbra.com/apt/8815 ./repo --no-sign-request --delete

And I figured that I could use rclone instead of the AWS client. Listing the remote with:

rclone ls :s3,provider=AWS,endpoint=zimbra.com:repo/apt

seems successful (note that I'm using the on-the-fly backend generation with a connection string.) However, when I try to use the sync command, rclone will hang.

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-43-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

AWS S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync :s3,provider=AWS,endpoint=zimbra.com:repo/apt ./repo

The rclone config contents with secrets removed.

(none since I'm using connection strings)

A log from the command with the -vv flag

2022/08/11 16:27:27 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "sync" ":s3,provider=AWS,endpoint=zimbra.com:repo/apt" "./repo/"]
2022/08/11 16:27:27 DEBUG : Creating backend with remote ":s3,provider=AWS,endpoint=zimbra.com:repo/apt"
2022/08/11 16:27:27 DEBUG : :s3: detected overridden config - adding "{rdsIh}" suffix to name
2022/08/11 16:27:27 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
2022/08/11 16:27:28 DEBUG : fs cache: renaming cache item ":s3,provider=AWS,endpoint=zimbra.com:repo/apt" to be canonical ":s3{rdsIh}:repo/apt"
2022/08/11 16:27:28 DEBUG : Creating backend with remote "./repo/"
2022/08/11 16:27:28 DEBUG : fs cache: renaming cache item "./repo/" to be canonical "/home/user/tmp/zimbra/repo"
2

hello and welcome to the forum,

rclone needs certain basic information, as requested when creating a remote.
so if you want to use connection string, need to supply info such as client id/secret, etc.

aws s3 tools create their own config files, which rclone can use as is.
https://rclone.org/s3/#authentication

and provider=AWS should be provider=Other
https://rclone.org/s3/#s3-provider

3

Thank you!

Indeed. Since this particular source is available for anonymous users, I followed rclone's own documentation for those cases. This is the resulting command with the full connection string:

rclone sync :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,
  access_key_id='',secret_access_key='',region=us-east-1,
  location_constraint='',acl=private,server_side_encryption='',
  storage_class='':repo/apt ./repo

(line broken for readability)

And still Iḿ faced with the same issue: a simple ls will properly list all the remote files but a sync will stay there, hung.

Honestly, I did not want to install the AWS tools, but thanks to your comment I will; and continue debugging this.

4

and that debug log is the complete debug log, rclone does not append anything after that?

5

this worked

+ rclone copy -vv /home/user01/zork :s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork
DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "-vv" "/home/user01/zork" ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork"]
DEBUG : Creating backend with remote "/home/user01/zork"
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : Creating backend with remote ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork"
DEBUG : :s3: detected overridden config - adding "{NylMo}" suffix to name
DEBUG : fs cache: renaming cache item ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork" to be canonical ":s3{NylMo}:zork"
DEBUG : S3 bucket zork: Waiting for checks to finish
DEBUG : S3 bucket zork: Waiting for transfers to finish
DEBUG : zork1.dat: md5 = b732a93a6244ddd92a9b9a3e3a46c687 OK
INFO  : zork1.dat: Copied (new)
6

That seems to work fine for me. I am not sure the hang is what you think as I think it's just slow.

rclone ls works fine for me and returns stuff:

rclone ls :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,access_key_id='',secret_access_key='',region=us-east-1,location_constraint='',acl=private,server_side_encryption='',storage_class='':repo/apt

anything else takes forever as I'm not familiar with whatever that is.

 rclone ls :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,access_key_id='',secret_access_key='',region=us-east-1,location_constraint='',acl=private,server_side_encryption='',storage_class='':repo/apt
      734 8.8.15.p32-ne/conf/distributions
       15 8.8.15.p32-ne/conf/options
   499712 8.8.15.p32-ne/db/checksums.db
    16384 8.8.15.p32-ne/db/contents.cache.db
   184320 8.8.15.p32-ne/db/packagenames.db
  2121728 8.8.15.p32-ne/db/packages.db
   253952 8.8.15.p32-ne/db/references.db
    69632 8.8.15.p32-ne/db/release.caches.db
       31 8.8.15.p32-ne/db/version
     2471 8.8.15.p32-ne/dists/bionic/InRelease
     1603 8.8.15.p32-ne/dists/bionic/Release
      819 8.8.15.p32-ne/dists/bionic/Release.gpg

that goes on and on..

7

maybe try adding --fast-list.
and if you want a deeper look, add --dump=headers

8

[ Note add --use-server-modtime and it will run a lot quicker as it isn't querying the objects for rclone metadata which isn't there. ]

A bit of testing reveals it has got stuck in a directory loop listing the same directory over and over again. This is likely a bug in the server or rclone...

This is what we send the server (see with -vv --dump bodies)

2022/08/12 10:49:20 DEBUG : HTTP REQUEST (req 0xc000269500)
2022/08/12 10:49:20 DEBUG : GET /?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=apt%2F HTTP/1.1
Host: repo.zimbra.com
User-Agent: rclone/v1.60.0-beta.6419.b3b520090
Accept-Encoding: gzip

And this is what it replies with

2022/08/12 10:49:20 DEBUG : HTTP RESPONSE (req 0xc000269500)
2022/08/12 10:49:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Age: 12534
Connection: keep-alive
Content-Type: application/xml
Date: Fri, 12 Aug 2022 06:20:26 GMT
Server: AmazonS3
Via: 1.1 7a71153df5fe7b23e438dedb00b3bf4a.cloudfront.net (CloudFront)
X-Amz-Bucket-Region: us-east-1
X-Amz-Cf-Id: rHN4u7P7CbKwMy11UTeIVc4c4H0TFSE4ygWi6WCMmeSdQtiOJqc1HA==
X-Amz-Cf-Pop: LHR62-C3
X-Cache: Hit from cloudfront

Plus an XML body which I'd shortened and tidied here

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>repo.zimbra.com</Name>
  <Prefix>
  </Prefix>
  <Marker>
  </Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>true</IsTruncated>
  <Contents>
    <Key>apt/</Key>
    <LastModified>2017-12-04T07:50:20.000Z</LastModified>
    <ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag>
    <Size>0</Size>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>apt/8.8.15.p32-ne/conf/distributions</Key>
    <LastModified>2022-07-27T07:45:36.000Z</LastModified>
    <ETag>&quot;33e0d3d724a1706df1763fb599d40aba&quot;</ETag>
    <Size>734</Size>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1852.1571227624-1.u16_amd64.deb</Key>
    <LastModified>2022-07-27T07:51:03.000Z</LastModified>
    <ETag>&quot;db28a29641a56c261ac14b8397dac2ad-10&quot;</ETag>
    <Size>80033616</Size>
    <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1852.1571227624-1.u18_amd64.deb</Key>
  <LastModified>2022-07-27T07:51:04.000Z</LastModified>
  <ETag>&quot;dea9ae821ad6f3d811e017a2559f1f6f-10&quot;</ETag>
  <Size>80049588</Size>
  <StorageClass>STANDARD</StorageClass>
  </Contents>
  <Contents>
    <Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1856.1572932785-1.u14_amd64.deb</Key>
    <LastModified>2022-07-27T07:51:05.000Z</LastModified>
    <ETag>&quot;0a8c6443c03a392b81e767922850289c-10&quot;</ETag>
  <Size>79984402</Size>
  <StorageClass>STANDARD</StorageClass>
  </Contents>
</ListBucketResult>

Notice that this returns <IsTruncated>true</IsTruncated> so we are expecting more entries to list, but it doesn't return <NextContinuationToken>string</NextContinuationToken> so rclone just starts listing the directory again which is not too clever but explains the loop.

So the results look out of spec for s3 responses. It might be that rclone is supposed to be doing something different...

So there is definitely something whacky going on there! I don't think it is rclone's fault, but I'm not 100% sure!

Adding --disable ListR to rclone to stop recursive listings doesn't seem to stop the server sending them.

So I think the server is broken in some way.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.