PentiumBug
(Carlos Alberto Carnero Delgado)
August 11, 2022, 8:30pm
1
What is the problem you are having with rclone?
I'm trying to mirror the Zimbra repositories, for which they provide a public endpoint from S3 . Their own documentation have example incantations using the aws
CLI, like:
aws s3 sync s3://repo.zimbra.com/apt/8815 ./repo --no-sign-request --delete
And I figured that I could use rclone instead of the AWS client. Listing the remote with:
rclone ls :s3,provider=AWS,endpoint=zimbra.com:repo/apt
seems successful (note that I'm using the on-the-fly backend generation with a connection string.) However, when I try to use the sync
command, rclone will hang.
Run the command 'rclone version' and share the full output of the command.
rclone v1.59.1
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-43-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.5
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
AWS S3
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone sync :s3,provider=AWS,endpoint=zimbra.com:repo/apt ./repo
The rclone config contents with secrets removed.
(none since I'm using connection strings)
A log from the command with the -vv
flag
2022/08/11 16:27:27 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "-vv" "sync" ":s3,provider=AWS,endpoint=zimbra.com:repo/apt" "./repo/"]
2022/08/11 16:27:27 DEBUG : Creating backend with remote ":s3,provider=AWS,endpoint=zimbra.com:repo/apt"
2022/08/11 16:27:27 DEBUG : :s3: detected overridden config - adding "{rdsIh}" suffix to name
2022/08/11 16:27:27 NOTICE: Config file "/home/user/.config/rclone/rclone.conf" not found - using defaults
2022/08/11 16:27:28 DEBUG : fs cache: renaming cache item ":s3,provider=AWS,endpoint=zimbra.com:repo/apt" to be canonical ":s3{rdsIh}:repo/apt"
2022/08/11 16:27:28 DEBUG : Creating backend with remote "./repo/"
2022/08/11 16:27:28 DEBUG : fs cache: renaming cache item "./repo/" to be canonical "/home/user/tmp/zimbra/repo"
asdffdsa
(jojothehumanmonkey)
August 11, 2022, 9:00pm
2
hello and welcome to the forum,
rclone needs certain basic information, as requested when creating a remote.
so if you want to use connection string, need to supply info such as client id/secret, etc.
aws s3 tools create their own config files, which rclone can use as is.
https://rclone.org/s3/#authentication
and provider=AWS
should be provider=Other
https://rclone.org/s3/#s3-provider
PentiumBug
(Carlos Alberto Carnero Delgado)
August 12, 2022, 12:01am
3
Thank you!
rclone needs certain basic information, as requested when creating a remote.
so if you want to use connection string, need to supply info such as client id/secret, etc.
Indeed. Since this particular source is available for anonymous users, I followed rclone's own documentation for those cases . This is the resulting command with the full connection string:
rclone sync :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,
access_key_id='',secret_access_key='',region=us-east-1,
location_constraint='',acl=private,server_side_encryption='',
storage_class='':repo/apt ./repo
(line broken for readability)
And still Iḿ faced with the same issue: a simple ls
will properly list all the remote files but a sync
will stay there, hung.
Honestly, I did not want to install the AWS tools, but thanks to your comment I will; and continue debugging this.
asdffdsa
(jojothehumanmonkey)
August 12, 2022, 12:13am
4
and that debug log is the complete debug log, rclone does not append anything after that?
asdffdsa
(jojothehumanmonkey)
August 12, 2022, 12:40am
5
this worked
+ rclone copy -vv /home/user01/zork :s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork
DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "copy" "-vv" "/home/user01/zork" ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork"]
DEBUG : Creating backend with remote "/home/user01/zork"
DEBUG : Using config file from "/home/user01/.config/rclone/rclone.conf"
DEBUG : Creating backend with remote ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork"
DEBUG : :s3: detected overridden config - adding "{NylMo}" suffix to name
DEBUG : fs cache: renaming cache item ":s3,provider=Wasabi,access_key_id=redacted,secret_access_key=redacted,endpoint=s3.us-east-2.wasabisys.com:zork" to be canonical ":s3{NylMo}:zork"
DEBUG : S3 bucket zork: Waiting for checks to finish
DEBUG : S3 bucket zork: Waiting for transfers to finish
DEBUG : zork1.dat: md5 = b732a93a6244ddd92a9b9a3e3a46c687 OK
INFO : zork1.dat: Copied (new)
That seems to work fine for me. I am not sure the hang is what you think as I think it's just slow.
rclone ls works fine for me and returns stuff:
rclone ls :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,access_key_id='',secret_access_key='',region=us-east-1,location_constraint='',acl=private,server_side_encryption='',storage_class='':repo/apt
anything else takes forever as I'm not familiar with whatever that is.
rclone ls :s3,provider=AWS,endpoint=zimbra.com,env_auth=false,access_key_id='',secret_access_key='',region=us-east-1,location_constraint='',acl=private,server_side_encryption='',storage_class='':repo/apt
734 8.8.15.p32-ne/conf/distributions
15 8.8.15.p32-ne/conf/options
499712 8.8.15.p32-ne/db/checksums.db
16384 8.8.15.p32-ne/db/contents.cache.db
184320 8.8.15.p32-ne/db/packagenames.db
2121728 8.8.15.p32-ne/db/packages.db
253952 8.8.15.p32-ne/db/references.db
69632 8.8.15.p32-ne/db/release.caches.db
31 8.8.15.p32-ne/db/version
2471 8.8.15.p32-ne/dists/bionic/InRelease
1603 8.8.15.p32-ne/dists/bionic/Release
819 8.8.15.p32-ne/dists/bionic/Release.gpg
that goes on and on..
asdffdsa
(jojothehumanmonkey)
August 12, 2022, 12:48am
7
maybe try adding --fast-list
.
and if you want a deeper look, add --dump=headers
ncw
(Nick Craig-Wood)
August 12, 2022, 10:26am
8
[ Note add --use-server-modtime
and it will run a lot quicker as it isn't querying the objects for rclone metadata which isn't there. ]
A bit of testing reveals it has got stuck in a directory loop listing the same directory over and over again. This is likely a bug in the server or rclone...
This is what we send the server (see with -vv --dump bodies
)
2022/08/12 10:49:20 DEBUG : HTTP REQUEST (req 0xc000269500)
2022/08/12 10:49:20 DEBUG : GET /?delimiter=&encoding-type=url&list-type=2&max-keys=1000&prefix=apt%2F HTTP/1.1
Host: repo.zimbra.com
User-Agent: rclone/v1.60.0-beta.6419.b3b520090
Accept-Encoding: gzip
And this is what it replies with
2022/08/12 10:49:20 DEBUG : HTTP RESPONSE (req 0xc000269500)
2022/08/12 10:49:21 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Age: 12534
Connection: keep-alive
Content-Type: application/xml
Date: Fri, 12 Aug 2022 06:20:26 GMT
Server: AmazonS3
Via: 1.1 7a71153df5fe7b23e438dedb00b3bf4a.cloudfront.net (CloudFront)
X-Amz-Bucket-Region: us-east-1
X-Amz-Cf-Id: rHN4u7P7CbKwMy11UTeIVc4c4H0TFSE4ygWi6WCMmeSdQtiOJqc1HA==
X-Amz-Cf-Pop: LHR62-C3
X-Cache: Hit from cloudfront
Plus an XML body which I'd shortened and tidied here
<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Name>repo.zimbra.com</Name>
<Prefix>
</Prefix>
<Marker>
</Marker>
<MaxKeys>1000</MaxKeys>
<IsTruncated>true</IsTruncated>
<Contents>
<Key>apt/</Key>
<LastModified>2017-12-04T07:50:20.000Z</LastModified>
<ETag>"d41d8cd98f00b204e9800998ecf8427e"</ETag>
<Size>0</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>apt/8.8.15.p32-ne/conf/distributions</Key>
<LastModified>2022-07-27T07:45:36.000Z</LastModified>
<ETag>"33e0d3d724a1706df1763fb599d40aba"</ETag>
<Size>734</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1852.1571227624-1.u16_amd64.deb</Key>
<LastModified>2022-07-27T07:51:03.000Z</LastModified>
<ETag>"db28a29641a56c261ac14b8397dac2ad-10"</ETag>
<Size>80033616</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1852.1571227624-1.u18_amd64.deb</Key>
<LastModified>2022-07-27T07:51:04.000Z</LastModified>
<ETag>"dea9ae821ad6f3d811e017a2559f1f6f-10"</ETag>
<Size>80049588</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
<Contents>
<Key>apt/8.8.15.p32-ne/pool/zimbra/z/zimbra-zco/zimbra-zco_8.8.15.1856.1572932785-1.u14_amd64.deb</Key>
<LastModified>2022-07-27T07:51:05.000Z</LastModified>
<ETag>"0a8c6443c03a392b81e767922850289c-10"</ETag>
<Size>79984402</Size>
<StorageClass>STANDARD</StorageClass>
</Contents>
</ListBucketResult>
Notice that this returns <IsTruncated>true</IsTruncated>
so we are expecting more entries to list, but it doesn't return <NextContinuationToken>string</NextContinuationToken>
so rclone just starts listing the directory again which is not too clever but explains the loop.
So the results look out of spec for s3 responses. It might be that rclone is supposed to be doing something different...
So there is definitely something whacky going on there! I don't think it is rclone's fault, but I'm not 100% sure!
Adding --disable ListR
to rclone to stop recursive listings doesn't seem to stop the server sending them.
So I think the server is broken in some way.
system
(system)
Closed
September 11, 2022, 10:26am
9
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.