Rclone copy dry-run of big S3 bucket is taking super long time to start

[Martin_Sandell]

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I'm trying to start a dryrun on a copy for a big S3 bucket (44TB and 44,5 million objects) and the command I ran stood in index/pre-copy for 58h's before I aborted it

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

• os/version: redhat 7.9 (64 bit)
• os/kernel: 3.10.0-1160.90.1.el7.x86_64 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.19.4
• go/linking: static
• go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3 compliant...
From Hitachi HCP
To Scality ring

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy hcp:bucket1/RES/tblIMAGE/ scality:spring-bucket/RES/tblIMAGE/ --s3-versions --b2-versions --progress --create-empty-src-dirs --log-file=tblIMAGE_migration_2023.06.12.14.00.txt --dry-run --fast-list --progress-terminal-title --verbose

The rclone config contents with secrets removed.

[wbthcp]
type = s3
provider = Other
access_key_id = xxx
secret_access_key = yyy
endpoint = https://my-s3-endpoing
location_constraint = us-west-1
acl = private

[scality]
type = s3
provider = AWS
access_key_id = xxx
secret_access_key = yyy
region = us-west-1
endpoint = my-s3-endpoing
location_constraint = us-west-1
acl = private

A log from the command with the -vv flag

no output, only takes long time

[kapitainsky]

1. update rclone to the latest version
2. remove --fast-list

You should see progress immediately.

BTW why you use: --b2-versions on S3 remotes?

[Martin_Sandell]

Hello.
Upgrade is a good suggestion.
When comes to -fast-list, it's the second attempt but it seems to have made no difference with or without as to this issue.
It was +24h's when I aborted both with and without.

The --b2-versions is what my colleague used during his first migrations, so I used his options and built on that.
Both have description "Include old versions in directory listings" so might be useless with the b2 here...

[kapitainsky]

Definitely the size of this repo (44TB and 44,5 million objects) makes things slower - especially number of files.. Still rclone should handle it assuming you have enough RAM.

One more thing to try is to use --checksum flag to avoid metadata reads for every file.

There is also workaround:

List all files to file - it also will take time but you should see data immediately (tail -f file.txt)

rclone lsf --files-only hcp:bucket1/RES/tblIMAGE > file.txt

then you can split it into many more manageable chunks (split command on Linux) and copy one by one:

rclone copy --checksum --files-from file01.txt hcp:bucket1/RES/tblIMAGE scality:spring-bucket/RES/tblIMAGE/

[kapitainsky]

For test you can also try to run:

rclone lsf --files-only hcp:bucket1/RES/tblIMAGE -vv --dump bodies

to see if rclone is "talking" to this S3 remote - because maybe problem is on S3 side? and it takes ages to respond?

[kapitainsky]

And similar issue you maybe find something useful there:

Long delay when copying millions of tiny objects

Hi all, I am seeing a long idle time between submitting the copy command and start of the actual transfer. Source and Target are two different S3-compatible Cloud Object Storages. There are millions of tiny (few KB) keys with the specified source prefix. The destination bucket also contains millions of objects, but none share the prefix that I am trying to copy. Delay before rclone uploads files, a newbie question looks similar, but my objects are tiny and calculation for the downloaded keys s…

[Martin_Sandell]

Ah thanks for good input!
I'll try those things then.

[Martin_Sandell]

Seems to be no issue talking to the backend/remote.
bodies were huge... but that's maybe how it always is.

But think I'm going to try a dry-run with --checksum

Here is an example with the command (headers, not bodies, but that worked as welll)

2023/06/15 13:13:04 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/06/15 13:13:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/06/15 13:13:05 DEBUG : HTTP RESPONSE (req 0xc00026a900)
2023/06/15 13:13:05 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Thu, 15 Jun 2023 11:13:04 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/06/15 13:13:05 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/06/15 13:13:05 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/06/15 13:13:05 DEBUG : HTTP REQUEST (req 0xc000605200)
2023/06/15 13:13:05 DEBUG : GET /bucket1yyyxxxwww HTTP/1.1
Host: s3-endpoint-address
User-Agent: rclone/v1.61.1
Authorization: XXXX
X-Amz-Content-Sha256: xxx
X-Amz-Date: 20230615T111305Z
Accept-Encoding: gzip

[kapitainsky]

I think one HTTP response contains 1000 objects.. so for your repo it would be 44k such responses to list all. But maybe you can estimate how long it takes for one - it should give you rought time how long it will take to finish

[Martin_Sandell]

I get about 3-4 requests per second with lsf so should be 44.000 / 3 / 60 (minutes) / 60 (hours) = about 4 hours...
But math isn't my strongest subject

[Martin_Sandell]

Well... it's not going great...
rclone copy hcp:bucket1/RES/tblIMAGE/ scality:spring-bucket/RES/tblIMAGE/ -vv --checksum --s3-versions --progress --create-empty-src-dirs --log-file=tblIMAGE_migration_2023.06.15.13.20.txt --dry-run
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 114h37m13.5s

And it give not output at all, so hard to say what it's actually doing.

[Martin_Sandell]

Still not having ANY success regardless of the options I use or if I use an input list or not...
Seems it's still building an index even when I user input list from file...

Anyone who has any insight on how to get it to do large namespace migrations?!
All help is wanted

[kapitainsky]

Have a look at this thread - maybe it applies to your situation and you can try provided fixed version

Almost 24 hours CPU compute time during sync between two large S3 buckets

Perfect for next release, thanks. I work for a company indeed, I'll touch a word to my team about that, I'll be glad to support the project.

[Martin_Sandell]

Oh wow !
That actually looks hand in glove on paper!
Will be on my list of tvings to test tomorrow, for sure!!

Thanks a bunch!

[ncw (Nick Craig-Wood)]

That fix will help with --fast-list if you have a large number (100,000 or more) of directories within a directory and the patch is scheduled for 1.63.1.

@Martin_Sandell

Still not having ANY success regardless of the options I use or if I use an input list or not...
Seems it's still building an index even when I user input list from file...

One possibility is rclone is looping on the directory listing.

We've seen this before when rclone is configured to use ListObjectsv2 but the server only supports v1. Newer rclone should detect this.

If it gets stuck can you try again but add -vv --dump headers and then post the last 10 HTTP transactions?

[Martin_Sandell]

Absolutely. I’ll apply the parched version and then past the dumped headers.

[Martin_Sandell]

Updated now to 1.63.0 but still having the same kind of result...

"Smaller" sub dir is fine
$ time rclone lsf --files-only hcp:bucket1/dir/subdir/ | wc -l
64883

real 0m15.475s
user 0m2.563s
sys 0m0.210s

The large one goes on for very long time.

How should I dump headers for best result for you to look at?
Because if it's the end of the listing, I don't think I can do that.

But here are 10 REQUEST/RESPONSE (Which I think was what you were asking for)

rclone lsf --fast-list --files-only hcp:bucket1/dir/tsubdir/ -vv --dump headers
2023/07/05 09:50:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:24 DEBUG : HTTP REQUEST (req 0xc0008f7400)
2023/07/05 09:50:24 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13764309-preview&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075024Z
Accept-Encoding: gzip

2023/07/05 09:50:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:25 DEBUG : HTTP RESPONSE (req 0xc0008f7400)
2023/07/05 09:50:25 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:24 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:25 DEBUG : HTTP REQUEST (req 0xc0008f7c00)
2023/07/05 09:50:25 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13765299-thumb&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075025Z
Accept-Encoding: gzip

2023/07/05 09:50:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:25 DEBUG : HTTP RESPONSE (req 0xc0008f7c00)
2023/07/05 09:50:25 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:25 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:25 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:25 DEBUG : HTTP REQUEST (req 0xc000247600)
2023/07/05 09:50:25 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13766216&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075025Z
Accept-Encoding: gzip

2023/07/05 09:50:25 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : HTTP RESPONSE (req 0xc000247600)
2023/07/05 09:50:26 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:25 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : HTTP REQUEST (req 0xc00738be00)
2023/07/05 09:50:26 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13767320-preview&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075026Z
Accept-Encoding: gzip

2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : HTTP RESPONSE (req 0xc00738be00)
2023/07/05 09:50:26 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:26 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : HTTP REQUEST (req 0xc0075aa600)
2023/07/05 09:50:26 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13768375-thumb&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075026Z
Accept-Encoding: gzip

2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : HTTP RESPONSE (req 0xc0075aa600)
2023/07/05 09:50:26 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:26 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:26 DEBUG : HTTP REQUEST (req 0xc0075aab00)
2023/07/05 09:50:26 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13770445&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075026Z
Accept-Encoding: gzip

2023/07/05 09:50:26 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:27 DEBUG : HTTP RESPONSE (req 0xc0075aab00)
2023/07/05 09:50:27 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:26 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:27 DEBUG : HTTP REQUEST (req 0xc0075ab200)
2023/07/05 09:50:27 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13771778-preview&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075027Z
Accept-Encoding: gzip

2023/07/05 09:50:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:27 DEBUG : HTTP RESPONSE (req 0xc0075ab200)
2023/07/05 09:50:27 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:27 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:27 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:27 DEBUG : HTTP REQUEST (req 0xc0095aa600)
2023/07/05 09:50:27 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13773421-thumb&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075027Z
Accept-Encoding: gzip

2023/07/05 09:50:27 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:28 DEBUG : HTTP RESPONSE (req 0xc0095aa600)
2023/07/05 09:50:28 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:27 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:28 DEBUG : HTTP REQUEST (req 0xc00040be00)
2023/07/05 09:50:28 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13776169&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075028Z
Accept-Encoding: gzip

2023/07/05 09:50:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:28 DEBUG : HTTP RESPONSE (req 0xc00040be00)
2023/07/05 09:50:28 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:28 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:28 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:28 DEBUG : HTTP REQUEST (req 0xc000837500)
2023/07/05 09:50:28 DEBUG : GET /bucket1?delimiter=%2F&marker=RES%2FtblIMAGE%2F13777746-preview&max-keys=1000&prefix=RES%2FtblIMAGE%2F HTTP/1.1
Host: my.s3.endpoint.com
User-Agent: rclone/v1.63.0
Authorization: XXXX
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230705T075028Z
Accept-Encoding: gzip

2023/07/05 09:50:28 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2023/07/05 09:50:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2023/07/05 09:50:29 DEBUG : HTTP RESPONSE (req 0xc000837500)
2023/07/05 09:50:29 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: no-cache,no-store,must-revalidate
Content-Security-Policy: default-src 'self'; script-src 'self' 'unsafe-eval' 'unsafe-inline'; connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline'; object-src 'self'; frame-ancestors 'self';
Content-Type: application/xml;charset=utf-8
Date: Wed, 05 Jul 2023 07:50:28 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Strict-Transport-Security: max-age=31536000; includeSubDomains
Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
Vary: Accept-Encoding, User-Agent
X-Content-Type-Options: nosniff
X-Dns-Prefetch-Control: off
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

2023/07/05 09:50:29 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
type or paste code here

[Martin_Sandell]

Oh! But this is interesting....
I'm looking in real-time now as I'm dumping headers.

In the example above I have 10 Requests in 5 sec, when I now took a sample (twice), because I thought it looked slower, I have 7 requests in 10 sec and 8 requests in 11 sec.
That is 65% slower in just a few minutes!

I looked at the VM running this job and it's not CPU/memory of the machine that's the bottleneck at least... it just slows down, quite a lot!

Then I understand why this is taking forever. 44 million second is what, 509 days something!

Please help me speed this up, significantly!

[kapitainsky]

@Martin_Sandell

Updated now to 1.63.0 but still having the same kind of result...

The fix from other thread is only in the the latest beta - so it should be 1.64

sudo -v ; curl https://rclone.org/install.sh | sudo bash -s beta

[Martin_Sandell]

Any way to get that as an rpm?
We've in a DMZ and aren't allowed to connect servers to the internet for stuff like this.