Slow transfer big amount of slow objects - S3, no cloud provides

Help and Support
1

Hi,

I'm trying to copy more than a billion small files from an S3 compliant object storage to an another one. The total size on the files are arround 150TB. My idea is to copy each file one by one because the whole process must be able to follow. I splitted the list of the objects from the original bucket into a txt files which contains 1million object attributes (name, size, modification date) from these files I have 1194 pieces. I was writing a script in PowerShell (6.2.0) which reads this file and starts the rclone copy processes in a separate thread. In a sequential mode I can reach 2 objects/second speed in a paralell mode i can reach max. 20-38 objects/seconds

S3compliantStorage1: Cloudian
S3compliantStorage2: Pure Storage FlashBlade

rclone v1.59.2
- os/version: Microsoft Windows Server 2019 Standard 1809 (64 bit)
- os/kernel: 10.0.17763.2803 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: cmount

Command:
rclone.exe --no-check-certificate --log-file=rclonelog.$threadID.txt --log-level DEBUG --checksum copy S3compliantStorage1:bucket1/$objname S3compliantStorage1:bucket

rclone.conf:
[S3compliantStorage1]
type = s3
provider = Other
access_key_id = ***
endpoint = ***
secret_access_key = ***
disable_checksum = false
v2_auth = false
disable_http2 = false
acl = private
bucket_acl = private

[S3compliantStorage2]
type = s3
provider = Other
access_key_id = ***
secret_access_key = ***
endpoint = ***
acl = private
bucket_acl = private
disable_checksum = false

the machine has a separate 10Gbits connection to the storages and they have a 2 hop distance

I made some tests:

Count of object Paralell sessions Throttling (Gbit/s) Speed (objects/sec)
300 90 1,074 28,5
10000 600 1,074 24
4000 600 5,369 33,29
4000 600 5,369 30,06
1000 600 0 23,51
1000 600 0 35,91

It seems to that the number of paralell sessions has no effect on the speed (object/second)
The speed calculation it's easy. This is the difference of the datum of starting create threads and the last finished thread and I'm dividing with the count of objects.

The interesting this it is if I try to download a big file to a RAMdisk into the Windows machine from the same source storage from a different bucket, than I can reach 1.2Gbits througput. If I try to upload this object from RAMdisk to the target object storage, I can reach 2.6Gbits. If I try to copy direct from S3compliantStorage1 to S3compliantStorage2 than I can reach only 300 Mbits speed.

Do you have any idea how can I speed up the copy procedure? I would like to reach min 300 objects/sec speed. The objects has 1-2KB size.

Best regards
Rednek1987

2

Hi Rednek1987,

I guess you will see a speed improvement if using --no-traverse, especially as there are getting more and more folders and files in the target. More info here: https://rclone.org/docs/#no-traverse

You may get even better results with --no-check-dest, but it comes with a higher risk, so it is only to be used if significantly better than --no-traverse and the downsides are acceptable, so please test and read the documentation carefully here: https://rclone.org/docs/#no-check-dest

You may also see a speed improvement by removing --checksum, more information here: https://rclone.org/docs/#c-checksum

If the above isn't enough, then we can probably reduce the startup overhead by starting an rclone deamon with rclone rcd and then sending asynchronous copy commands using rclone rc sync/copy. I will defer the details until we know if it is needed.

Is the copy from S3compliantStorage1 to S3compliantStorage2 comparable, that is the same big file?

What are the object per second transfers speeds if you transfers a lot of small files in each of the three tests?
(Tip: You can use --ignore-times to make the commands repeatable)

Does the speed change if you add --s3-disable-http2 to workaround some known issues?
(More info here: https://rclone.org/s3/#s3-disable-http2)

3

Hi Ole,

I tried your suggestions but I not achieved increased speed.

"Is the copy from S3compliantStorage1 to S3compliantStorage2 comparable, that is the same big file?"
Yes, I downloaded the same file and tried to upload it to another storage.

4

OK, this sounds strange.

I would like to make a little very controlled test which is possible to reproduce with additional debug logging, if needed.

First, I would like you to prepare a small test sandbox like this:

# Create an empty test folder with a copy of the big test file in S3compliantStorage1
rclone mkdir S3compliantStorage1:testbucket/testfolder1
rclone copyto S3compliantStorage1:yourBucket/yourFolder/yourBigFile S3compliantStorage1:testbucket/testfolder1/bigtestfile
# Create empty test folders on the RAM Disk (I call it R:) and S3compliantStorage2
rclone mkdir R:\testfolder1
rclone mkdir S3compliantStorage2:testbucket/testfolder1

Note: You may need to add --no-check-certificate to some of the commands.

Next, I would like to see the full output from executing these three commands (including the commands):

rclone copy --ignore-times -v S3compliantStorage1:testbucket/testfolder1 R:\testfolder1
rclone copy --ignore-times -v R:\testfolder1 S3compliantStorage2:testbucket/testfolder1
rclone copy --ignore-times -v S3compliantStorage1:testbucket/testfolder1  S3compliantStorage2:testbucket/testfolder1

I suggest you try executing the sequence twice to verify each of the commands have a (somewhat) stable speed.

5

Hi Ole,

Because of deadline I had to see an another solution. I moved to a linux (installed a Linux subsystem on that widonws) and it can grants the expected speed 250-500 objects/second.
I think it can be some limitation of the number of paralell TCP connection on windows level.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.