Too slow to start copy or move with --s3-versions

Help and Support
1

What is the problem you are having with rclone?

When I executed copy, move with --s3-versions, it seems not to start copy or move for 110,000 files.
I guess the problem happens when I use the --s3-version options.
I hope the transfering and finding are executing concurrently.

Run the command 'rclone version' and share the full output of the command.

1.63.0

Which cloud storage system are you using? (eg Google Drive)

custom s3 storage ( e.g. ceph s3 )

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --s3-versions remote_s3:tmp other_s3:tmp
remote_s3:tmp has a lot of files ( e.g. 100,000 ea including versioned old files )

The rclone config contents with secrets removed.

N/A

A log from the command with the -vv flag

N/A

2

And it is "too slow" compared to what? Depending on number of files and directories it can take anything from 1s to many hours before transfers start.

Then there is network speed/latency and server performance to add to the equation.

This is good example to illustrate possible differences:

3

@kapitainsky Thank you for your response.
As you said it depands on network to increase the transfering speed.
My case is different. I was waiting for almost 30 min to start copy action for 100,000 versioned small files(total 2GB). but I didn't see the start.
I mean that I was just watching Totaltransfers=0 of stats for long time.
I concerned that --s3-versions option impact to the problem. but you look like it's not.
And I try to apply --fast-list. when I use --fast-list with --s3-version, some of files aren't transfered.
Which options do I consider to solve the problem?

4

Good point.

Could you compare how long it takes without --s3-versions flag? and how many files is involved in such case?

5

There are 6000 files without --s3-versions, it started well without delay. I don't know how much it takes for 100,000 files without --s3-versions.
Do you think it's possible that --s3-versions option impacts the start?

6

definitely yes - but I have no data to quantify it

7

You left the hints for large directories case.

  1. add --checksum flag to avoid reading modtime
  2. try to increase --checkers value
  3. add --fast-list to buffer all the objects in memory first (it will use about 0.5GB of RAM for 500k objects)
  4. try --s3-list-version 2 flag

is it acceptable for my case?

8

@ncw I saw the answers that you left about --s3-versions or s3 options. could you help me about this problem?

9

--checksum or --size-only should work with versions to speed up the transfer start. I'd try one of those first.

10

Does --s3-versions impact this problem? If I use --s3-list-version int, Is it possible to be different?
I guess that the number of files impacts to delay for start even without --s3-versions
e.g.
A bucket has 100,000 files with the versions.
B bucket has 100,000 files without the versions.
If I try to transfer to C bucket, My guessing that A -> C and B -> C take same transferring time.

11

It makes more files, so yes.

If it is working as it is, then don't change --s3-list-versions - it won't make it quicker.

Yes that is right.

The time to start is how many files you have in one directory. Are all your files in one directory?

12

Thank you for your quick response.
I will let you know how many files in a directory.
but I just know s3 doesn't have directory hierarchy.
Is it important how many files in a directory even though s3 object storage?

I thought that --checkers option could improve the finding speed.
but it's not. When I increase --checkers to 32, it seems not to impact at all.
what unit does a checker check files? directory?

And If I execute rclone copy --s3-verisons object_1:bucket object_2:bucket,
it seems to apply --s3-version option to both object_1(source) and object_2(target).
Could I apply --s3-versions to only object_1(source) ?

13

Yes. Even though s3 doesn't have directories, rclone lists the objects directory by directory.

Checkers increases concurrency. But if you have one very big directory it won't help.

Like this

rclone copy object_1,versions:bucket object_2:bucket
1 Like
14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.