When I executed copy, move with --s3-versions, it seems not to start copy or move for 110,000 files.
I guess the problem happens when I use the --s3-version options.
I hope the transfering and finding are executing concurrently.
Run the command 'rclone version' and share the full output of the command.
1.63.0
Which cloud storage system are you using? (eg Google Drive)
custom s3 storage ( e.g. ceph s3 )
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone copy --s3-versions remote_s3:tmp other_s3:tmp
remote_s3:tmp has a lot of files ( e.g. 100,000 ea including versioned old files )
@kapitainsky Thank you for your response.
As you said it depands on network to increase the transfering speed.
My case is different. I was waiting for almost 30 min to start copy action for 100,000 versioned small files(total 2GB). but I didn't see the start.
I mean that I was just watching Totaltransfers=0 of stats for long time.
I concerned that --s3-versions option impact to the problem. but you look like it's not.
And I try to apply --fast-list. when I use --fast-list with --s3-version, some of files aren't transfered.
Which options do I consider to solve the problem?
There are 6000 files without --s3-versions, it started well without delay. I don't know how much it takes for 100,000 files without --s3-versions.
Do you think it's possible that --s3-versions option impacts the start?
Does --s3-versions impact this problem? If I use --s3-list-version int, Is it possible to be different?
I guess that the number of files impacts to delay for start even without --s3-versions
e.g.
A bucket has 100,000 files with the versions.
B bucket has 100,000 files without the versions.
If I try to transfer to C bucket, My guessing that A -> C and B -> C take same transferring time.
Thank you for your quick response.
I will let you know how many files in a directory.
but I just know s3 doesn't have directory hierarchy.
Is it important how many files in a directory even though s3 object storage?
I thought that --checkers option could improve the finding speed.
but it's not. When I increase --checkers to 32, it seems not to impact at all.
what unit does a checker check files? directory?
And If I execute rclone copy --s3-verisons object_1:bucket object_2:bucket,
it seems to apply --s3-version option to both object_1(source) and object_2(target).
Could I apply --s3-versions to only object_1(source) ?