you asked "how do I have an estimate of time needed" and i think i answered that.
rclone calculates checksums for every file copied.
so your performance is much more then just a theoretical max network speed.
what is your cpu, how many cores, how many threads can it handle. how much ram do you have?
is the server dedicated just to rclone, are there other users,
you have a lot of data to move.
are the s3 files in use, are you able to saturate your connection without interfering?
i was suggesting what i do.
i often install rclone for different customers with different systems.
so i do about 10 test runs, tweaking the parameters and then i know for sure.
i would tweak --transfers and --checkers and add --progress to see the bandwidth used.
-s3-upload-concurrency should be --s3-upload-concurrency
many uses have copied larger data sets than you.
i have done a couple of 30+TB transfers.
rclone limits are dependent on the flags you use and the cpu and ram.
perhaps your computer cannot handle your settings or can handle much more.
you can get an estimate by reading this https://rclone.org/s3/#multipart-uploads
" Multipart uploads will use --transfers * --s3-upload-concurrency * --s3-chunk-size extra memory. Single part uploads to not use extra memory."
I'd recommend using the --checksum flag - this will save transactions for an S3 to S3 copy.
What the keys in the buckets look like is important. If the keys have / in them so simulating a directory heirachy rclone can copy them by loading each "directory" at once.
However if there is no directory structure rclone will have to load them all into RAM at once. For 250M objects that will take lots of RAM! So much RAM that it might actually make copying with rclone impossible...
Assuming all the files aren't in one directory then rclone will hardly use any RAM - it will be using mostly network with a bit of CPU. A 4GB VM would be plenty I'd say.
Will you be repeating the copy, so trying to keep the source and destination in sync?
->However if there is no directory structure rclone will have to load them all into RAM at once. For 250M objects that will take lots of RAM! So much RAM that it might actually make copying with rclone impossible...
Any idea of possible ram usage?128GB,256GB?
->Will you be repeating the copy, so trying to keep the source and destination in sync?
There are some application working on source bucket, we need to migrate the data and the start the application on destination bucket. And then we will remove the old infrastructure.
So yes probably we will start the sync more then one time, to keep everything in sync.