I am in the process of evaluation a migration method for our environment. I was wondering if rclone could able to handle our migration need.
We have on-premises object storage that needs to be migrated to different vendor. We have roughly around 2.5 Billion small objects (avg. size is around 20K) in our largest bucket. We’re using keys similar to this example: 0111f0/60/00/00/object.dat.
During the migration, we would need to achieve around 1000-1500 TPS, faster than that will stress the source too much and slower will take too much time.
As far as I understood, Rclone requires quite a lot of memory on and CPU when working with large number of objects. Is it able to handle this much?