For the cpu, i beg to slightly differ IF you have a high bandwith machine to do the transfer (like capable of something like 1gb/s down 1gb/s up), and the source and destination rclones are crypted. It will hammer one core if you do this in one job.
If these are two remote directories, like google drive, dropbox, or even just separate NAS's. Then the limiting factor will be your network speed.
If your source and destination directories are on the same machine, then disk IO is going to be your limiting factor.
You could use rclone copy with the -P flag and a small subset of your data. That will allow you to see your transfer rate, as something like 64.388 MiB/s. You can then use a download time calculator to give you an estimate time based on the total amount of data you need to transfer.
FWIW, I just transferred 5.8TiB of data from Google to Dropbox using a Linode cloud server with a 1Gbps connection, and it took 27hrs and 13 minutes.
Thank you estimate, these are some good points!! @H4v0k Thanks for your numbers, it does give me some estimate!
To add more details to my query, I have to transfer about 10TB of data from Akamai Netstorage to Amazon S3.
I don't have a lot on insight into Akamai netstorage currently and hence the issue. Based on their documentation, they support 1000read/sec and 90 concurrent connections.
I was wondering if I have to make this faster, do I need to use 2-4 machines with rclone running
It should fly given you have fast internet Cx. rclone will not be bottleneck here.
You can like @H4v0k use some temporary server with fast two way connections (Linode, Digitalocean or why not Amazon).
10TB with 1Gb Cx will take no longer than 2 days max. I would not overcomplicate it with multiple machines unless for whatever reason you have to do it much faster.
Yeah, I know, however I don't have access to Netstorage yet and unfortunately, I have to give them an estimate before that
So I was trying to figure out if there was a way. If using a machine with specific specs can help me figure this out
Unless you will run it on Raspberry Pi then any modern computer should handle it without issues - even with 10Gb Internet. Only moving to much higher bandwidths it would require more hardware planning. I assume you do not have 10Gb+ Internet
Sorry do not have it on top of my head now:) But as you need it only for less than 48h do not penny pinch too much. Better to have something that can rock. 8+ cores, 32+ GB RAM I would use just to have nice sleep knowing that it will handle all traffic.
What you should check is networking cost for 10TB traffic (assume 12TB to have some margin for testing).
well, that is a big problem, no way to offer an accurate estimtate.
as this is a one time transfer, do not need to be super cheap with the vm specs, the cost difference will not matter much.
this should be more than enough but as per @kapitainsky, make sure you can sleep.
note: for free ingress into amazon, the cloud vm must be in the same region as the bucket.
as for egress from Akamai, i have no idea.
I have one more question;
Looking at the documentation for netstorage, it says -
"This will effectively allow commands like copy/copyto, move/moveto and sync to upload from local to remote and download from remote to local directories with symlinks."
Can I not directly copy from remote server 1 ( netstorage) to remote server 2 (s3)?
The netstorage page only refers to copying to local directories.
Also didn't understand the concept of symlink