I am trying to get an estimate of transfer from a remote (SFTP) to another remote (Google Cloud Storage). I set the flag --dry-run since I do not want to execute the transfer now. Does rclone go through all objects to compute this, and have to display the estimated time (Elapsed time:) at the end? So for example, if syncing the file can go for 6 hours, I will also have to wait for 6 hours for that information?
Is there a way to print the file size and the average transfer speed and compute the estimated time without going through the a complete --dry-run transfer?
Run the command 'rclone version' and share the full output of the command.
[3:34 pm] Nikko Pante
rclone v1.59.2
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2006 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: cmount
The command you were trying to run (eg rclone copy /tmp remote:tmp)
Try rclone about remote-1:/data - this will give you the size of all the objects in the source and you can work out a worst case transfer time like that assuming you know how fast your network connection is.
I used --dry-run again and waited for it to finish. The Elapsed time for this is 53mins. Question. This is the time the --dry-run was completed right? This will not be the estimated time when the actual sync runs? Also, from the result. I was able to transfer 3.102 TiB. I am not sure where the number came from when my data is only 1.036 TiB.
Is knowing my download speed in a speed test reliable to calculate the transfer time? In the speed test, I have197Mbps. If I calculate a file transfer of 1.1TB. The transfer time is ~1hr30mins.
Another question. Let's say I wanted to run this on an external machine (e.g. Google's Compute Engine). Would I also be able to know the estimated transfer time since I do not have information on its network?
hi,
with rclone sync, if there is any error, rclone will retry the entire sync three times
if rclone has upload 99% of a file, and there is an error,
then rclone will have to re-upload the entire file again.
a debug log would show exactly what is going on but
1.1TiB X 3 equals approx 3.102 TiB
running rclone on a google compute engine, is not much different from running rclone on local.
ingress should be at no cost, tho for egress, you can be charged.
@asdffdsa answered this above - it is because there was an error rclone retried. You'll have to take a look at the log to work out what is happening.
197 Mbps is 197 Mbits/s, which is approx 24 MByte/s so I make your transfer time nearly 13 hours.
Rclone can saturate your network bandwidth no problem. Take care that you measured the right upload or download speed? For home connections at least, uploads are usually much slower.
You can do a test, but I'd expect around 100 MiB/s on GCE or faster so your 1TB would take 3 hours or so.