How to get an estimated time of transfer in --dry-run?

What is the problem you are having with rclone?

I am trying to get an estimate of transfer from a remote (SFTP) to another remote (Google Cloud Storage). I set the flag --dry-run since I do not want to execute the transfer now. Does rclone go through all objects to compute this, and have to display the estimated time (Elapsed time:) at the end? So for example, if syncing the file can go for 6 hours, I will also have to wait for 6 hours for that information?

Is there a way to print the file size and the average transfer speed and compute the estimated time without going through the a complete --dry-run transfer?

Run the command 'rclone version' and share the full output of the command.

[3:34 pm] Nikko Pante
rclone v1.59.2
- os/version: Microsoft Windows 10 Pro 21H2 (64 bit)
- os/kernel: 10.0.19044.2006 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.18.6
- go/linking: static
- go/tags: cmount

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --P --dry-run remote-1:/data remote-2:/data

Thank you in advance.

Usually a --dry-run sync is very quick.

Try rclone about remote-1:/data - this will give you the size of all the objects in the source and you can work out a worst case transfer time like that assuming you know how fast your network connection is.

1 Like

Thank you @ncw.

I would like to ask for more details. I was able to know the file size rclone about remote-1:/data.

Total:   1.921 TiB
Used:    1.036 TiB
Free:    906.868 GiB

I used --dry-run again and waited for it to finish. The Elapsed time for this is 53mins. Question. This is the time the --dry-run was completed right? This will not be the estimated time when the actual sync runs? Also, from the result. I was able to transfer 3.102 TiB. I am not sure where the number came from when my data is only 1.036 TiB.

Transferred:        3.102 TiB / 3.102 TiB, 100%, 1.664 GiB/s, ETA 0s
Errors:                 1 (retrying may help)
Transferred:      4992885 / 4992885, 100%
Elapsed time:     53m27.3s

Is knowing my download speed in a speed test reliable to calculate the transfer time? In the speed test, I have197Mbps. If I calculate a file transfer of 1.1TB. The transfer time is ~1hr30mins.

Another question. Let's say I wanted to run this on an external machine (e.g. Google's Compute Engine). Would I also be able to know the estimated transfer time since I do not have information on its network?

hi,
with rclone sync, if there is any error, rclone will retry the entire sync three times

if rclone has upload 99% of a file, and there is an error,
then rclone will have to re-upload the entire file again.

a debug log would show exactly what is going on but
1.1TiB X 3 equals approx 3.102 TiB

running rclone on a google compute engine, is not much different from running rclone on local.
ingress should be at no cost, tho for egress, you can be charged.

1 Like

@asdffdsa answered this above - it is because there was an error rclone retried. You'll have to take a look at the log to work out what is happening.

197 Mbps is 197 Mbits/s, which is approx 24 MByte/s so I make your transfer time nearly 13 hours.

Rclone can saturate your network bandwidth no problem. Take care that you measured the right upload or download speed? For home connections at least, uploads are usually much slower.

You can do a test, but I'd expect around 100 MiB/s on GCE or faster so your 1TB would take 3 hours or so.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.