Does the rclone syn command have file transfer limit?

What is the problem you are having with rclone?

When using rclone sync, the --progress flag is not showing the correct amount of data to transfer.

What is your rclone version (output from rclone version)

rclone v1.47.0

  • os/arch: linux/amd64
  • go version: go1.12.9

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian Bullseye x64

Which cloud storage system are you using? (eg Google Drive)

pCloud

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync
--config=/home/user/.config/rclone/rclone.conf
--log-file /home/user/Logs/Rclone/rclonesync.log
--log-level "DEBUG"
--progress
--crypt-show-mapping
'/media/Data/Files' 'psync:/Files'

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

The amount of files to transfer is: 49,448
The amount of data in gb to transfer is: ~300 GBytes

The amount of files rcync is reporting: 10,011
The amount of data in gb rcync is reporting: 116.301 GBytes

When the sync operation is started, the progress screen shows:

Transfered: 100M / 116.301 GBytes ...
Errors: 0
Checks: 5
Transfered: 24/ 10011, 0%
...

So the GBytes and number of files to transfer is off by a huge factor. Does rclone have any built in file or data transfer limits? Has some buffer been reached where I cannot transfer over x number of files?

I have used this same command and testing rclone for a while now on data sets with larger files and found it to be completely accurate when reporting on the number of files and the amount of Gbits to transfer. In this case it is way off. I am not sure what to look at next.

How is this Transferred files and Transferred Gbits values calculated?

You are using a really old version so I'd update to start.

It does the calculation by using a backlog of items and it builds. It doesn't compute the whole thing in the beginning.

 --max-backlog int                      Maximum number of objects in sync or check backlog. (default 10000)
     

From the docs

--max-backlog=N

This is the maximum allowable backlog of files in a sync/copy/move queued for being checked or transferred.

This can be set arbitrarily large. It will only use memory when the queue is in use. Note that it will use in the order of N kB of memory when the backlog is in use.

Setting this large allows rclone to calculate how many files are pending more accurately and give a more accurate estimated finish time.

Setting this small will make rclone more synchronous to the listings of the remote which may be desirable.


So set --max-backlog larger which will use (a bit more memory) but it will make your stats more accurate.