B2 Class C calls exceeding 2500 suddenly

[rclone86]

What is the problem you are having with rclone?

What is your rclone version (output from rclone version)

1.50.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Unraid plugin - linux/amd64

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --fast-list "dir/to/folder name" crypt:folder/location

I am running this command in a script with about ten lines.

Total file amounts are ~850 GB and 150,000 files. The initial upload took awhile but went smooth. I was unaware of the "--fast-list" flag so I paid some extra money for thousands of "b2_list_file_names" calls.

I then added "--fast-list" to the commands and the calls dropped below the free 2500 threshold.

The cron job runs nightly and is using 4-5k Class C transactions. It's only $.02 or so per day but I believe something is up. My Unraid stats show 20-30% CPU usage for ~60 minutes during the sync every night (which is fine) and minimal memory usage (system has 16GB). So --fast-list is working but I expected to stay below 2500 calls.

I am syncing minimal amounts of data usually. Just a daily backup of a laptop and phone pictures. I'm not adding or changing dozens of GB per day.

[ncw (Nick Craig-Wood)]

Using --fast-list should do lists of size 1,000 so it should list the 150k files with 150 transactions if nothing has changed.

If you use -vv --dump headers you can see the http transactions - maybe you could count them (with a tool!) and see how many there actually are. If it is significantly different then there is a problem. Maybe you could post that log (or email it to nick@craig-wood.com with a link to this forum page).

You can reduce transaction like this also: use rclone copy --max age 1d --no-traverse to do a "top up" sync. This will copy all files which are less than a day old (it won't copy files that you have transferred already) without scanning the remote which is what costs the transactions. Then maybe once per week you could do an rclone sync to sync deletes and anything that got missed.