When using Backblaze B2 for backing up individual devices, initial backups are clocking up quite a large bill due to the number of b2_list_file_names transactions.
This is the same issue as [Rclone (v1.46) using millions of b2_list_file_names](Rclone (v1.46) using millions of b2_list_file_names) however:
- using the
--fast-listflag is not an option on my systems due to its unconstrained nature and subsequently potential issues arising from the process using excessive amounts of memory, and
- my device backups are set up and run automatically i.e. I am not able to manually try syncing specific directories to reduce the amount of memory used.
- I need to successfully backup file systems of varying sizes and depths
- The process is triggered automatically when the device is instansiated
- I want to minimise the amount of b2_list_file_names transactions (to reduce cost)
- I need to be able to limit the amount of ram used
- Ideally it would be great if the was an option like the
--fast-listflag but in which I could specify a memory limit. Does something like this exist?
- Are there any other built in rsync options/config that could help?
- I guess another option would be to script around the problem, would it be possible to programmatically buffer a list of files to sync?
rclone sync / :b2:backup/device --b2-account XXX --b2-key XXX flags --b2-hard-delete --skip-links -v
Not currently using one, passing --b2-account and --b2-key flags