What is the problem you are having with rclone?
When using Backblaze B2 for backing up individual devices, initial backups are clocking up quite a large bill due to the number of b2_list_file_names transactions.
This is the same issue as [Rclone (v1.46) using millions of b2_list_file_names](Rclone (v1.46) using millions of b2_list_file_names) however:
- using the
--fast-listflag is not an option on my systems due to its unconstrained nature and subsequently potential issues arising from the process using excessive amounts of memory, and - my device backups are set up and run automatically i.e. I am not able to manually try syncing specific directories to reduce the amount of memory used.
TLDR:
- I need to successfully backup file systems of varying sizes and depths
- The process is triggered automatically when the device is instansiated
- I want to minimise the amount of b2_list_file_names transactions (to reduce cost)
- I need to be able to limit the amount of ram used
- Ideally it would be great if the was an option like the
--fast-listflag but in which I could specify a memory limit. Does something like this exist? - Are there any other built in rsync options/config that could help?
- I guess another option would be to script around the problem, would it be possible to programmatically buffer a list of files to sync?
What is your rclone version (output from rclone version)
1.56
Which cloud storage system are you using? (eg Google Drive)
Backblaze B2
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync / :b2:backup/device --b2-account XXX --b2-key XXX flags --b2-hard-delete --skip-links -v
The rclone config contents with secrets removed.
Not currently using one, passing --b2-account and --b2-key flags
A log from the command with the -vv flag
N/A