I have rclone copy set in a cronjob to copy 3 different sources to three different buckets on B2 Cloud.
I have transfers set to 32 for each job, and --fast-list on all, to reduce Class C transactions.
Source folder 1 to DestBucket:A, copies every hour. (important files always being updated/changed)
Source folder 2 to DestBucket:B, copies once a day at 1:30am. (less important files)
Source folder 3 to DestBucket:C, copies once every other day at 2:30am. (least important files)
If they happen to overlap, could there be any issues that I should be concerned with? Especially with regards to memory and bandwidth.
Memory might be a problem depending on exactly what kind of files you are transferring. The b2 backend needs to buffer large files in 96M chunks which it holds in memory. You are doing 32 at once so rclone could use up to 3GB of memory doing that. You can reduce --b2-chunk-size to make it use less memory, though the default value is B2’s recommended value.
Thank you for the insight. Maybe rather than reduce the chunk size, i should just reduce the transfer count. If i go with 16 threads per run @ 96M chunks I would only be dealing with 4.5GB max. Or i could just see if the system can handle a potential 9GB ram hit at 2am.