I’m relatively new to rclone but love what it does with syncing to cloud storage.
My goal is to sync home security footage to cloud storage and I’m testing out both S3 and Backblaze B2. I use Zoneminder for camera control and motion detection and have a script that the detection calls to rclone the videos to the cloud.
The script has more before and after, but these are the two rclone calls (back to back in the script).
rclone sync --fast-list --skip-links --include $czone/** --log-file $LOGFILE --config /srv/http/rclone.conf /mnt/storage/zoneminder/ b2drive:my-bucket
rclone sync --fast-list --skip-links --include $czone/** --log-file $LOGFILE --config /srv/http/rclone.conf /mnt/storage/zoneminder/ s3-test:my-bucket
$czone is a number that refers to the directory of camera that was triggered (1, 2, 3, etc). This helps reduce the numbers of invalid transfers due to other cameras being triggered at the same time.
This month, I’ve had a total of 657 events across all cameras (not unexpected as there are people usually home).
I am seeing an exponential number of more transactions on S3 than B2.
For S3, I used 180,000 GET requests for December (5 days)
For B2, I used 5,720 total transactions for December (5 days)
Is --fast-list no longer working with S3? Am I using it incorrectly?
Any information is appreciated!
- os/arch: linux/amd64
- go version: go1.11.2