So, I managed to kill all three of my Kubernetes nodes this weekend due to a file to be uploaded missing. Long story short, rclone retried the upload several times a second, happily logging every attempt to disk, and after filling the disk of one node was evicted, and proceeded to fill the second node, etc...
I would like to propose (and am willing to attempt implementation of) an exponential backoff in the case of an error condition in the background upload (https://github.com/ncw/rclone/blob/master/backend/cache/handle.go#L594) to avoid flooding the logs and potentially being rate limited/banned by the remote API.
I would also open for consideration an option to discard failed uploads after a certain retry period expires.
Looking forward to your feedback,