Proposal: Implement Exponential Backoff for Cache Background Uploads

[chaosaffe (Jason Murray)]

So, I managed to kill all three of my Kubernetes nodes this weekend due to a file to be uploaded missing. Long story short, rclone retried the upload several times a second, happily logging every attempt to disk, and after filling the disk of one node was evicted, and proceeded to fill the second node, etc...

I would like to propose (and am willing to attempt implementation of) an exponential backoff in the case of an error condition in the background upload (https://github.com/ncw/rclone/blob/master/backend/cache/handle.go#L594) to avoid flooding the logs and potentially being rate limited/banned by the remote API.

I would also open for consideration an option to discard failed uploads after a certain retry period expires.

Looking forward to your feedback,
chaosaffe

[ncw (Nick Craig-Wood)]

So, I managed to kill all three of my Kubernetes nodes this weekend due to a file to be uploaded missing. Long story short, rclone retried the upload several times a second, happily logging every attempt to disk, and after filling the disk of one node was evicted, and proceeded to fill the second node, etc...

I would like to propose (and am willing to attempt implementation of) an exponential backoff in the case of an error condition in the background upload (https://github.com/ncw/rclone/blob/master/backend/cache/handle.go#L594) to avoid flooding the logs and potentially being rate limited/banned by the remote API.

That would be great! Do you want to make a new issue on github to work from?

I would also open for consideration an option to discard failed uploads after a certain retry period expires.

An option would be fine I'm sure. I'm not sure it should be the default.