Just wondering what the best provider would be for rclone to backup 1 TB of data and thousands of files daily, with maybe 10 new files and 100 mb added per day. Currently there are some files around 100-500 mb, and a lot of 100-200 kb files.
I was interested in Google drive business account. But upon testing it, I find that syncing 200 files that didn't change takes 10 seconds which I feel is pretty slow. I tried to increase the api quota limit but then said I needed to attach it to a billed account.
Does anyone know how fast rclone can be made for google drive? I am used to using rsync and it seems to be able to sync thousands of files in a few seconds. Or is there a better provider to consider?
Although increasing the quota limit needs to be attached to a billing account, you will not be billed. Below is a screenshot of my quota:
I have no issue syncing/uploading 10 of thousands of small files, notice that my queries per second is 300 which is derived from 30,000/100.
I'm also uploading almost 1TB per day to GSuite Shared drives, but it's all big files. Here is my command for small files: --tpslimit 300 --fast-list -P
hello,
check out wasabi, i have been using them for many years.
wasabi is hot storage and does not have all those limits with transactions/sec and so on.
i have a verizon fios gigabit connection and using rclone i can max out that connection.
How did you get google to increase your quota? They are refusing to increase it for me and gave me a long laundry list of questions to answer.
Thank you for your email. We understand that you would like to further evaluate the request as you can see on your end that you are reaching the caps and getting rate limit error. My name is Angelique from Google Cloud Platform Team and I'm glad to assist you further.
At this point, in order for us to better understand your concern, we would like to ask you the following information below, so that we could relay it to our Engineering team:
1.Will you be hitting 80% of your QPS on a daily basis?
2.What is your current error rate %?
3.The number of users.
4.Average number of requests per day/per user (calculation describing your expected usage of the API).
5.Which API methods will be called and what will be the frequency.
6.Will you be polling the Drive API to check if files have been modified?
7.Have you looked into implementing push notifications[1] to save on quota?
8.Have you implemented exponential backoff[2] for your application?