Are there any recommended settings for rclone to be (almost) sure that you don't get 403 banned?
I have 16 TB of files on my school account which they told me they're gonna delete after the end of May. I tried the Google's own migration service from the school account to another Unlimited Drive account. It was very slow, did only about 4 TB in a week and then gave up. In addition it ruined the file dates by sharing/unsharing every file it could so far one by one. On the support page it says it may take up to a week but not that it might never complete!
So I tried another method, bought 16 TB of local storage, upgraded my internet speed from 50/50 mbps to 300/300 mbps and tried to download everything using Google File Stream and Unstoppable Copier which retains file dates, is able to continue after errors and not download all the already downloaded files again if I have to retry (which I have had to). I'm getting inconsistent speeds well below 300 mbps. And after a while of downloading, I have been banned twice for an unspecified amount of time!
How can I be sure under what limits of speed, data amount and file count of download per what period I have to keep in order to not get banned? Or is the Unstoppable Copier making too many API requests? The daily data limit was supposed to be 10 TB but I have been banned after around 1 TB of downloading, ~27k files.
If I leave unstoppable copier TRYING TO DOWNLOAD all the time after a ban, does that affect the ban staying on for longer or indefinidtely?
It's EXTREMELY FRUSTRATING because I don't know what I should do to save my data. If I try too slow, I run out of time. If I try too fast WITHOUT having any info about the limits, I get a BAN and I lose even more than when trying slowly!
I'm not really sure how to set up a VPS or what the best settings for RCLONE would be. Also, VPS is not gonna get me done it faster if there's the 750 GB/day upload limit. Or does it count in this situation?
By now I've tried it and using rclone on Google Cloud Compute and got banned again.
If you have local storage, your best bet imo will be to download it all using rclone to that storage. Download limits are I think 10TB. That will get you out of your time crunch. What made you think you were banned after 1TB? what method were you using?
I used Unstoppable Copier as described above. Later rclone on a Google VPS. Had to add the "--disable copy" for it to work, otherwise I got 404 errors. It showed only network usage, no disk usage and I got one 60gig directory done as a test. But now getting 403 again. For some time it did work on the VPS when I was banned on my home network. The problem is that regarding VPS and command line I'm not really sure what I'm doing.
So what settings should I use on rclone to avoid bans? If I just use default, isn't it unsafe? I can't afford 24h bans, time is running out! There are settings like --tps-limit and ---size-only and something regarding traverse and what not.
The problem with downloading locally is that I'm not getting 300 mbps, more like 180 mbps, sometimes 250 but inconsistently, throttling, stopping... It wouldn't be fast anough to finish before June anyway, let alone when I get limited repeatedly. Isn't there a better way? Maybe I can add GSuite accounts to get more than 750 GB/day upload.
I HAD made my own API key/client ID. Tried even changing the oauth, didnt help. OK I'll have to wait now and then try local downloading via rclone. Thanks
You had your own API as in you aren't using it anymore?
Can you share what command you are running, what version of rclone you are running and a log of the error you are getting now? I can't imagine you downloaded 10TB in 24 hours on a 300Mbs link.
This is extremely useful, thank you. But, please, forgive my naive question: how can I define "one query"?
Basically, I have the same problem as the OP, and I decided to sync a directory with 44 files as a test. Somehow, this was translated as 148 queries (checked the metrics on my API key later)... So, although I understand I can control bwlimit, I don't seem to be able to link bandwidth with number of queries.
I'm trying to do the exact same thing as the OP. The difference is that I have roughly 50 TB on local disk that I want to upload and my expectation was to be able to setup --bwlimit in a way that I don't exceed the limits set by the API...
I just re-read the original question and noticed that my case is different. I have the files on a local server and want to upload all the 50TB to a Team Drive. I know that I want to control bwlimit, but I don't know how to associate it with number of files / size of upload in a way the I can get an optimal estimate for bwlimit.
bw-limit won't affect anything but your upload speed. I've spoken to GSuite support on the phone recently as I migrated 35TB from Dropbox to Drive via rclone sync. They spefically said that it's fine to recursively let it hammer the API. I've also noticed if you navigate into the API's console from the dev console you CAN request an increase in API limits, but it requires a pretty detailed justification.
The fact of the matter is you're worrying about the wrong thing. GDrive has a hard-coded intake limit of 750GB/day. It's non-negotiable even on enterprise accounts. Corporate policy.
My advice: Don't care about the bwlimit unless it affects you locally, personally. Hit it as hard as you can - they've given the blessing to do so. Do the transfer as a recursive sync to skip existing files and use --size-only to ensure you don't half-copy a file. They won't penalize your account for this action - I specifically asked about this before initiating my migration.
