What is the problem you are having with rclone?
Not really a problem, just a questions so I can better understand how to adapt to it. I'm copying a bunch of data over to GDrive, and I've getting my photos sync'd now, so lots of relatively small files. This eventually caused me to hit a 403 userRateLimit, but it's not clear to me when you get a 403 if it's an API calls rate limit, a number of files rate limit (I read in several places that can happen in this kind of operation), or the per day quota? Or is it possible to be any of those but not possible to distinguish which?
My initial copy line for the larger files rate limited to 8670Kbit/s, and there were several hours in the last day that I wasn't maintain anywhere near that from the metrics on the VM, so I don't think I hit the data cap. My line for the smaller files, I upped the simultaneous transfers to 2, and tweaked the transfer rate to 10Mbit, figuring I wouldn't hit that anyway with lots of small files, but obviously something still got hit.
So I'm trying to understand what I should be tweaking to try and stay under rate limits but get the best performance out of this transfer(lots of small files). The transfer rate seems straightforward, as does the simultaneous transfers, mostly. But I'm not clear where the API calls are influenced. Are transactions in the tps limit equivalent to API calls? I saw another thread suggesting checkers be set very high, like 40, to speed up a situation like this. But I'm not clear what impact a checker has, or what it is even checking? Is having a lot going to dramatically up the API calls? Is it limited by tpslimit in that case? I'm happy to read about these more if there's a good post somewhere explaining it, the docs currently seem a little bit vague in these particulars, and how/what to consider adjusting, or what impact some of the less obvious things have on what problems, for example, bwlimit is obvious, tpslimit less so, since it's not clear what constitutes a transaction.
Thanks!
What is your rclone version (output from rclone version
)
rclone v1.52.3
Which OS you are using and how many bits (eg Windows 7, 64 bit)
- os/arch: linux/amd64
- go version: go1.14.7
Which cloud storage system are you using? (eg Google Drive)
Google Drive target, OpenDrive source
The command you were trying to run (eg rclone copy /tmp remote:tmp
)
rclone copy od:photos/ sec:/ --multi-thread-streams 1 -P --transfers 2 --bwlimit 10240 --checkers=1 --tpslimit 10 --tpslimit-burst 20 --cutoff-mode=soft --drive-stop-on-upload-limit --retries 1 --low-level-retries 1 --timeout 20s --ignore-existing --log-file copy_log.txt --ignore-case --include *.jpg -P -v
I'm currently re-running, but transfer limited to 1, and bandwidth limited to 8670, and fast-list enabled, otherwise same command now (no problems so far).
The rclone config contents with secrets removed.
[od]
type = opendrive
username = [redacted]
password = [redacted]
[gd]
type = drive
client_id = [redacted].apps.googleusercontent.com
client_secret = [redacted]
scope = drive
token = {"access_token":"[redacted]","token_type":"Bearer","refresh_token":"[redacted]","expiry"
:"2020-09-17T14:47:24.789114638Z"}
[sec]
type = crypt
remote = gd:/sec
filename_encryption = standard
directory_name_encryption = true
password = [redacted]
A log from the command with the -vv
flag
Sorry, I messed up and started a new command before I renamed it like I usually do, but I captured the end of the log from my terminal. There weren't any errors until it quit (presumably because I set --drive-stop-on-upload-limit). I haven't been temp banned as far as I can tell (I have been able to run commands, and it's only been a few hours).
2020/09/17 07:07:49 INFO : !CloudMove/!Archives/Mz1/WB1/sh02.jpg: Copied (new)
2020/09/17 07:07:49 INFO : !CloudMove/!Archives/Mz1/WB1/sh03.jpg: Copied (new)
2020/09/17 07:07:50 INFO : !CloudMove/!Archives/Mz1/WB1/sh04.jpg: Copied (new)
2020/09/17 07:07:51 INFO : !CloudMove/!Archives/Mz1/WB1/sh05.jpg: Copied (new)
2020/09/17 07:07:51 INFO : !CloudMove/!Archives/Mz1/WB1/sh07.jpg: Copied (new)
2020/09/17 07:07:52 INFO : !CloudMove/!Archives/Mz1/WB1/sh10.jpg: Copied (new)
2020/09/17 07:07:53 INFO : !CloudMove/!Archives/Mz1/WB1/sh11.jpg: Copied (new)
2020/09/17 07:07:53 INFO : !CloudMove/!Archives/Mz1/WB1/sh12.jpg: Copied (new)
2020/09/17 07:07:53 INFO : !CloudMove/!Archives/Mz1/WB1/sh15.jpg: Copied (new)
2020/09/17 07:07:54 INFO : !CloudMove/!Archives/Mz1/WB1/sh17.jpg: Copied (new)
2020/09/17 07:07:55 INFO : !CloudMove/!Archives/Mz1/WB1/sh18.jpg: Copied (new)
2020/09/17 07:07:55 INFO : !CloudMove/!Archives/Mz1/WB1/sh19.jpg: Copied (new)
2020/09/17 07:07:55 INFO : !CloudMove/!Archives/Mz1/WB1/sh26.jpg: Copied (new)
2020/09/17 07:07:56 INFO : !CloudMove/!Archives/Mz1/WB1/sh27.jpg: Copied (new)
Errors: 68 (fatal error encountered)
Checks: 11652 / 11652, 100%
Transferred: 6012 / 16013, 38%
Elapsed time: 1h45m41.7s
2020/09/17 08:24:40 Failed to copy with 68 errors: last error was: googleapi: Error 403: User rate limit exceeded., userRateLimitExceeded