I am using rclone 1.51.0 on Windows 7 x64. I created google encrypted remote and generated client id and secret. When i start the upload it starts with good speed, about 1.8 Mb/s. But then gradually is starting to declining and after 30-40 minutes reaches about 300 Kbyte/s and stays like that. Why is this happening? I am using this command: rclone sync e: gdrive-crypt: --filter-from filter-file.txt.
to understand what is going on, you need debug info
- you can add flag
-vvto your existing command
or - you can add a log file with debug info
https://rclone.org/docs/#log-file-file
https://rclone.org/docs/#log-level-level
rclone sync e: gdrive-crypt: --filter-from=filter-file.txt --logfile=logfile.txt --log-level=DEBUG
What is your upload bandwith?
Doe the slowdown happen when you start to transfer a lot of very small files? Or does it happen even with larger files, for example 50MB or more? (you can use -P to see a progress report that shows you which files is currently being handled).
rclone crypt runs pretty light, so it should not impact transfer speeds unless you are trying to run it on a 15 year old CPU or atom ect. It would be useful to know if you have tried uploading directly to the Gdrive (no crypt) and see any difference though. I would not expect it, but it is easy to test so we can eliminate that possibility from the equation.
In the log file is see this error:
Received error: googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=808654597803, userRateLimitExceeded - low level retry 1/10
But if i go to the url in the error i see this: The API "drive.googleapis.com" doesn't exist or you don't have permission to access it .
gdrive has many kinds of quotas and it looks like you have excedded one or more of them.
@thestigma knows all about it.
My upload bandwidth is 30 Mbps. I tested uploading data using the standard non encrypted remote and i am seeing the same thing. It started uploading with speed about 2.5 Mb/s. And then started decreasing gradually. The file size doesn't seem to matter. Sometime i also see this error:
2020/03/23 19:21:22 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 500: Internal Error, internalError)
2020/03/23 19:21:22 DEBUG : pacer: Rate limited, increasing sleep to 1.811516352s
2020/03/23 19:21:22 DEBUG : pacer: Reducing sleep to 0s
That particular 403 (there are a few different ones with slightly different text) is an API-throttle error.
You can disregard it, especially if they are 1/10 or 2/20. They will appear in normal use for systems that work perfectly fine. The server is basically just telling rclone it needs to slow down the rate of requests a little bit - and rclone will adjust it's frequency automatically.
With your bandwidth you should be able to get about 3,2 MB/sec rather easily if you include the protocol overhead. If you get less than this on larger files then there is an issue that needs to be fixed.
It would help if I could see the full log. Is that possible? names of files may leak in that log, but that's about it security-wise.
What region of the world do you live in? I know some very remote places have issues with routing and some places (especially in china it seems for some reason) experience network throttling for some reason. It would nice to be able to just cross this of the list if it doesn't apply to you.
Finally, I would recommend you change your Gdrive entry in the config file to have the line:
chunk_size = 64M
(32M will be decent too if you have very limited RAM).
This is a general performance optimization on larger files, not a fix for your spesific issue - which I suspect is very much unrelated to this setting. (but it's nice to have for later when we fix the problem) 
I don't upload large files. 90% of the files are with less than 60MB size.
I can't post the log, sorry, it contains my private files.
My location is in eastern europe. I have to check with my ISP provider if they don't throttle my upload speed.
there have been a lot of issues with internet in europe recently.
for example, netflix and other websites, are forced, by the government, to reduce the bandwidth being used.
When I say "large files" I mostly refer to "not tiny" files hehe. 50-60MB is fine and you will get good performance out of those. There is very little point to setting a higher chunk value than the size of the largest files you usually transfer - so 64M or even 32M is probably good for you (larger is always better, but it does use a bit more RAM is the tradeoff).
But if you upload let's say 1000 x 100kb documents then just be aware that this will be slow (on Gdrive and many other cloud providers) no matter the setup (unless you zipped them to a single file). This is because Gdrive will not allow more than about 2-3 new transfers to be started pr second (on upload that is - download is a little more liberal), so on tiny files your bandwidth will not be well utilized since that limit is effectively going to cap the performance you can see (on such small files).
Just make sure to not log speed-test results from these types of tiny files is what I am saying - or else they will completely ruin your results.
If you transfer a lot of small files (but also some large ones) then you can use these flags in the newest beta to help alleviate a lot of this issue:
--order-by size,mixed,80 --transfers 5
(but you should still test your optimal speed on only large files so we can get a good baseline datapoint).
It is true that various networks and providers (including Google's Gdrive) have some intermittent issues currently (pretty bad a few days ago) due to a massive spike in user activity. I doubt you will really feel it much since your max upload is fairly low though - but of course it is perfectly possible that an eastern european country with so-so network infrastructure may be locally overloaded too right now. That is very difficult to really say from where I sit. If this was the case you would almost certainly see varying results throughout the day - with speeds being far better in the late night most likely.
I don't initially think this is the problem you are seeing here, but it's something to at least be aware of as it's such an overarching problem that potentially can cause issues for anything on the internet.
I am uploading lot of files with small size, so that might be the issue. Will try the suggested flags.
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.