Copy/Download takes 40-50 sec before receiving first chunk/byte from google drive


#1

I just started to use rclone recently together with google drive, and mount it to plex.
When I start streaming something either thought plex or doing a “rclone copy” to the server, it takes 40-50 seconds, before it starts the download. When it has received the first byte/chunk, it downloads at full speed (~130 MB/s), so doesn’t seem to be the speed that is a problem. Hopefully someone knows what could might be the issue :slight_smile:

rclone.conf

[gdrive]
type = drive
scope = drive
token = ******

[gcrypt]
type = crypt
remote = gdrive:Private
filename_encryption = standard
directory_name_encryption = true
password = ******

Mount command

rclone mount  --allow-other --dir-cache-time 48h --vfs-read-chunk-size 16M --buffer-size 2G gcrypt: /home/plex/gdrive

#2

What version are you running?

Can you run a log level debug and share the output when you do whatever operation you are running?

Are you using unionfs or mergerfs or playing / copying directly from the mount?


#3

Version: 1.45

Debug log: https://pastebin.com/raw/F6FuDBHC

I’m playing / copying directly from the mount position (/home/plex/gdrive)


It seems that it took about 40 seconds before it got the docs.googleusercontent.com link, but still took about 20-30 seconds after that to start the download, so not sure if it’s rate limited related or not.


#4

No need for such a large buffer - try decreasing it to 64M which should be ~10s of streaming.


#5

Seems like it helped!
First try of a copy took 20 sec to start, but second time started after 5-10 sec, so improved a lot!


#6

Are you using your own API key?

Error 403: Rate Limit Exceeded, rateLimitExceeded)

Is why it is taking so long as you get those repeating.

I wouldn’t use this either:

--drive-v2-download-min-size

That’s a copy command as well and not a play from Plex so not sure if it is the same thing or not. If you can play something and share that log, it would be helpful.


#7

Not sure where to get and use an API key for rclone and google drive.

I have removed --drive-v2-download-min-size from the copy command. Read another thread about that it could result in some kind of lock, because of it, so have been removed since.

And here is a debug log from the mount with a play from plex. Still seem to be some issue with playing, as it took around 60 sec to start.
https://pastebin.com/raw/EcSAKhm8


#8

Yeah, the slow start is due to the rate limiting.

https://rclone.org/drive/#making-your-own-client-id

is how to make your own key.

You can’t really tune out of that as you’d want to create your own key.


#9

Have created a client id and secret and added to my gdrive config now. Still seems to give Rate Limit Exceeded. Not sure if I should wait a few hours before I can see the impact?
Debug log: https://pastebin.com/raw/11v7UVCc

[gdrive]
type = drive
scope = drive
token = ******
client_id = ******.apps.googleusercontent.com
client_secret = ******-******

[gcrypt]
type = crypt
remote = gdrive:Private
filename_encryption = standard
directory_name_encryption = true
password = ******

#10

Next issue, your chunk size way too low.

--vfs-read-chunk-size" "16M" 

You should make that at least 64M or 128M.


#11

I have changed that to 128M now.


#12

How is the start time now and what do you the logs look like?


#13

Took about 30 sec to start the episode now. Still some Rate Limit Exceeded in the debug log.
Debug log: https://pastebin.com/raw/6QCxv2GR


#14

It doesn’t seem like you are using your own API key.

Did you rclone conf update with a token and such?

Are you using a regular Google Drive?

My config looks like:

[GD]
type = drive
client_id = clientid.apps.googleusercontent.com
client_secret = secret
token = {"access_token":"acccesstoken","token_type":"Bearer","refresh_token":"token","expiry":"2019-01-14T08:14:03.756391301-05:00"}

#15

I used rclone config to edit “existing remote remote”, and added the client_id (clientid.apps.googleusercontent.com) and client_secret (secret). I removed scope = drive. Not sure what it does, so it looks like your config.

I use a Google Drive Business account, with a domain for it. (1 user)

[gdrive]
type = drive
token = {"access_token":"access_token","token_type":"Bearer","refresh_token":"refresh_token","expiry":"2019-01-14T14:00:20.052604978+01:00"}
client_id = clientid.apps.googleusercontent.com
client_secret = secret

[gcrypt]
type = crypt
remote = gdrive:Private
filename_encryption = standard
directory_name_encryption = true
password = ******

#16

That’s the same thing I use as well.

If you still get that, might be easier to just nuke the config and remake it as you shouldn’t see those 403s on your own ID.

If you check the API console -> https://console.cloud.google.com/apis/api/drive.googleapis.com

You should see hits on your ID as another indicator.


#17

I resat the client_secret in the API console and re-entered the client_id and client_secret and update the token in the config. I can see some information on the API console now, and there was no previous information, so it didn’t use the API as you suspected. Have not received a Rate Limit Exceeded yet. Also tried to start a stream, and it starts in 10 sec without a Rate Limit Exceeded, so seems like it has helped.


#18

That seems much better than before.

10 seconds is a bit longer than I’d expect but that does depend on if you are transcoding or direct streaming or a few other things like your bandwidth and such.

I usually start in 2-3 seconds at most but I am direct streaming or direct playing and I have a gigabit link.


#19

The server is on a gigabit connection, but Google API v3 is slow compared to v2 (Hence why I tested with --drive-v2-download-min-size earlier). I just tested with direct streaming, and takes about 15 sec to start the streaming in plex. There is no rate limit exceeded in the debug log, so that has been fixed.

Another issue is that it shows the correct playtime, before the stream has started playing, but after that it says the playtime is 2 hours, no matter how long the episode is.



#20

With the mount the V3 is just as fast as it does chunked reading. There is some data that shows the V2 was leading to bans but nothing confirmed so I personally would not recommend using it.

Start times could be peering or files not being analyzed. Would need to see the rclone logs and Plex logs when you hit play to confirm.

Times changing are usually due to analysis as well.