Google Drive. How to have video playback start immediately while caching on disk?

#1

After several problems with Google Drive File Stream, I’m trying to see whether rclone mount could substitute it for my media watching.

With this config:
rclone mount Google: q: --cache-dir E:\rCloneCache --vfs-cache-mode full --vfs-read-chunk-size 100M

Opening a video with MPC-HC generates a single download event in Admin console, but waits till the file has been fully downloaded to start playback.

All other vfs cache modes do not cache on disk, as far as I was able to check, and generate a big amount of download events.

Is there a way to have the file entirely cached while starting playback as soon as enough “material” is available? Google Drive File Stream does this.

I am using rclone 1.47 with WinFsp 1.4 if that matters somehow. Thanks for any help. :slight_smile:

0 Likes

#2

Some extra clarification.

I am not planning on using rclone to upload stuff to Drive. I only need it to access my media through Kodi or DirectShow based players.

Through Google Drive File Stream a file gets cached locally but playback starts in a matter of a couple of seconds (gigabit connection). What I would like to get through rclone is playback starting but downloading to cache to proceed uninterrupted until the file has completely been cached.
I’ve had mixed luck in getting this through GDFS (good on some machines, bad on others), hence my looking into rclone, that appears to be much more configurable.

Whatever setting should keep in mind that Kodi has library functionality that scans media directory for new content and such (less intensive than Plex, from what I can tell from other friends’ experience).
Thanks again.

0 Likes

#3

The cache backend can cache the file to disk while your stream starts at the expense of a few second nore delay of the start time in my experience. Personally, I just use vfs cache writes and i use a webdav/http to serve and even on a remote vps things start in a few seconds in Kodi or any player.

0 Likes

#4

Why does it have to be cached on disk? If that is a need, the cache backend would be better.

I don’t cache anything on disk and just use a standard mount with a few minor tweaks.

0 Likes

#5

First of all, thanks to the both of you for your replies.

I’m a total newbie here, so be patient while I try to wrap my head around the concepts you explain.
When you speak about the cache backend, is this what you are referring to?
https://rclone.org/cache/

Considering I have a 1000Gbps download connection, what would be “ok” settings? I don’t need to cache a lot before playback starts, it can start as soon as it’s technically possible, as download speeds far exceeds the bitrate of whatever I can be watching.

The idea is to minimize download events, as those have brought me 24 hours bans in the past (there’s no way to have Google state clearly how they count multiple download events for the same video file being watched with direct connection streaming).

Considering that there’s library scans going on (tipically once a day), I need to be sure that this “cache backend” doesn’t attempt to download whole files while they’re being accessed for simple scanning reasons.
The machine in use has 16GB of RAM and a 512GB free SSD for caching purposes.

Edit: I see that “Cache” has several options specific to Plex, which I don’t use and I have no intentions of using. I hope it can be useful for my case just the same.

0 Likes

#6

The regular VFS backend and Cache backend both do chunked downloading so that was a very legacy reason why standard rclone caused a ban as it made google think you downloaded the file many times with Plex as an example. That was fixed mid 2018 or sometime around there so assuming you grab the latest version (1.47 as of today), you’ll have no issues.

Neither VFS nor Cache do anything on their own as it’s all Plex or Emby or whatever that dictates how much of a file comes down.

Plex is the one I’m the most familiar with and that gets a few pieces to analyze the file when it’s first added and from that point, it’s just size/modtime checks which download nothing.

I use an encrypted GD backend for my media and a very simple mount:

/usr/bin/rclone mount gcrypt: /GD --allow-other --dir-cache-time 96h --drive-chunk-size 32M --log-level INFO --log-file /opt/rclone/logs/rclone.log --timeout 1h --umask 002 --rc

rc can be removed that’s just used if you want to interact via remote control commands.
umask is for setting permissions if you want user and group access along with some other access.
timeout is for pausing and resuming playback as that helps keep a connection open for an hour before closing is out.
dir-cache-time is high because that’s how it keeps the directory/file structure in memory and the longer the better as any changes are picked up every minute via polling.

1 Like

#7

Tried with this:

[cache-gdfs]
type = cache
remote = gDrive:
chunk_size = 250M
chunk_total_size = 50G

Double clicked on a video file, it opened in MPC-HC (pretty standard DirectShow player) and in C:\Users\ashlar\AppData\Local\rclone\cache-backend\cache-gdfs\ I saw several 250MB files being downloaded… but playback was not starting. I stopped it at 11 files being created, for a total size of 2.62GB (whole file is 13.1GB). 11 download events were registered in the admin console.
Do you have any clue as to why playback isn’t starting? Files on Drive are not encrypted or anything. They play normally when accessed through Google Drive File Stream.

0 Likes

#8

250M is huge for a chunk size.

Try something like 32M or 64M.

I’ve got zero familiarity with how MPC-HC plays files so not sure how it works.

0 Likes

#9

Problem is that every chunk creates a download event, which is what I’m trying to minimize. Or are you saying it doesn’t matter if these are generated through rclone?

In any case, I tried playing back a different movie with Kodi and, again, playback was not starting. I stopped it at 2.69 GB having already been saved in cache (12 files, having the offset size in bytes, such as 0, 262144000, etc.).

I wonder if there’s anything “special” that needs to be configured for this to work normally in Windows.
I’m using this to mount:
rclone mount --allow-other --timeout 1h --cache-db-path E:\rCloneCache cache-gdfs: Q:

Edit: I tried opening the “0” file with MPC-HC and it opened perfectly for the .mkv file that in reality it’s a chunk of. There must be some sort of “disconnect” at play here… :-/

0 Likes

#10

The issue before was that each time a file was read, it would count as a complete download of that file.

So if you had a 10GB file and it got read 10 times, it counted as 100GB for the download quota for that file. The ban/issue/error was that files were exceeding download quota.

If you can push the API quota per day, which is 1 billion, good luck :slight_smile:

I push barely 20k API hits per day with 60Tb of data and 4-5 people streaming.

If you want to use cache, you need a small chunk size or it’s going to be tough.

You also need to make and use your own client ID/API key:

https://rclone.org/drive/#making-your-own-client-id

That being said, I wouldn’t use the cache backend unless you have a reason to store chunks as it’s just overhead.

0 Likes

#11

Thanks once more, I cannot tell you how much I appreciate the time and patience. I’m sadly aware of my ignorance here so… really thanks!

So you’re telling me that I could safely ignore the number of download events and mount without cache or vfs configured? Like… you’re 100% sure that the number of download events displayed at https://admin.google.com/AdminHome?pli=1&fral=1#Reports:subtab=drive-audit don’t matter and the only thing that matters is the whole file size being transferred?

I’m not afraid of the number of daily API hits, for the reasons you express I’ll never, ever risk reaching 1 billion hits. But download quota for a single file… 20GB file, hit hundreds of time… it’s easy to reach 10TB downloaded watching a couple of movies, if download events are counted every time as full size.

Edit: I say without vfs because the only reason I thought I needed vfs was to minimize the number of download events. I could use a memory buffer if that’s not important (I have 16GB after all). Or is vfs useful in my scenario?
Also, yes, I created my cliend ID/API key. I read instructions as well as I could before beginning.

0 Likes

#12

rclone only does chunked downloading so you don’t hit an issue of download quota if for example you run mediainfo on a 50GB movie 1,000,000 times.

I use my mount command I shared above on a encrypted GD without any issues. I am a Plex user though so that’s my use case. A linux server running plex with sonarr/radarr/etc.

0 Likes

#13

So “chunked downloading” means that the API call specifies the amount of data that is being transferred for every download event? Is that what is happening and is this why one doesn’t risk surpassing the download quota?
Without using the cache backend but just vfs playback starts. I’ll see if any windows user has any more suggestions and in the meantime, thank you Animosity022. :slight_smile:

0 Likes

#14

There is a good write up by the guy that wrote the chunked reading part here:

He explains the problem before and what he did to solve the issue.

I mean, you can still hit the quota assuming you have a nice fat pipe and download the same file many, many times :slight_smile:

0 Likes

#15

That looks like a very informative read. Thanks for sharing. I will study it.

0 Likes

#16

Is anybody knowledgeable enough to tell me whether this log entry from Google Drive File Stream suggests that it, too, is using chunked downloading?

“26”: {
time: “2019-04-02 15:52:08.504Z”
type: “DISCOVER_URL”
status: “”
entry {
url: “https://www.googleapis.com/drive/v2internal/files/fileID?alt=media&revisionId=revisionID
hash: 14421339640934727316
}
}
“27”: {
time: “2019-04-02 15:52:08.504Z”
type: “BEGIN_HTTP_REQUEST”
status: “”
entry {
request_id: 7
url: “https://www.googleapis.com/drive/v2internal/files/fileID?alt=media&revisionId=revisionID
hash: 14421339640934727316
}
}
“28”: {
time: “2019-04-02 15:52:09.305Z”
type: “CELLO_FS_READ”
status: “SUCCESS”
latency_ms: 618.411
entry {
pid: 5296
entry_id: 2847
handle: 3033
offset: 8510898828
length: 131072
actual_length: 99432
stable_id: 22803
process_name: “player.exe”
}
}
“29”: {
time: “2019-04-02 15:52:09.322Z”
type: “BEGIN_HTTP_REQUEST”
status: “”
entry {
request_id: 8
url: “https://www.googleapis.com/drive/v2internal/files/fileID?alt=media&revisionId=revisionID
hash: 14421339640934727316
}
}
“30”: {
time: “2019-04-02 15:52:09.921Z”
type: “END_HTTP_REQUEST”
status: “”
entry {
size: 99432
request_id: 8
url: “https://www.googleapis.com/drive/v2internal/files/fileID?alt=media&revisionId=revisionID
hash: 14421339640934727316
http_code: 206
duration_ms: 599
}
}
“31”: {
time: “2019-04-02 15:52:09.923Z”
type: “CELLO_FS_READ”
status: “SUCCESS”
latency_ms: 0.067
entry {
pid: 5296
entry_id: 2847
handle: 3033
offset: 8510998260
length: 131072
actual_length: 0
stable_id: 22803
process_name: “player.exe”
}
}
“32”: {
time: “2019-04-02 15:52:09.924Z”
type: “CELLO_FS_READ”
status: “SUCCESS”
latency_ms: 0.142
entry {
pid: 5296
entry_id: 2847
handle: 3033
offset: 4135
length: 131072
actual_length: 131072
stable_id: 22803
process_name: “player.exe”
}
}
“33”: {
time: “2019-04-02 15:52:09.924Z”
type: “CELLO_FS_READ”
status: “SUCCESS”
latency_ms: 342.952
entry {
pid: 5296
entry_id: 2847
handle: 3033
offset: 8510853493
length: 131072
actual_length: 131072
stable_id: 22803
process_name: “player.exe”
}
}

0 Likes

#17

If you hit play, does it start right away or do you see it download the entire file? Regardless of the logs, it should be pretty easy to figure that out.

You can see some key words in those logs like ‘offset’ and “actual_length”.

0 Likes

#18

Yes, playback is almost instant.
And yeah, I noticed those key words. Those are what prompted me to ask.

0 Likes

#19

Drive File Stream uses Dokany, with Google contributing to the main codebase (https://github.com/google/google-drive-dokany).
It has by default a cache mode, where files being accessed are temporarily copied online before being accessed. What pointed me to rclone direction is that I’ve experienced very different behavior on two machines I’m using.

One starts playback and starts caching, with caching continuing while playback is ongoing and stopping only when 100% of the file has been cached (with gigabit connection it means that the file is safely cached in a matter of minutes while watching a movie). This generates a really small amount of download events in the admin console (API calls I guess, although there’s no way to check API calls for GDFS, as that’s an internal Google app). If playback is stopped before caching is complete, caching stops immediately.

The other machines acts more similarly to what I’m seeing with rclone, caching a small portion of the file being played back and resuming the caching process once the player asks for “more chunks” to playback. Start, stop, start, stop. This generates many more download events.

Since for my use case I’d rather use more transfer quota than API calls, I would have loved to get the first behavior on the second machine (which is my main playback machine) but, alas, there appears to be no way to make it happen and no way to configure GDFS to make it happen. Even though space available on disk and RAM is far in excess of what’s been used (32GB and 16GB respectively). To this day the difference remains a mistery.

Sorry for the offtopic, I just wanted to provide context. From what I’ve seen so far through experimentation, there’s no way to get the first behavior to happen through rclone too.

0 Likes

#20

You are trying to solve something that simply just isn’t a problem.

There are plenty of API calls to make during a single day as you can’t really go over the limit as Google only allows ~10 per second with their default quotas.

The term download ‘event’ does not really apply and just adds to confuse other folks as it makes an issue that doesn’t exist.

You have a few options in rclone.

  • Standard - you can stream a file using just memory which is more the default settings
  • Two cache mode options
    • use a vfs cache mode and keep a file on disk for a period. this requires getting a whole file
    • ‘cache backend’ - this does chunked downloading and retains parts of a file. bigger chunk storage size keeps files around longer.
0 Likes