I am not planning on using rclone to upload stuff to Drive. I only need it to access my media through Kodi or DirectShow based players.
Through Google Drive File Stream a file gets cached locally but playback starts in a matter of a couple of seconds (gigabit connection). What I would like to get through rclone is playback starting but downloading to cache to proceed uninterrupted until the file has completely been cached.
I’ve had mixed luck in getting this through GDFS (good on some machines, bad on others), hence my looking into rclone, that appears to be much more configurable.
Whatever setting should keep in mind that Kodi has library functionality that scans media directory for new content and such (less intensive than Plex, from what I can tell from other friends’ experience).
The cache backend can cache the file to disk while your stream starts at the expense of a few second nore delay of the start time in my experience. Personally, I just use vfs cache writes and i use a webdav/http to serve and even on a remote vps things start in a few seconds in Kodi or any player.
First of all, thanks to the both of you for your replies.
I’m a total newbie here, so be patient while I try to wrap my head around the concepts you explain.
When you speak about the cache backend, is this what you are referring to? https://rclone.org/cache/
Considering I have a 1000Gbps download connection, what would be “ok” settings? I don’t need to cache a lot before playback starts, it can start as soon as it’s technically possible, as download speeds far exceeds the bitrate of whatever I can be watching.
The idea is to minimize download events, as those have brought me 24 hours bans in the past (there’s no way to have Google state clearly how they count multiple download events for the same video file being watched with direct connection streaming).
Considering that there’s library scans going on (tipically once a day), I need to be sure that this “cache backend” doesn’t attempt to download whole files while they’re being accessed for simple scanning reasons.
The machine in use has 16GB of RAM and a 512GB free SSD for caching purposes.
Edit: I see that “Cache” has several options specific to Plex, which I don’t use and I have no intentions of using. I hope it can be useful for my case just the same.
The regular VFS backend and Cache backend both do chunked downloading so that was a very legacy reason why standard rclone caused a ban as it made google think you downloaded the file many times with Plex as an example. That was fixed mid 2018 or sometime around there so assuming you grab the latest version (1.47 as of today), you’ll have no issues.
Neither VFS nor Cache do anything on their own as it’s all Plex or Emby or whatever that dictates how much of a file comes down.
Plex is the one I’m the most familiar with and that gets a few pieces to analyze the file when it’s first added and from that point, it’s just size/modtime checks which download nothing.
I use an encrypted GD backend for my media and a very simple mount:
/usr/bin/rclone mount gcrypt: /GD --allow-other --dir-cache-time 96h --drive-chunk-size 32M --log-level INFO --log-file /opt/rclone/logs/rclone.log --timeout 1h --umask 002 --rc
rc can be removed that’s just used if you want to interact via remote control commands.
umask is for setting permissions if you want user and group access along with some other access.
timeout is for pausing and resuming playback as that helps keep a connection open for an hour before closing is out.
dir-cache-time is high because that’s how it keeps the directory/file structure in memory and the longer the better as any changes are picked up every minute via polling.
Double clicked on a video file, it opened in MPC-HC (pretty standard DirectShow player) and in C:\Users\ashlar\AppData\Local\rclone\cache-backend\cache-gdfs\ I saw several 250MB files being downloaded… but playback was not starting. I stopped it at 11 files being created, for a total size of 2.62GB (whole file is 13.1GB). 11 download events were registered in the admin console.
Do you have any clue as to why playback isn’t starting? Files on Drive are not encrypted or anything. They play normally when accessed through Google Drive File Stream.
Problem is that every chunk creates a download event, which is what I’m trying to minimize. Or are you saying it doesn’t matter if these are generated through rclone?
In any case, I tried playing back a different movie with Kodi and, again, playback was not starting. I stopped it at 2.69 GB having already been saved in cache (12 files, having the offset size in bytes, such as 0, 262144000, etc.).
I wonder if there’s anything “special” that needs to be configured for this to work normally in Windows.
I’m using this to mount:
rclone mount --allow-other --timeout 1h --cache-db-path E:\rCloneCache cache-gdfs: Q:
Edit: I tried opening the “0” file with MPC-HC and it opened perfectly for the .mkv file that in reality it’s a chunk of. There must be some sort of “disconnect” at play here… :-/
I’m not afraid of the number of daily API hits, for the reasons you express I’ll never, ever risk reaching 1 billion hits. But download quota for a single file… 20GB file, hit hundreds of time… it’s easy to reach 10TB downloaded watching a couple of movies, if download events are counted every time as full size.
Edit: I say without vfs because the only reason I thought I needed vfs was to minimize the number of download events. I could use a memory buffer if that’s not important (I have 16GB after all). Or is vfs useful in my scenario?
Also, yes, I created my cliend ID/API key. I read instructions as well as I could before beginning.
So “chunked downloading” means that the API call specifies the amount of data that is being transferred for every download event? Is that what is happening and is this why one doesn’t risk surpassing the download quota?
Without using the cache backend but just vfs playback starts. I’ll see if any windows user has any more suggestions and in the meantime, thank you Animosity022.
If you have Drive File Stream questions, I’m sure they can help out with those particular things. We can happily answer any rclone questions here especially me with Plex as that’s my use case.
I’ve got ~60TB of encrypted data and switched over to rclone back in June 2018 once the new release hit. So I’m approaching almost 10 months now and never have seen a 24 hour ban nor any major API hits.
Thanks. No more GDFS talking, I agree. I was providing context but it became overly long.
I recover a couple of questions from the post in the old discussion that I (wrongly) resurrected.
Would it make sense to have a --vfs-read-chunk-size very low for initial library scanning purposes and then raise it once the library has been scanned (as subsequent additions, from day to day, would be a fraction of the initial scan)? Or is there a shared “optimum” value for scanning purposes?
I use Kodi, not Plex but the scanning process is quite similar. I think Kodi uses a call through the ffmpeg code it uses internally to scan for media info. In that it should be similar to Plex, I guess.
Also, as my experimentation has shown me, using vfs read chunks has very “little” video available, with 64MB of buffer to counter “network hiccups”, especially in high bitrate scenarios and I might want to consider increasing the memory buffer to something like 256 or 512MB. Any cons that I should bear in mind, were I to do this?