Google Drive vs. ACD for Plex

Stokkes · March 6, 2017, 8:15pm

This should, from what I`m hearing, absolutely avoid bans when doing a Plex scan.

jmacul2 · March 6, 2017, 10:21pm

Grep to see if there are any folders that it couldn’t parse. Do you have any 0 length folders or files?

thejinx0r/node-gdrive-fuse/blob/ba99b5ac254cf3afeac15a3427fd58289f38ac95/src/client.es6.js#L298


    }else{
        left.push(f);
    }
}




// logger.info "Finished parsing files"
// logger.info "Everything should be ready to use"
// saveFolderTree()
__items_to_parse_from_google__ = left;
logger.debug(`After attempting to parse, there is ${inodeTree.map.size} items in the inodeTree and ${__items_to_parse_from_google__.length} items that were not yet parseable`);
}




function loadFolderTreeCallback(err){
if(err){
    getAllFiles();
    return;
}
const changeFile = pth.join(dataLocation,'largestChangeId.json');
fs.exists( changeFile, function checkLargestChangeIdJsonExistsCallback(exists){

The code indicates that it seems like it won’t parse things that don’t have a content length.

I would grep to see if you have any other log messages along those lines. I bet its garbage files honestly. How organized and clean are your files? I mean 30 TB? Thats a lot…haha

jmacul2 · March 6, 2017, 10:24pm

You could alter the code to and log the output of items_to_parse_from_google to see what it is that is not parsing.

Stokkes · March 6, 2017, 11:53pm

@jmacul2

Hmm, I’m quite rusty at javascript and little experience with node.

I’m mounting my Google Drive, but the Media folder is encrypted with rclone crypt, not sure if it’s balking at that.

For example, I recently added Planet Earth II, the documentary and it isn’t showing at all when I mount the folder. When I mount via rclone mount, it shows fine (and plays fine).

Also, every time I do a node client.es6.js to rebuild the index, I get a slightly different number of files/folders that are not parseable, which makes me think it’s an API limitation, timeout, lack of retries, or something. One time I"ll get 165 folders, if I re-run immediately it’s 164…

Very odd.

jmacul2 · March 6, 2017, 11:57pm

Yeah…I checked your issue to see if I had the same problem. I blew away my database and set everything up on a testing machine. I got 1 file that was not yet parseable. It was a subtitle (.srt) file. But I checked back and found that a few seconds later it had parsed and shown up. So it might be that you are being rate limited. You can tail -f the log file and see if new files are being picked up every 60 seconds. I suspect cause your library is so large that it might take a bit for the initial scan to be completed. Just my guess.

I decided not to do a encrypted volume on Google drive. I know that google encrypts the contents of drive automatically. So…they say, that they cant read them. We know that amazon does NOT do this so I did have an encrypted mount when I was using ACD. Also, I have plex pass, and its nice to use Plex Cloud. If I get a DMCA takedown by google…meh, I’ll just use ACD until I replicate back to gdrive encrypted.

jmacul2 · March 7, 2017, 12:03am

You can just add this line after line 298 on client.es6.js

`logger.debug(__items_to_parse_from_google__);`

Then cat your log file, you will have tons to see…hahaha.

Stokkes · March 7, 2017, 12:18am

I may go that route, but here’s a question

For the initial scan, it tells me it can’t find 3000+ files and 165 folders, so there’s probably lots of stuff missing.

Now, if I’ve missed the initial scan, what are the chances it will find them later on? MY understanding of the process is the initial scan (node client.es6.js) is used to build the initial database and then it just looks for “changes” from the API.

If these are not really changes and should have been picked up in the initial scan, these folders won’t necessary change, so I got the impression they wouldn’t appear in the changes API.

Not sure if that makes sense.

Also, I did edit the file to add the output of the notFound variable and did get a lot of data. Unfortunately I can’t match any of the names with my rclone lsl crypt: --crypt-show-mapping and rclone lsd crypt: --crypt-show-mapping output.

jmacul2 · March 7, 2017, 12:23am

Huh…interesting. Remember, node-gdrive-fuse mounts to the root of your google drive account where rclone gives you the option to mount within a folder off the root. Is your plex stuff off the root? is it possible those files are outside your rclone root? make sense?

Did your Planet Earth stuff eventually show up? I’ve never seen a problem of missing files from my end. So I wonder if the rclone crypt is the problem.

Stokkes · March 7, 2017, 12:47am

So here’s my setup so we’re on the same page:

~/.media/gd-encrypted: raw Gdrive mount (via node-gdrive-fuse)
~/.media/gd-decrypted: rclone local crypt mount which points to above folder with Media subdir (where encrypted content is) - ~/.media/gd-encrypted/Media

When I do it this way, there are folders I can tell right away don’t appear in the gd-decrypted folder, specifically that Planet Earth one and some others (I have a Plex plugin / channel that’s called FindMissing.bundle and it will find files that are missing, so this makes it easy).

If I mount the crypt directly with rclone (instead of the way I mentioned above), the folders are definitely there (and I can play from them)

So it’s weird.

Seigele · March 7, 2017, 3:40pm

Looks like there’s a lot of talk about node-gdrive-fuse in here. The biggest downside is the fact that it is no longer being developed. My main Plex server uses node-gdrive-fuse with a script that runs every 30 seconds and checks the mount status. I last tweaked around with it about two weeks and I’m happy to report that the setup is completely hands off. I can’t even remember the last time someone bugged me about any media being unavailable.

Right now the only thing I’m using rclone for is uploading media, haven’t used ACD in quite awhile because I wasn’t happy with how seeking/startup time worked. Google Drive just seems to be a superior service and that’s no fault of rclones, node-gdrive-fuse works because it’s caches the directory.

Also, for reference sake my library is just about 70tb. If you have issues with node-gdrive-fuse you need to clear the cache and try again.

I’ve setup a couple download servers over the last week and it takes me right around 30 minutes to get Sonarr/Radarr/NZBGet/node-gdrive-fuse etc setup and going without a hitch.

Stokkes · March 7, 2017, 3:47pm

@Seigele

Thanks for sharing your setup. This is the way I’m going. Just doing some preliminary tests with my test GDrive account (unencrypted), playback has been rock solid with node-gdrive-fuse.

Playback is consistent and always starts within 5-10 seconds. Fast forward/rewind work so much better than ACD.

I think what I’m going to do is have a primary and backup GDrive with unencrypted content and keep a backup of my media on ACD encrypted (that’s regularly synced). That way, if there’s ever a DMCA or my accounts get shutdown, I can just copy things back from the encrypted ACD.

Seigele · March 7, 2017, 3:57pm

@Stokkes

This is what I do, I have a throwaway gmail account with unlimited storage that is used for downloading and primary playback. It’s completely unencrypted and I think it’s the way to go as far as performance goes. I have a personal edu account that I use as an encrypted backup, alongside a secondary (throwaway) gmail account that houses a copy of the unencrypted media as well as the encrypted media. (Both primary enrypted and unencrypted account share to this throwaway so I can utilize server-side copies.) My personal edu account copies the encrypted data over to ACD,

So in short I utilize 4 different accounts and have 2 copies of my data unencrypted and 3 copies of the encrypted data.

Unencrypted Google Drive with an SSD is a dream though.

Ajki · March 7, 2017, 4:02pm

I did quite some testing with encrypted vs unencrypted ( also have gdrive unecrypted one ) and preformance wise it was on pair, no difference at all. ( beside rclone’s CPU usage when using crypt account )

Seigele · March 7, 2017, 4:19pm

What was the CPU usage like on encrypted vs not?

Ajki · March 7, 2017, 4:29pm

Around 5% on my server, but since last testing ncw did quite some optimization i see much better ram and cpu usage since last few beta’s.

p.s. My unecrypted gdrive is in use now with Plex Cloud.

Seigele · March 7, 2017, 4:31pm

Oh, gotcha. I doubt I’ll change my setup for the time being I just remember when I first started using rclone the crypted mount seemed to use more resources than using even encfs so I opted to just go without it so long as I’ve got some encrypted backups I can fall back on. I may have to do some testing and see if any of my users complain

EnorMOZ · March 7, 2017, 4:42pm

Plex Cloud via Google unencrypted has actually gone a long way. Things run pretty smooth now from my testing.

Stokkes · March 7, 2017, 4:42pm

If rclone supported a Google implementation similar to node-gdrive-fuse, I’d be more inclined, but unless you’re building crazy scripts to update your library and have to do a special dance just to avoid being banned, it’s not really something I want to get involved with.

My ideal world would be to use Google Drive encrypted with rclone and Plex without the fear of being banned. Since that’s not really feasible now unless you’re willing to put in a bunch of hack job scripts (which you have to support, maintain, ensure they dont’ break, etc.) then you’re pretty much forced to use Google + node-gdrive-fuse, unencrypted (or encrypted). This is really the fastest way to vanilla Plex + Google

jmacul2 · March 7, 2017, 4:44pm

I completely agree! Its like have unlimited local storage. The performance is unbelievable.

Ajki · March 7, 2017, 5:55pm

Yea it working great for me too, but never tested with tons of concurrent streams.