Need help adding a cache to an existing crypt setup

I currently have an existing gdrive setup with a crypt mount. I am trying to add a cache between my current gdrive and my current crypt (https://rclone.org/cache/#cache-and-crypt). Since my files are already stored under a crypt folder on gdrive I assume my cache should point to 'gdrive:/crypt' pointing to my old folder and my crypt would point to the cache with no folder, 'gcache:'? When I set it up it stated not using a folder was not recommended. I am not sure if it is just to prevent an unorganized mess or if there are other reasons. I am afraid I will lose my encrypted data if I mess this up (I have backups and other items I can't lose and have no drives large enough to store it all as a second backup), so this is my attempt at "measure twice, cut once" to prevent a gigantic mistake.

Please let me know if you see any issue with the config listed below, or if you have any other recommendation or words of warning.

New Config

[gdrive]
type = drive
scope = drive
token = {...}
client_id = ...
client_secret = ...

[gcrypt]
type = crypt
remote = gcache:
filename_encryption = standard
directory_name_encryption = true
password = ...
password2 = ...

[gcache]
type = cache
remote = gdrive:/crypt
plex_url = ...
plex_username = ...
plex_password = ...
chunk_size = 10M
chunk_total_size = 10G

Old Config

[gdrive]
type = drive
scope = drive
token = ...
client_id = ...
client_secret = ...

[gcrypt]
type = crypt
remote = gdrive:/crypt
filename_encryption = standard
directory_name_encryption = true
password = ...
password2 = ...
[gdrive]
type = drive
[gcrypt]
type = crypt
remote = gcache:
[gcache]
type = cache
remote = gdrive:/crypt

This (your current) is correct and fine to use.
It would make more sense to me to use the crypt folder with the crypt remote rather than the cache but... effectively that ends up happening here anyway. I'd change it for your own organization's sake and so you don't get confused.

Maybe go ahead and arrange (cut&paste) the remotes in the "proper" order too (gdrive<--gcache<--gcrypt). This has no technical significance, but when they are laid out in the order the traffic flows it makes it much easier to check yourself for mistakes.
(edit: see bottom of post for my suggestion).

As far as I know there is no technical reason why you need to use folders with either crypt or cache, but I absolutely do recommend having encrypted files in their own folder - just because otherwise you can end up with encrypted and unencrypted files mixed. This turns messy real quick and there is little benefit to such an arrangement. (this is basically how you have it now, so that's fine).

As for losing data, you don't have to worry very much. The only way that could realistically happen here is you lose your crypt key (So keep a hard backup somewhere safe!!). It's not like a harddrive where you can overwrite stuff by a wrong partitioning or something. A wrong setup will just mean it doesn't work - or works in the wrong way - until you fix it. You would still have to actually delete your files to lose them. Do be careful about sync commands though... since they are allowed to delete for you, a badly worded sync can delete something you didn't intend. Always do a test with --dry-run until you feel very comfortable with how it works.

As a last point (Even though this is a bit of a separate topic), why do you want to use the cache? It has benefits, but also downsides. I advice you to not use it "blindly", but rather because you know it will solve some spesific problem or limitation you have. A lot of the advanced users here (myself included) have decided to forgo the cache. I can elaborate if you wish - but knowing your spesific use-case and reason for wanting the cache would help...

Effectively the same setup, just slightly cleaned up organization (order + folder swapped to crypt):

[gdrive]
type = drive
[gcache]
type = cache
remote = gdrive:
[gcrypt]
type = crypt
remote = gcache:/crypt

cache applies to whole drive, but crypted files are in a separate folder. If you ever want to use non-encrypted space on the gdrive, then you just access the gcache or gdrive remote directly instead - and put your files anywhere aside from the crypt folder.

@thestigma Thank you for the detailed reply. I originally had it the way you did, but figured caching would then happen on the whole drive instead of just the files in the crypt, but after reading your post I don't think that is a huge concern even if it did that.

The reason I am adding a cache is due to a recommendation on my previous post Buffering on Direct Play. I had cache a long time ago when I first used rclone on my mac, but removed it as I didn't add any benefit at that time. I have since switched to a linux based NAS and since I have done that, I have been having issues with plex playing files. After digging around it appears to only be an issue with direct play from my rclone files. If you have a different suggestion to address this without caching, I am open to that.

Playback usually has much more to do with the software that is requesting the data than about rclone.
If the software behaves in a network-friendly way (primarily in not closing and reopening the connection each time it fetches data, as that is slow on most cloud-backends) then a cache should not be necessary for any streaming. I don't use one, but I just play stuff in VLC directly in Windows. It's only big benefit at that point is literally to be a read-cache - which certainly can have it's uses especially for people with low bandwidth, but often isn't worth the drawbacks otherwise.

(The cache backend's drawbacks are mostly connected to it being old and not updated for a long time - and the idea is that the VFS system will take over this functionality eventually.)

I unfortunately have no direct experience with direct-play, and thus I don't know it's behavior. Is that for the shield device?

If you have the option to do so, then often best and most direct solution is to either try out alternative playback software, or sometimes fix it in the software settings (although the relevant things in this regard are rarely options). However, I understand that you may not have any options on your shield for example.

And in that case - the cache can indeed help remedy the problem in many cases. Mainly by aggressively pulling down more of the file to local storage long before you actually access it. It won't solve the fact that the player is opening and closing the file - but since that at least happens locally to the cache it's not so slow that it will cause stuttering - and more data can be sent before the player-buffer runs dry.

So yea, in your case it seems like you may indeed need it.
The more workers and the larger chunk-size you use in the cache - the larger effective safety-guard you should get against the spesific problem you are talking about. So maybe play around with those a bit and set them high enough that you never get stuttering in practice, but not so overkill that you become wasteful with how much data you pre-cache each time you access a file.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.