Files seem slow to open from encrypted, shared folder

[Chiefmas]

What is the problem you are having with rclone?

I should start with: can I make these reads by apps happen faster, or is that just how it is?

Hopefully this will make sense. I'm experimenting with sharing a google drive folder whose contents are encrypted by rclone (for testing, I'm sharing from my GSuite account to my personal Google account). I've gotten that working with a little hoop jumping. The shares will only be mounted read-only by anyone I share with. One thing that's odd though, file reads seems fast or slow, based on what's doing the reads.

So for instance, in the logs, as a test, I'm opening the file, this is essentially reading an archived file and decompressing it (to memory I believe). I tested this with a 86MB file, so it's not super huge. I've seet the chunking size to 20MB, figuring that this file would not take more then 3 read requests, based on how I understand the chunk size to increase. Instead, it takes a lot, this causes it to take perhaps a minute or so to get the entire file, then open it. I think the app is loading the whole file, and then decompressing it and opening it, as the app appears to hang, and then suddenly is loading the file at once, which coincides with when the last reads happen in the log.

If I just use Explorer to copy the file to my local desktop, instead of opening it with an app, the file copies in just a few seconds, which was more the speed I was looking for. So I'm trying to figure out if I can do something other then setting the cache mode to full, to speed this up. It almost feels like the app is internally chunking and sending a bunch of small requests. But if it's something else that's a known thing I can improve that would be great!

I'd like to avoid setting the cache mode to full because I saw this should cause the entire file to load to local before handing off to whatever made the request, but if this works well, I might start putting larger files on that I wouldn't want to have the whole thing pre-cache before responding.

Side question: is sharing an encrypted folder, read-only like this bad/problematic in some way?

Thank you very much!

What is your rclone version (output from rclone version)

rclone v1.52.3
winfsp-1.7.20172

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I'm going to put this in, but the command and setup seems to be working, assuming this isn't a Very Bad Thing to do:
rclone mount sec: N: --fuse-flag --VolumePrefix=\server\gdSec --dir-cache-time=15s --read-only --vfs-cache-mode off --vfs-read-chunk-size 20M  -vv

The rclone config contents with secrets removed.

[gdShare]
type = drive
client_id = [redacted].apps.googleusercontent.com
client_secret = [redacted]
scope = drive.readonly
token = {"access_token":"[redacted]","token_type":"Bearer","refresh_token":"[redacted]","expiry":"2020-08-26T01:10:59.5716148-04:00"}

[sec]
type = crypt
remote = gdShare:secure
filename_encryption = standard
directory_name_encryption = true
password = [redacted]

A log from the command with the -vv flag

https://pastebin.com/Yc5knbwG

[ncw (Nick Craig-Wood)]

Try the latest beta with VFS cache mode full.

This doesn't download the entire file first any more and I think it should help a lot.

[Chiefmas]

Ok, I'll give that a shot, thank you!

[Chiefmas]

That's proven to be interesting! There's still a hit on the initial load, but subsequent loads, even after the cache has flushed itself, and even after I stopped rclone, and restarted it, are very fast. It makes it seem like there's a little caching happening on Google's side as well, perhaps. But, whatever it is, I wasn't getting the performance boost on successive hits.

It seems like an improvement so far, but now that I do have the cache set to full, I'm a little concerned about how it will work if I do pull a large file. I get that it won't pull the whole file at once, from what you've said, but it is keeping the entirety cached at some point?