Google Drive Crypt and Cache

Help and Support
1

I've been testing a setup for media storage on a spare Ubuntu 18.04 box (actually a virtualbox guest)

I've downloaded and setup the latest version of rclone, with what I believe is the standard gdrive->gcache->gcrypt and then mounted that too a local directory, where I can dump files, which rclone encrypts and uploads in due course.

However the issue I'm seeing is that the files I dump into the local mounted directory, and only copied into it at google-drive upload speed. What I was thinking I would get is copy into the local drive at "local" speed, the rclone sorts out the encryption and upload in the background - hence cache.

Is my understanding wrong or should it work how I expected??

2

What version are you running?
What mount command are you using?

Sounds like you want to use write cache mode, which is documented here:

https://rclone.org/commands/rclone_mount/#file-caching

3

Okay, well from you answer I guess my exceptions were correct. So I guess I goofed the setup somewhere. I have 3 "drives" setup in the config as follows

[gdrive]
type = drive
client_id =
client_secret =
token = xxxxx
root_folder_id = xxxx

[gcache]
type = cache
remote = gdrive:/media
info_age = 1h0m0s
chunk_total_size = 4T
db_path = /mnt/4tb/rclone/cache-backend
chunk_path = /mnt/4tb/rclone/cache-backend

[gcrypt]
type = crypt
remote = gcache:/crypt
filename_encryption = standard
directory_name_encryption = true
password = xxxx
password2 = xxxx

I then use the following command to start my mount

rclone mount gcrypt: /mnt/gdrive --allow-other --allow-non-empty --cache-db-purge --cache-workers=5 --cache-tmp-upload-path=/mnt/4tb/rclone/tmpupload &

I am open to any advice, looking at the stuff you linked to I would guess adding --vfs-cache-mode full maybe what I am looking for.

4

is somewhat bad as it allows for over mounting and hiding things. I would not use that.

downloads the entire file for reads and writes and probably not what you want. You probably just want writes.

I personally do not use the cache backend as it adds unneeded complexity in my setup.

What version are you running as well?

5

Sorry forgot that bit

rclone v1.50.2

  • os/arch: linux/amd64
  • go version: go1.13.4
6

Is that before it allows access (on a read), or would that be seamless in the background, one location I use has a flakey 4G internet connection, so background cache of files once accessed would be good.

7

It's documented on the link I shared:

https://rclone.org/commands/rclone_mount/#vfs-cache-mode-full

In this mode all reads and writes are buffered to and from disk. When a file is opened for read it will be downloaded in its entirety first.

The cache backend system is completely separate/different and has an offline uploading feature:

https://rclone.org/cache/#offline-uploading

That has a few other items to configure on that.

8

So the way I read that I will have to wait for the entire download before accessing the file, which is not ideal ,if I used the vfs-cache-mode-full.

9

If the goal is to make writes faster, vfs-cache-mode writes should suit your needs as it writes locally and uploads after.

10

Thanks, I'll give that a whirl, if I add that to my mount command and leave the rest the same, how would you expect reads to be handled??

Well I tried it, however it first file copies like normal, however at the end of the copy it appears to stall, I'm not sure if it's waiting for the cache to catchup or it's just stalled

rsync --progress -r /mnt/4tb/TV/'Big Little Lies' /mnt/gdrive/TV
sending incremental file list
Big Little Lies/Season 01/Big Little Lies - S01E06 - Burning Love.mkv
5,472,747,201 100% 69.67MB/s 0:01:14 (xfr#1, to-chk=8/12)
Big Little Lies/Season 01/Big Little Lies - S01E07 - You Get What You Need.mkv
32,768 0% 95.24kB/s 19:31:12

11

What's your mount command you are using?

12

rclone mount gcrypt: /mnt/gdrive --allow-other --allow-non-empty --cache-db-purge --cache-workers=5 --cache-tmp-upload-path=/mnt/4tb/rclone/tmpupload --vfs-cache-mode writes &

Actually i've already removed --allow-non-empty aswell.

13

If you want to use the offline feature in the cache backend, do not use the vfs-cache-mode writes. You should just stick with the offline uploading that you have configured with cache-tmp-upload-path and that would be fine.

I assume the goal was to write locally, return back to the OS and let rclone handle uploading later, which is the workflow and how the cache backend offline uploading works.

14

Okay, I've been playing, but not really got any further. I've been trying some smaller files, and I can confirm that when I copy files into the rclone mount, the first file copies fine, but then the second file waits for the first to upload, and this follows on the line. It's as if the cache has an upload queue of one file. I don't think it's buffer\cache size related, because even if I use lots of small files the behaviour is the same.

My current command line is

rclone mount gcrypt: /mnt/gdrive --allow-other --cache-db-purge --cache-workers=10 --cache-tmp-upload-path=/mnt/4tb/rclone/tmpupload -vv

15

That should not be how it works.

Can you run the mount in debug -vv and share the log and reproduce the issue?

16

I saved a file so that I could attach it here, but doesn't look like an option, is there a way I can post it without literally quoting 100s of lines ??

17

You can link to any Google Drive / etc or something like pastebin too.

18

Okay try this

rclone.dump

19

I just captured another log, in this one a file is copied to the local drive, then rclone is deleting it don't know why. if you look in the directory it actually there for a few seconds etc.

rclone.dump2