Cached crypted mount: moving content into it, slow or not working at all

Help and Support
1

Hi,

I’ve a cached crypted mount, but I am trying to work out the best way of adding content to it.

Here’s my rclone config:

[gdrive]
type = drive
client_id =
client_secret =
token = ***REDACTED***

[media-cached]
type = cache
remote = gdrive:Backup
info_age = 1440h
chunk_total_size = 32G
plex_token =

[media]
type = crypt
remote = media-cached:
filename_encryption = standard
password = ***REDACTED***
password2 = ***REDACTED***

[media-upload]
type = crypt
remote = gdrive:Backup
filename_encryption = standard
password = ***REDACTED***
password2 = ***REDACTED***

and Here’s my systemd unit:

[Unit]
Description=rclone FUSE mount
Documentation=http://rclone.org/docs/
After=network-online.target

[Service]
Type=simple
User=gmount
Group=gmount
ExecStart=/usr/bin/rclone mount media: /mnt/remotes/google \
   --allow-non-empty \
   --allow-other \
   --buffer-size=0M \
   --dir-cache-time=1h \
   --fast-list \
   --cache-tmp-upload-path=/mnt/storage/CloudTemp/rclone/cache-upload \
   --cache-db-path=/mnt/storage/CloudTemp/rclone/cache-backend \
   --cache-chunk-size=24M \
   --cache-chunk-total-size=100G \
   --cache-info-age=672h \
   --cache-tmp-wait-time=10m \
   --cache-writes \
   --cache-workers=8 \
   --umask=000 \
   --rc \
   -v
ExecStop=/bin/fusermount -uz /mnt/remotes/google/
Restart=always

[Install]
WantedBy=multi-user.target

I first tried uploading content with the following command:

rclone -v move /mnt/storage/staging/Backup media-upload:

Which seems to work, except the content never actually appears in the cached mount, obviously because it is totally seperate and doesn’t update the cache. I can clear the cache but that then seems to defeat the point in it.

I then tried using the following instead:

rclone -v move /mnt/storage/staging/Backup media:

but that failed to start entirely saying it couldn’t write to the mount.

So now I’m using the following:

rclone -v move /mnt/storage/staging/Backup /mnt/remotes/google/

Which should obviously be going via media: over it’s mount.

Now this works, mostly, every now and then the chtimes part fails as the cache hasn’t updated yet and i have to flush cache for that directory to work again using rclone rc cache/expire Remote=“Directory/”

And while I say it works, it’s like it’s uploading it directly rather than caching the write and uploading in the background as I’d expect.

Am I doing something wrong? Is my config awry or am I just uploading content the wrong way?

2

--cache-writes & --cache-tmp-upload-path are contradictory. Remove --cache-writes and increase the value of --cache-tmp-wait-time.

3

Ahh, it wasn’t clear from the docs that the two were exclusive. I have now removed --cache-writes and it’s actually working nicely in terms of function, not so great in terms of performance.

Moving items into the mount seems ok, but renaming, deleting etc are all really slow (as if the changes are being persisted to the remote immediately)

My expectation is that changes to the mount would be completed near instantly, then the cache workers behind the scenes work to put the remote into the state of the cache, that is what I would expect to happen.

Instead it seems something as simple as an rm or mv takes upwards of a minute to complete, I presume as it pushes to the remote while blocking behind the scenes.

Is there any way to get it to handle all changes through the offline cache?

4

You are probably hitting this issue with your lower --cache-tmp-wait-time: https://github.com/ncw/rclone/issues/2703

5

Even setting it to 8 hours doesn’t seem to change anything.

Making changes to the cached / crypted mount with offline uploading seems to persist to the remote instantly.

Could this be something to do with --vfs-cache-mode or is that another separate thing?

6

I’m confused as to what you are trying to do.

When I use rclone copy / move, I would write directly to a non cached remote.

Once the copy is done to the remote, it takes 1 minute for it to show up on a mount based on the polling interval.

7

When I do that, It never appears in the cached remote.

I was originally doing rclone -v move /mnt/storage/staging/Backup media-upload: and media-upload: is a non-cached crypt to the same path.

The cache mount never updates to show the uploaded files.

8

Can you replicate what you are talking about? I can copy a file up and in 1 minute it appears on the mount.

felix@gemini:~$ date
Tue Dec 11 09:25:10 EST 2018
felix@gemini:~$ cd /gmedia/
felix@gemini:/gmedia$ rclone copy /etc/hosts gcrypt:
felix@gemini:/gmedia$ ls
mounted  Movies  Radarr_Movies  torrents  TV  TV_Ended
felix@gemini:/gmedia$ date
Tue Dec 11 09:25:45 EST 2018
felix@gemini:/gmedia$ ls
hosts  mounted  Movies  Radarr_Movies  torrents  TV  TV_Ended
felix@gemini:/gmedia$

That’s my example.

gcrypt: is my remote.

9

Here is the same test on a cached mount:

felix@gemini:/Test$ date
Tue Dec 11 09:28:14 EST 2018
felix@gemini:/Test$ rclone copy /etc/hosts gcrypt:
felix@gemini:/Test$ ls
hosts  mounted  Movies  Radarr_Movies  TV  TV_Ended
felix@gemini:/Test$ date
Tue Dec 11 09:29:50 EST 2018
felix@gemini:/Test$
10

Hmm, that is actually working now.

I wonder if having --cache-writes was somehow stopping it from working.

cheez@talyn /mnt/remotes/google $ date
Tue 11 Dec 14:30:47 GMT 2018
cheez@talyn /mnt/remotes/google $ rclone copy ~/test_file media-upload:
cheez@talyn /mnt/remotes/google $ ls
Archives  ContainerImages  verification  VHDs
cheez@talyn /mnt/remotes/google $ date
Tue 11 Dec 14:31:50 GMT 2018
cheez@talyn /mnt/remotes/google $ ls
Archives  ContainerImages  test_file  verification  VHDs
11

When you say cache-writes, you mean vfs-cache-mode writes? You aren’t copying to a mount though so that wouldn’t have an impact. You are copying directly to a remote.

12

No i mean look at my post above, I originally had --cache-writes and offline uploading going on at the same time, which are mutually exclusive, it was that mount that was failing to update when i uploaded content.

Seems to work now i’ve removed --cache-writes.

13

That’s what I’m saying though, those mount options have nothing to do with rclone copy to a remote.

The mount itself just polls every 1 minute by default and checks for changes.

Have any other options on the mount isn’t relevant as we are doing a rclone copy to a remote and not the mount.

14

You misunderstand I think

I’d like to be able to move files to the mount, at which point the cache is updated immediately and uploaded in the background, @darthShadow solved my issue and got it working the way I wanted using offline uploading, but it’s good to know I can now use rclone copy for other things if I need to now. Before that wasn’t working but after @darthShadow’s fix to the mount it is, I suspect that somehow the conflicting options to the mount was breaking the mounts polling, but now it isn’t and everything is happy

I’d mark this thread solved if I knew how. :slight_smile:

15

Misunderstanding takes two people in a conversation to end up at the same place. If I’m not getting what you are trying to do, please set me straight so I can benefit as well.

I’ve spent the time, read through your thread and tried to offer a solution based on how I’m interpreting your requirements.