Need help to understand the process of vfs-cache-full

Help and Support
1

What is the problem you are having with rclone?

Want to test with a download directly to the encrypted mount folder with vfs-cache-full enabled.

What is your rclone version (output from rclone version)

rclone v1.53.3

  • os/arch: linux/amd64
  • go version: go1.15.5

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu Desktop 18.04 LTS

Which cloud storage system are you using? (eg Google Drive)
Googledrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Systemd:

[Unit]
Description=RClone Service
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/home/plex/.config/rclone/rclone.conf
ExecStart=/usr/bin/rclone mount Googlecrypt: /home/plex/mnt \
--allow-other \
--dir-cache-time 1000h \
--log-level INFO \
--log-file /home/plex/logs/rclone.log \
--poll-interval 15s \
--umask 002 \
--cache-dir=/home/plex/cache \
--vfs-cache-mode full \
--vfs-cache-max-size 100G \
--vfs-cache-max-age 120h \
--bwlimit-file 16M
ExecStop=/bin/fusermount -uz /home/plex/mnt
Restart=on-abort
User=plex
Group=plex

[Install]
WantedBy=default.target

The download works just fine directly to the encrypted folder , so far so good and I can see the content is located at gdrive when I check with rclone browser.
But after the download, I can see my local drive is getting filled up.
When I check my cache folder for my encrypted drive, I can see that the downloaded files are present in the cache?

Plex plays the content just fine. But if I clear the downloaded content, my drive is still filled up.
So I read that I could delete the cache folder, so I tried that and the drive was freed up.
Then noticed the content I just downloaded, was also deleted from the encrypted drive which is mounted locally to my NAS?

What am I doing wrong here. I thought vfs-cache-full worked when you do a 1st stream of movie1, then it will store it in the cache, and if I then start up the movie again , it don't need to redownload from the gdrive.

So I need a bit of help trying to understand what's going on :slight_smile:

Thanks :slight_smile:

2

You shouldn't be manually deleting anything.

You've told rclone to keep 100G of cache data in the default cache dir location for 120 hours. So it'll use that space until it's filled and then remove items that are past the age by first in, first out assuming they are not actively being used.

By deleting things manually, you are confusing rclone as it see those things disappear.

Plan to have the cache the right size for your scenario as you want enough space for the total size of all the files you plan to stream concurrently. So if I had 5 streams of 50GB files, I need to make sure I have 250 GB of space if they are all going concurrently.

1 Like
3

Thank you very much for the explanation Ani.

Can you maybe also elaborate what happens in "technical" terms of what is going on with vfs-cache-full running vs Systemd without vfs-full-cache?

[Unit]

Description=RClone Service

After=network-online.target

Wants=network-online.target

[Service]

Type=notify

ExecStart=/usr/bin/rclone mount --allow-other --dir-cache-time 1000h --log-level INFO --log-file /home/plex/logs/rclone.log --bwlimit-file 16M Googlecrypt: /home/plex/mnt

ExecStop=/bin/fusermount -uz /home/plex/mnt

Restart=on-abort

User=plex

Group=plex

[Install]

WantedBy=multi-user.target

I just tried both:

Vfs-cache-full = content it saved to encrypted mount without a problem.

Without = I cant download to my mnt folder even though it has read write.

So if cache is getting filled up - and its clearing with first in - first out principle . Will it then automatically delete the content from the gdrive as well?

4

The cache mode doesn't remove anything from your remote as it's just used for caching.

With no cache mode, you stream the file from the mount/remote via memory.
With full, you download the parts of the file you need. It uses sparse files so it puts a shell if the file and as it streams, puts more pieces inside the file with the final outcome being the full file if you play the whole thing.

I am not sure what you mean by 'download to my mnt' folder.

1 Like
5

Thanks :slight_smile:

I was running rclone with vfs-cache-full. I want to test a download via download client directly my encrypted mount folder /home/plex/mnt which is my gdrive decrypted on my NAS.
After the download, I could see the files where present at my googledrive if I used 3rd party software to see my encrypted content. ( Rclone Browser ). Seems like that I could download files directly to the specific folder.

When I run systemd without the vfs-cache it doesn't seem to work via the download client, but I can create files directly by typing via ssh touch test.txt in /home/plex/mnt . But via the download client , I cant download anything to /home/plex/mnt .

Sorry im not so good explaining.

6

I'm sorry but still not quite following.

You are running something to download a file directly to your mount? I don't use rclone browser either so don't really know what it is / does.

You'd need to explain the steps on what you are doing and create a debug log and share the full debug log.

You normally don't need a cache mode to write to a mount unless it gives an error about it.

felix@gemini:/GD$ cp /etc/hosts .
felix@gemini:/GD$ ls -al
total 1
-rw-rw-r-- 1 felix felix 225 Dec 21 13:37 hosts
-rw-rw-r-- 1 felix felix 243 May 19  2019 mounted
felix@gemini:/GD$ rclone ls gcrypt:hosts
      225 hosts

And it uploads the remote. That's a small file so it's pretty instant. If it's a large file, the copy would take longer.

7

I will try first without posting logs, I don't really get any errors. But let me try explain my current setup :slight_smile:
It may sound a bit confusing, but its actually working fine :slight_smile:

Rclone config:
Googledrive: Google
Encrypted: Googlecrypt
From your :
https://github.com/animosity22/homescripts/blob/master/systemd/gmedia.service I altered to my needs.:
Plex media folder: /home/plex/media (your Gmedia )
Mount folder: /home/plex/mnt ( your GD )
Download folder: /home/plex/download
My "move" folder : /home/plex/move (your local folder )
3rd party rclone browser : http://martins.ninja/RcloneBrowser/ to see what's present in googledrive encrypted folders via rclone.conf

Mergerfs: /usr/bin/mergerfs /home/plex/move:/home/plex/mnt /home/plex/media -o rw,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=auto-full

Today I use sonarr, radarr to get the things I want. To ease up my downloads/uploads . I use mergerfs to have the folders mirrored at my /home/plex/media .
In that folder it shows my gdrive decrypted:

4K
Animations
Movies
and etc.

So when radarr , sonarr is finished , it automatically rename the content and copy it directly to the media folder with renaming it plex friendly with folders and file names.
When the content is present in the media folder , I can see everything in the "move" folder .

4K/Movie4K.mkv
Animations
Movies

At night an upload scripts moves everything from the /home/plex/move folder to gdrive encrypted. So everything is placed in the right folders, uploaded the right places etc. Works like a charm. So my Plex see newly content download in the respective folders and I can play it locally if I want.

This setup was without vfs-cache-full.

With vfs-cache-full enabled.

I run the exact same setup with the exception of I want to download directly to /home/plex/mnt folder to save HD drive space with a large 4K movie since drive is not that big.
Its seems to be working fine, but after the download I thought "Hey downloading directly to /home/plex/mnt" saves me tons of space since its saved directly to the gdrive. But I saw that my harddrive was filling up so fast.
Then I tried to delete the item I just downloaded, the space was still filled up on the drive.
Then I tried to delete the cache folder and then I saw my space was getting good again.
But after I deleted the cache - the content was gone from gdrive.?

I know it may sound really confusing still trying to make different setups , and maybe smarter setups. But no there yet :slight_smile:

8

i would not use that version, given that the author of it, does not recommend using it.
this is the one to use https://kapitainsky.github.io/RcloneBrowser/

1 Like
9

Thank you :slight_smile: gonna try that one :slight_smile:

10

With any cache mode, it writes locally first and then uploads it so you have to have space to store it before it's uploaded.

1 Like
11

Ahh :slight_smile:

So I still have to use my upload script with cache mode enabled ?

12

Copying directly to the mount doesn't make sense to me as I upload directly to the remote instead from my local storage.

1 Like
13

I thought it was copying to the remote via the mount. :slight_smile:

Thanks for all your help Ani , so much appreciated

14

No.
Saving to /home/plex/mnt does not mean it gets saved directly to gdrive as it gets downloaded.
It saves the files local and automatically uploads/writes the file to gdrive. Hence you would still need space on your local for the large files.

You dont need to run a separate upload script.

1 Like
15

@St0rm

Thanks for clarifying. Just curious - you say I don't need an upload script. When does the system upload to gdrive? Any specific times. ?

Thanks and Merry X mas

16

if you want you can have two mounts.

one mount optimized for streaming using vfs.
one mount for copy/sync/move not using vfs

17

Hi @asdffdsa

Can you elaborate ? :slight_smile:

At the moment I have vfs cache mode full enabled. That system will automatically upload right? How fast does it upload after I have something downloaded?

Because im also using an upload script which is running at night .

A bit confused

18

well,
as @Animosity022 mentioned, he does use a mount to upload.

as @St0rm mentioned, two limitations for a mount with vfs when used for uploading.
does not mean it gets saved directly to gdrive
and
you would still need space on your local for the large files.

so to upload, i use a second mount without any vfs cache.
in that way:

  • no delay when uploading files. files are saved directly to cloud.
  • no additional local space for large files.
1 Like
19

Thanks @asdffdsa

I think I will go for my regular setup without the vfs-cache-mode full since my NUC harddrive isn't that big.
Need a bigger drive to have that part enabled.

20

which part?

not sure you know that:
vfs full using sparse files, so when streaming media with plex, it only download chunks as needed, not the entire file.