VFS Cache manual options?

Help and Support
1

What is the problem you are having with rclone?

Migrating server and want to manually force whatever is in cache to upload ahead of time. I searched and didn't find the command, but assume this is probably available? Can I check what needs to be written and force it manually so that I can complete my server migration?

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-52-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.6
  • go/linking: static
  • go/tags: none

Are you on the latest version of rclone? You can validate by checking the version listed here: Rclone downloads
-->

Which cloud storage system are you using? (eg Google Drive)

Google Drive

2

hi,
what cache? what command are you running, what is the redacted output of the config file?

by default, after five seconds, rclone mount uploads files from the vfs file cache to the remote
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)

3

My concern is anything that is "local" and not copied to Gdrive yet I want to force before I migrate my server. Here is my config.

--allow-other
--dir-cache-time 5000h
--log-file /home/plex/log/rclone.log
--log-level NOTICE
--poll-interval 10s
--umask 002
--rc
--rc-addr 127.0.0.1:5574
--rc-no-auth
--cache-dir=/home/plex/rclone-cache
--drive-pacer-min-sleep 10ms
--drive-pacer-burst 200
--vfs-cache-mode full
--vfs-cache-max-size 1500G
--vfs-cache-max-age 5000h
--vfs-cache-poll-interval 5m
--bwlimit-file 32M

4

if you add a file to the mount, rclone uploads it to the remote.

take a look at the log file.
2022/11/04 11:23:29 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

uploading 0 - no files are in the process of being uploaded
to upload 0 - no files need to be uploaded.
in use 0 - no files are in use/locked, and thus not able to be uploaded.

5

I should have asked if there is stuff "to upload" is there a way to force it to upload from the command line?

6

not that i know of.

is there a reason you need that feature?

7

So I don't have to wait for it to upload. right now based on my config it could take 5000 hours?

8

not 5000 hours, 5 seconds
from the docs
--vfs-write-back duration Time to writeback files after last use when using cache (default 5s)

once a file is in the cache, after 5 seconds, rclone starts to upload it or queues it for upload.
as rclone transfers four files per second, until there are no more files to transfer.

and --bwlimit-file 32M can reduce the upload speeds.
i think that can be tweaked on the fly using https://rclone.org/rc/#core-bwlimit

are there files in the cache that have not uploaded yet?

9

here is a log snippet,

--dir-cache-time=5000m and --vfs-cache-max-age=5000m
those flags do not affect when rclone starts to upload a file.

at 14:34:36 - i copy 7za.exe to the mount.
at 14:34:42 - 5 seconds later rclone starts to upload the file.

2022/11/04 14:34:25 DEBUG : rclone: Version "v1.58.1" starting with parameters ["C:\\data\\rclone\\rclone.exe" "mount" "hetznersbox01_sftp_crypt:" "b:\\rclone\\mount\\hetznersbox01_sftp_crypt+cache" "--rc" "--rc-addr=localhost:6011" "--vfs-cache-mode=full" "--dir-cache-time=5000m" "--vfs-cache-max-age=5000m" "--cache-dir=d:\\rclone\\cache\\hetznersbox01_sftp_crypt+cache" "--log-level=DEBUG" "--log-file=.\\\\log.mount.hetznersbox01_sftp_crypt+cache.txt"]
2022/11/04 14:34:36 DEBUG : /test/7za.exe: CreateEx: flags=0x502, mode=0700
2022/11/04 14:34:42 DEBUG : test/7za.exe: vfs cache: starting upload
2022/11/04 14:34:44 DEBUG : test/7za.exe: md5 = 53cdab251648ae92eaffff0df0427f4f OK
2022/11/04 14:34:44 INFO  : test/7za.exe: Copied (new)
2022/11/04 14:34:44 INFO  : test/7za.exe: vfs cache: upload succeeded try #1
2022/11/04 14:35:27 DEBUG : vfs cache RemoveNotInUse (maxAge=300000000000000, emptyOnly=false): item test/7za.exe not removed, freed 0 bytes
2022/11/04 14:35:27 INFO  : vfs cache: cleaned: objects 1 (was 1) in use 0, to upload 0, uploading 0, total size 1.099Mi (was 1.099Mi)
10

Thank you for the clarification. I think I had a switch prior that held it local for a longer period of time, but to your point, I do not have any files waiting to be written. You have been helpful. It's been a while since I've played with the config and understood all the variables.

11

well, it is confusing,
as there are two types of rclone cache and each cache has its own set of flags.

let me know if this link is useful.
rclone has two caches

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.