Rclone is not using specified cache chunk path after using vfs cache

Help and Support
1

What is the problem you are having with rclone?

cache-chunk-path is not using the mentioned directory

Run the command 'rclone version' and share the full output of the command.

rclone v1.66.0

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 6.5.0-1021-oracle (aarch64)
  • os/type: linux
  • os/arch: arm64 (ARMv8 compatible)
  • go/version: go1.22.1
  • go/linking: static
  • go/tags: none

Are you on the latest version of rclone?
Yes

Which cloud storage system are you using? (eg Google Drive)

One Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount odrive: /plex --cache-dir /home/nirzak/data/plex --vfs-cache-mode full --dir-cache-time 1h --allow-other --buffer-size 512M --umask 002 --vfs-read-chunk-size 512M --onedrive-chunk-size 120M --vfs-read-chunk-size-limit off --vfs-cache-max-age 300h --vfs-cache-max-size 15G --cache-chunk-path /dev/shm --cache-chunk-size=40M --cache-chunk-total-size 4G --cache-info-age=100h --cache-workers=6 --poll-interval 1m --use-mmap --tpslimit 10

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[dbox]
type = dropbox
token = XXX

[obox]
type = onedrive
client_id = XXX
client_secret = XXX
token = XXX
drive_id = XXX
drive_type = business

[odrive]
type = onedrive
token = XXX
drive_id = XXX
drive_type = business

A log from the command that you were trying to run with the -vv flag

2024/05/13 04:13:07 DEBUG : vfs cache: data root is "/home/nirzak/data/plexcache/vfs/obox"
2024/05/13 04:13:07 DEBUG : vfs cache: metadata root is "/home/nirzak/data/plexcache/vfsMeta/obox"
2024/05/13 04:13:07 DEBUG : Creating backend with remote "/home/nirzak/data/plexcache/vfs/obox/"
2024/05/13 04:13:07 DEBUG : fs cache: renaming cache item "/home/nirzak/data/plexcache/vfs/obox/" to be canonical "/home/nirzak/data/plexcache/vfs/obox"
2024/05/13 04:13:07 DEBUG : Creating backend with remote "/home/nirzak/data/plexcache/vfsMeta/obox/"
2024/05/13 04:13:07 DEBUG : fs cache: renaming cache item "/home/nirzak/data/plexcache/vfsMeta/obox/" to be canonical "/home/nirzak/data/plexcache/vfsMeta/obox"
2024/05/13 04:13:07 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

Some more info. I want to try to use disk as my vfs cache directory and my ram to store my read chunk caches. That's why I have used both cache-dir and cache-chunk-path. but on /dev/shm I can see that no data is storing over there. looks like it is only using the cache-dir parameter not the cache-chunk-path parameter. Thanks

2

Missing the debug part that actually shows the parameters being passed and it starting.

Can you include the full debug log?

3

Any —cache-…flags belong to cache remote. You are not using it - and you should not as it is deprecated ages ago. You can safely delete all these flags as they are ignored anyway with your configuration

4

That's wrong.

rclone mount

is part of the regular mount command and that is what the OP is asking about.

5

I must disagree:) It is correct...

--cache-chunk-path
--cache-chunk-size
--cache-chunk-total-size
--cache-info-age
--cache-workers

And this is exactly what OP is asking about and is not using cache remote:

6

OP wants to make sure

--cache-dir string

is set to where the OP wants to the cache chunks/storage to be written too.

--cache-dir does not belong to the cache remote.

image
image1035×384 55.1 KB

7

--cache-dir works:) OP wants to use both at the same time for reason explained above.

Maybe let @nirzak explain. English is not my first language so maybe I do not understand it correctly.

8

So we're in agrement then as this is wrong?

image

1 Like
9

OK. I do agree on --cache-dir... This one can be confusing... But it is called what it is called. Maybe for some legacy reasons...

10

Hi yes. @kapitainsky was right. I am trying to use both. I am trying to store my vfs caches to --cache-dir and trying to store my buffer and read/chunk caches to --cache-chunk-path. But is there any limitation? Can't I use both? I have read the documentation of rclone. But I am still a little bit confused if I can use both or not. Thanks a lot for your time guys.

11

rclone cache backend is original attempt to provide caching for other remotes. It is deprecated and for years nobody is working on it.

It has been replaced by VFS caching which is under active development.

I have to admit I have never used cache so my expertise here is limited but my understanding is that you should use one or another - cache or VFS.

What is exactly problem you are trying to solve? VFS provides all cache functionality and more.

12

Oh ok I got it. That means I can use only one of them at a time. Basically I want to use to RAM for temporary caches or ongoing read buffers instead of my disk. How can I do this in case of VFS cache? I have 18GB RAM in PC. So I want to use it for VFS cache. Thanks.

13

can try using a ramdisk

14

Linux will use all available RAM anyway. 18GB is not a lot but for sure some part will be used for fuse caching. I do not think there is any benefit in tweaking anything unless you have very special and demanding workflow.

15

Yeah this is not much but I have a lot of memory free for now. almost 80% of it. And I have noticed it didn't use that much memory even if I use --buffer-size 2G or so.
The problem is I was trying to scan my music library from onedrive to my plex and have noticed that it's taking too much disk IO. I think that might be because it's using the disk as a cache that's why. So, trying to store some portion of it in the memory. That's why I was using /dev/shm partition since it's the RAM's partition. Any solution if we can do that using rclone? like storing the temporary read buffers or the cache to the memory or so? thanks

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.