RAM usage only 1mb on single mount (vfs-full-cache option) - it's possible?

Help and Support
1

Could you please help me with below mount command. Does anyone know why is RAM filled with around 40Mb on single mount? I'm trying to mount 2000 similar commands on same VPS. It's working, but my RAM is full on 99% and swap file bigger than 80gb.

I would like to use only 1Mb of RAM usage per single mount (i mean RAM usage when mount don't do anything and just "waits" for tasks). Any help would be appriciated.

My current mount command:

rclone mount dropbox: /mnt/dropbox/ --cache-dir /mnt/cache/ --vfs-cache-max-age 168h --vfs-cache-mode full --vfs-cache-max-size 2M --vfs-write-back 5s --vfs-read-chunk-size 64k --vfs-read-chunk-size-limit off --attr-timeout 1s --dir-cache-time 1h --buffer-size 70K --no-checksum --no-modtime --vfs-read-wait 0 --vfs-write-wait 0 --max-read-ahead 0 --multi-thread-streams 70 --use-cookies --no-check-certificate --drive-pacer-min-sleep 1ms --use-mmap --vfs-cache-poll-interval 30m --read-only --multi-thread-cutoff 0

Keep in mind that is dropbox mounted through crypt option.

2

Hi FoxyFox,

Your mount command will keep a full directory listing of all the files in your mounts in RAM due to --dir-cache-time 1h and the default --poll-interval 1m.

Each directory entry takes roughly 2K, so the mount taking 40MB RAM probably has 20,000 files in it. All your mounts probably have 40M files in total (80G/2K).

I guess you can make your mount use minimal RAM by using:

--dir-cache-time=0s --poll-interval=0

You may however want to make it more responsive by caching directory listings in use in e.g. 1 minute, that is:

--dir-cache-time=1m --poll-interval=0

The mounts may read the directory listings at startup, so perhaps you need to start them with some delay in-between.

More information here:
https://rclone.org/commands/rclone_mount/#vfs-directory-cache

3

2000 mounts at once? That is ambitious!

Rclone will use less RAM if you use the rc to mount multiple mounts in one rclone process.

Also have you investigated the combine backend - maybe that could help?

If use of each of those 2000 mount commands is only used intermittently, then you'd be better off using autofs to mount each rclone mount when it is in use only. This is a bit fiddly to set up but it will work a lot better than starting 2000 mounts.

4

@ncw Good points!

Do you know if this will be reducing RAM usage:

Do you know the impact of --vfs-cache-max-age, does rclone keep a list of cached files in RAM?

5

Yes it does keep a list of files in RAM. I'd set --dir-cache-time to lower, but not 0 as that will cause lots of extra work for rclone fetching the directory listing every time. Maybe --dir-cache-time 60s might be a sensible minimum. Setting --poll-interval 0 probably doesn't make too much difference on memory as all it does is flush things from the dircache, but having 2000 background polls running will use lots of CPU.

On further though autofs is definitely the answer to this problem. That is why it was invented so you could have a lab/research group/university full of NFS mounts and not have to have all of them mounted all of the time.

6

Ahh, thanks for clarifying. I wrongly thought it triggered an immediate item update possibly extending cache expiry.

7

I've always used --poll-interval 0, because the data in my read-only media mount never changes from within. Memory usage tops out at around a gig, but of course my use-case is entirely different :slight_smile:

8

It's much less RAM usage now. Thank you for all different solutons.

1 Like
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.