VFS Cache Handling

First, thank you for the amazing software. The development has been top notch.

Second, with vfs really stepping into its own, I wanted to see if there was a feature to be added that I think a lot of users would use.

Using rclone with mergerfs has been extremely popular, probably kick-started by @Animosity022 's setup where new writes to the merged directory write to the local disk and then is moved to the cloud for a seamless end-user/app experience.

In the past, I chose not to use VFS as my "cache" method, since my upload speed isn't great. Uploading a 2gb in the middle of the day could easily choke out my internet connection. Since working from, I had to control when my uploads happened. Enter mergerfs. I put all writes to the local disk and then manually (or cron) upload it to the cloud at night/early in the morning.

Lately, mergerfs has been extremely slow. Sometimes, I can't even list the directory out. But the underlying rclone mount is blazing fast. This has caused me to consider switching to VFS. But I still need to consider my upload bw.

Is there a way to control when VFS uploads files? For example, could I set it to cache it locally and upload at a certain time of day? This presents some operational challenges, as the cache would need to persist across reboots and rclone mount restarts, to avoid losing the data cached.

I'm also open to other ways to accomplish what I'm currently doing, but I want to stick as close to "stock" as possible, to eliminate any dependencies, hence wanting to use vfs.

Edit: I will consider using vfs with controlling it's upload bandwidth to make sure it doesn't choke my upload connection. However, this is less than ideal, as it would mean during the night, it wouldn't use all available upload bandwidth.

hi, i am sure an expert at this will stop by soon.

until then, sorry, if i am telling you something you know, that you can schedule the speeds.
https://rclone.org/docs/#bwlimit-bandwidth-spec

That's a good start. I see I can set upload and download speeds. I also see I can schedule different speeds. But I'm not sure how to combine that to schedule upload speeds and download speeds. (as far as I can tell, the schedule option applies to both upload and download. Is that right?

not sure what you mean. just set the upload and download speeds you want.

and you can set in on demand
https://rclone.org/rc/#core-bwlimit

According to the docs:

An example of a typical timetable to avoid link saturation during daytime working hours could be:
--bwlimit "08:00,512k 12:00,10M 13:00,512k 18:00,30M 23:00,off"

To me, this means the bandwidth limit being applied is upload AND download limits. I want to schedule upload limits and leave download limits off.

you can set the values to whatever you want, whenever you want.

"Either limit can be "off" meaning no limit, so to just limit the upload bandwidth you would use

--bwlimit 10M:off

this would limit the upload bandwidth to 10MByte/s but the download bandwidth would be unlimited."

Correct, which leads us back to my original statement:

So at this point, I'm unsure of how to schedule speeds, while also having granular control over which speed I get to schedule. For example, between 8 am and 11pm every day, I want to schedule upload speed to use no more than 1mbps and download to have no limit.

in the docs, there are several detailed examples

@ncw any ideas on this topic?

If I understand the docs correctly, this is the combined syntax:
--bwlimit "08:00,1M:off 23:00,off"

Correct?

This is a decent workaround, IF the local vfs cache can persist (either across rclone binary restart or system-wide restart).

in rclone speeds are always based on bytes, never bits.

I know this. Does this statement have to do with my question above? If so, I'm not connecting the dots...

well, you wrote that you wanted 1mbps .
i thought that meant 1Mbps - one Megabit per second
not 1MBps which is one MegaByte per second

gotcha, no I want 1 megabyte, or 8 megabits, roughly 15% of my upload. So that's 1M.

great, you got it figured out now

Well, sorta. I'd like to see if there are any plans to add to VFS capability and verify:

the local vfs cache can persist (either across rclone binary restart or system-wide restart).

Otherwise, vfs is not a viable replacement.

no worries, you have already summoned the rclone master.
now, it is only a matter of time...

Yes, the rclone cache is persisted across restarts of the binary or system.

1 Like

@jiru, you've asked a bunch of questions and gotten some pretty decent answers. Now do a little testing and then report back any issues you run into.

2 Likes

I'm getting the following error on my bw flag:

Error: invalid argument "08:00,1M:off 23:59,off" for "--bwlimit" flag: bad suffix 'f'

I can't see anything wrong with the syntax:

--bwlimit "08:00,1M:off 23:59,off"

Full command below:

/usr/bin/rclone mount c-vault: /mnt/rclone/cvault \
--allow-other \
--attr-timeout 1000h \
--buffer-size 64M \
--dir-cache-time 48h \
--log-level INFO \
--log-file /opt/rclone/logs/cvault-service.log \
--poll-interval 30s \
--use-mmap \
--tpslimit 10 \
--tpslimit-burst 10 \
--cache-dir=/root/cache \
--vfs-cache-mode writes \
--transfers 1 \
--bwlimit "08:00,1M:off 23:59,off" \
--timeout 1h \
--umask 002 \
--user-agent vaultapp \
--rc \
--rc-addr 127.0.0.1:5573