Write local and delay upload?


#1

I have got things working nicely with this service:

[Unit]
Description=rclonemount
Wants=network-online.target
After=network-online.target

[Service]
Type=notify
Environment=RCLONE_CONFIG=/mnt/localstorage/rclone.conf
ExecStart=/usr/bin/rclone mount remote:/Media /mnt/gcache
–vfs-cache-mode=writes
–vfs-read-chunk-size 128M
–vfs-read-chunk-size-limit off
–cache-dir=/mnt/stagingstorage/rclone
–cache-tmp-upload-path=/mnt/stagingstorage/rclone_tmp
–cache-chunk-path=/mnt/stagingstorage/rclone_chunkpath
–cache-db-path=/mnt/stagingstorage/rclone_db_cache
–cache-tmp-wait-time=1h
–allow-other
–bind 5.9.236.19
–buffer-size 512M
–dir-cache-time 72h
–drive-chunk-size 32M
–log-level INFO
–log-file /mnt/localstorage/rclone.log
–umask 002
–config /mnt/localstorage/rclone.config
ExecStop=/bin/fusermount -uz /mnt/gcache

[Install]
WantedBy=multi-user.target

I’m trying to get writes to happen locally to free up the UI on applications and stop hanging, it looks like this is writing to a cache and uploading but the cache doesn’t appear to be clearing down after it has been uploaded.

Can someone sanity check my params?


#2

You are using both the vfs cache backend for writes and the cache tmp upload. I’m guessing your files are being written to the vfs first perhaps and not sure how that interacts with the cache backend.

Do you have any logs for what you are seeing and what you are expecting?


#3

With --vfs-cache-mode=writes, rclone will attempt to automatically upload the file in the background as it’s writing. So if you’re using an app like Sonarr, etc, when it moves the file into /mnt/gcache, rclone will immediately attempt to start uploading this file.

The VFS doesn’t have a similar setting to the cache backend upload interval. If you have --vfs-cache-mode=writes, it attempts to write immediately to the remote.

This is likely the reason you are seeing hangs in your UIs. I’ve seen them too.

You have a few choices - Use the cache backend exclusively without --vfs-cache-mode=writes and all writes will go into your tmp_upload_path folder.

However, as someone who has extensively used Cache and optimized VFS settings, I strongly recommend using VFS + mergerfs (for local storage) + a script to automatically move files from local storage to your remote every 1h.

Rclone even provides an option to move only files that have been modified in the last hour, so you’re just uploading files that have been fully written. Go read the docs on --min-age=X

HTH


#4

By cache backend - are you referring to https://rclone.org/cache/ or the cache options that I have supplied on mount? This is one of my biggest confusion points when reading responses from people, I can never tell if they mean rclone cache or the cache parameters :confused:

It does look like my vfs directory is being cleared down now I’ve left it for a day or so, doesn’t seem to reflect the 1h that is set on the launch parameters…


#5

Yeah, it’s a little confusing as there is a cache backend as you linked above.

The stock vfs backend also has some cache type abilities in term of the vfs-cache-mode that you saw, which you used for writes. That’s not related to the cache backend at all and has it’s own timing as it copies files up immediately and keeps them as “–vfs-cache-max-age” is how long something is kept based on how it is being used.

If things are working now, you can probably just use vfs-cache-mode writes and remove the cache backend from the whole config and use it that way.

You get those hangs due to the way it copies the files to the remote immediately, which is why a lot of folks use unionfs/mergerfs for a local storage and rclone move to upload at night.


#6

I’ve used --vfs-cache-mode=writes for a few months until i recently swtiched to MergerFS. I had the same problems you describe. Apps UI would hang, etc likely due to I/O on the mount point. The reason I say this is I can have a full rclone move operation going outside the mount and there are no UI slowdowns. I think this is just related to FUSE/Rclone

Your best bet to remove the hangs is switching to mergerfs and uploading your local files to rclone on a scheduled task/cronjob.

As FYI, this is the command I use every day at 2am
/usr/bin/rclone move /home/stokkes/media gdMedia: -v --transfers 50 --min-age 1h --exclude="/downloads/**" --log-file=/opt/rclone/logs/rclone.local2cloud.log


#7

Thanks for the input guys.

Stokkes - do you have a guide handy to setup that? I assume with your setup you can still point Sonarr at the rclone mount?

Animosity022 - Do you use https://rclone.org/cache/ ? Does that give you hangs?


#8

I’d suggest following the guide @Animosity022 has on his github. I use a similar setup but no point reinventing the wheel. We don’t use the cache backend. IMO it has its own problems I won’t get into.

You could always use my move command and adopt where necessary to periodically move local files to your remote.

With mergerfs you won’t get any hangs. You’ll notice too in my command that I exclude my download directory. That’s because my download folder is in the same root folder as my media (on the local disk only) so the moves from Sonarr, etc from download to your media folders are instant.


#9

You can check here as I have all my stuff documented: