Cache settings for a short expiration/synchronization

I use rclone to mount Mega remote storage and use this space to store the Zimbra backup. Here's how I set up remote storage:

/usr/bin/rclone mount mega_storage:backup/zimbra /mnt/mega --allow-other --allow-non-empty --config /root/.config/rclone/rclone.conf --log-file /var/log/rclone.log --cache-dir /var/cache/rclone --vfs-cache-mode writes --vfs-cache-max-age 1m

The problem is that if I unmount the remote storage and I delete the local cache, when I remount the remote store some data are gone.

How should I set the rmount parameters to synchronize the cache after a short time (my parameters seem correct to me, but obviously they are not)?

Or how can I force cache synchronization?

Thank you!

rclone --version
rclone v1.58.1

  • os/version: ubuntu 18.04 (64 bit)
  • os/kernel: 5.15.35-2-pve (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: static
  • go/tags: none

cat .config/rclone/rclone.conf
[mega_storage]
type = mega
user = xxxxx@xxxxxxxxxxxxxxxxxx.xxx
pass = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

--- the debug log should help answer your question
--- if a file is in-use, rclone will not remove it.
--- --allow-non-empty allows for over mounting, rarely needed.

1 Like

What does that mean? You are writing to the mount point and waiting for it to get uploaded?

Why are you deleting the local cache?

1 Like

Thanks @asdffdsa

I have a 100GB disk on my server and only 40GB is free. The backup generates 40 GB of data. To prevent the backup from filling up the disk, the backup data must be (almost) immediately written to remote storage. This means that the cache must be minimized. In addition, I must be sure that at the end of the backup all data is present in the remote storage (and deleted from the local disk).

Thanks for your help @Animosity022 !

i have a similar setup, but the backup files are from veeam, not zimbra.

after the veeam backup creates the backup file, i rclone copy to cloud.
tho in your case, can use rclone move

I can't: as the local disk fills up the server stops working, long before the backup is complete.

Has anyone ever needed to write files directly to remote storage?!?

I mean I can't wait for the backup to finish to get the cache synchronization, because the local disk gets full before the backup is finished.

As a test, just to check if all the files are really in the remote storage. Deleting the cache I realize that they are not, because when I remount the remote storage some files have disappeared. In this case I run a test with a small amount of files that does not cause the local disk to fill up, discovering that even in this case some files remain in the cache even after a few hours from the end of the backup. So I am convinced that I am doing something wrong with the rclone parameters.

You can ask rclone if it still has things it wants to upload via the RC:

In particular these items

    "uploadsInProgress": 0,
    "uploadsQueued": 0

There's no log file or any data to look at to see what's going on so it's really hard to tell.

rclone mount

Gives you information. You most likely want to set a size on your cache.

--vfs-cache-max-size SizeSuffix      Max total size of objects in the cache (default off)

as it defaults to off or unlimited.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.