Behaviour of mount/serve in power/internet outage scenario (for backup solution)


I would like to use rclone running on an raspberry pi as a backup solution. I want to use the serve command to serve a cloud storage to my network. Since my upload speed is pretty slow the always-running pi shall do the uploads to the cloud in the background while my devices can backup fast to the volume served by rclone.

What I want to know is how rclone behaves if my internet connection drops or the power goes out while files are not (yet) uploaded to the cloud.
From the perspective of the user (using the volume served by rclone) the backup would be complete (because it's already transmitted to the served volume) but how can I be sure that it eventually will be in the cloud even if the internet drops or the power goes out while the upload from the pi to the cloud is still running? Or in other words: Is the data persisted on the pi (even in a power-outage) until the upload is complete?

And related to that I have one more question: I will use restic on the clients to backup to the volume served by rclone. Restic has a feature to check the integrity of a backup. But if I use the feature in my configuratin described above how can I be sure that it's not using a state cached on the pi (which may be integer but is not necessarily the same state as in the cloud)?

Thank you in advance


Which --vfs-cache-mode are you using? If you aren't using any flags then rclone won't cache data. If you are using --vfs-cache-mode writes or higher, then the latest beta will retry and re-upload files in the event of a power outage or an internet connection drop.

What storage are you using on the Pi? I find that SD cards are quite liable to corrupt in unclean power offs.

You could do a restic check direct to the archive stored on the cloud?

In fact I'd recommend you configure restic to use rclone and backup direct to the cloud - cut out the middle man of the storage server if you want ultimate reliability.

Thank you for your quick reply :slight_smile:

Currently I don't use anything, I'm still waiting for the hardware to arrive and will set everything up in the next days. What do you mean with "won't cache data"? Does this mean that the upload would be piped directly to the cloud so the upload to the pi would go as fast as my internet connection from the pi to the cloud can handle? And can you estimate when the latest beta will be out of beta? Because if the current release doesn't support this I think I'll wait until the new release is out.

At least for the data I'll use an SSD since the micro SDs die pretty fast even with not that much writeload.

Yes you're right. But if possible I would like to avoid that since I have the pi in between anyway (reasons see below). I guess my main problem is that I don't fully understand how caching exactly behaves in rclone.
There is the vfs caching (which is missing in the global flags page, is this intentionally?) and there is the "normal" caching. What's the difference between them?
Also how does --vfs-cache-max-age exactly work for uploads? I.e. if I have --vfs-cache-mode=writes set and --vfs-cache-max-age=1h. Now if I upload a large file to the volume served by the pi and this file takes longer than 1h to upload from the pi to the cloud. What happens then? If rclone sees the --cfs-cache-ax-age as a hard limit wouldn't this mean that rclone would delete the file from it's cache before the upload to the cloud finished?

Regarding the restic checks: If I have a short caching time (let's say 1h) and I do a restic check of my backup after that time passed (so the read-cache should be empty) do you see any possibility that the check is successul even though the data was not successfuly stored in the cloud? In my understanding at that point the pi should have forgotten the data and should have to pull it from the cloud if restic requests them for the check, right?

Unfortunately I can't do this since some backups may need multiple days to upload to the cloud and I don't want to have the computer running 24/7. Also I'd like to keep the client side as simple as possible since there are multiple of them and some operated by people with a pretty bad relation to computers :smiley:

Thank you in advance
P.S. rclone is one of these few pieces of software where I really have to smile when looking through the documentation because in many cases it makes things possible of which you couldn't even dream of before. So thank you for your great work.

That is correct.

It should be a small number of weeks I'd guess!

However the beta should be stable - it is integration tested daily.

They aren't global flags (yet) they only work on mount and serve

There is the cache backend - is that what you mean? That is a backend devoted to caching which is being phased out gradually.

The time starts from when the file is closed.

After the data has expired, rclone has no choice but to pull it from the cloud.

You are welcome!

By vfs-caching I mean all flags starting with --vfs and by "normal" caching I mean all flags starting with --cache like --cache-chunk-no-memory etc.

To the rest of your replies: Perfect, that clarified almost everything

The former is the "vfs" cache which works at a higher level than the "cache" backend which are the other flags --cache (except for --cache-dir which affects everything)

If you do rclone help flags cache you'll see the two sections!

  rclone help flags [<regexp to match>] [flags]

  -h, --help   help for flags

Global Flags:
      --cache-dir string   Directory rclone will use for caching. (default "/home/ncw/.cache/rclone")

Backend Flags:
      --cache-chunk-clean-interval Duration   How often should the cache perform cleanups of the chunk storage. (default 1m0s)
      --cache-chunk-no-memory                 Disable the in-memory cache for storing chunks during streaming.
      --cache-chunk-path string               Directory to cache chunk files. (default "/home/ncw/.cache/rclone/cache-backend")
      --cache-chunk-size SizeSuffix           The size of a chunk (partial file data). (default 5M)
      --cache-chunk-total-size SizeSuffix     The total size that the chunks can take up on the local disk. (default 10G)
      --cache-db-path string                  Directory to store file structure metadata DB. (default "/home/ncw/.cache/rclone/cache-backend")
      --cache-db-purge                        Clear all the cached data for this remote on start.
      --cache-db-wait-time Duration           How long to wait for the DB to be available - 0 is unlimited (default 1s)
      --cache-info-age Duration               How long to cache file structure information (directory listings, file size, times etc). (default 6h0m0s)
      --cache-plex-insecure string            Skip all certificate verification when connecting to the Plex server
      --cache-plex-password string            The password of the Plex user (obscured)
      --cache-plex-url string                 The URL of the Plex server
      --cache-plex-username string            The username of the Plex user
      --cache-read-retries int                How many times to retry a read from a cache storage. (default 10)
      --cache-remote string                   Remote to cache.
      --cache-rps int                         Limits the number of requests per second to the source FS (-1 to disable) (default -1)
      --cache-tmp-upload-path string          Directory to keep temporary files until they are uploaded.
      --cache-tmp-wait-time Duration          How long should files be stored in local cache before being uploaded (default 15s)
      --cache-workers int                     How many workers should run in parallel to download chunks. (default 4)
      --cache-writes                          Cache file data on writes through the FS
      --union-cache-time int                  Cache time of usage and free space (in seconds). This option is only useful when a path preserving policy is used. (default 120)

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.