Copy takes too much space; Sync has potential to lose info from backup target. What would you do?

Help and Support
1

I am trying to find the best method for syncing a hosted file server to my NAS.

We run rclone with Wasabi mounted storage. This is what we utilize for a main file server. I want to run rclone to bring a local copy back to one of our Synology NAS devices.

I run different branches on different backups, we offload different Wasabi buckets to B2 Backblaze just to ensure we have multiple copies of files in case someone gets froggy or accidentally deletes something.

However, running 'copy' is a little difficult as we are talking about TB's of information. Saving every copy perpetually really isn't an option. This would take entirely too much room. "Sync" sounds fine, but then I'm worried that someone accidentally removes something and we don't catch it before the next "sync" cycle.

Has anyone experienced this before? Are there some flags that could assist with this to ensure we run copies and don't incur more data charges (a copy or two of a file is fine, 10 is not.)

Thanks,
David

2
  • Filtering, includes and excludes are used by copy and sync
  • sync will still bite by respecting any authoring mistakes
  • copy with just excludes is inadvertent delete resistant
  • --max-age is a potentially useful exclude
3

You can use sync with the --backup-dir option. Docs: https://rclone.org/docs/#backup-dir-dir

4

hi,
i run a sync on a daily schedule so i use --max-age, then once a month, i can run the sync without --max-age to make sure all files are backed-up.

i have a python script that generates my rclone commands.
here is a simplified example using --backup-dir with an archive folder.
rclone sync source remote:backup --backup-dir=remote:archive/2021.02.20.09.30

let's say there is a local file named 01.kdbx that has changed since the last time rclone sync was run.
when rclone sync is next run,

  1. rclone notices that 01.kdbx needs to uploaded.
  2. rclone notices that there is a file named 01.kdbx in the remote:backup.
  3. rclone server-side-move 01.kdbx from remote:backup to remote:archive/2021.02.20.09.30

the log would look like

DEBUG : 01.kdbx: Sizes differ (src 356798 vs dst 356638)
INFO  : 01.kdbx: Moved (server side)
DEBUG : 01.kdbx: MD5 = 4e826893b0d91d6a32df4c58dead91f6 OK
INFO  : 01.kdbx: Copied (new)
5

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.