Do nothing when more then 50% is changed

[SL-Gundam]

Is there a way to tell Rclone to not do anything when more then a certain percentage of the data has changed?

Let's say i'm syncing (rclone sync) 20 TB of data.
I'm pretty sure i'll never modify more then 10TB per day. So i would like Rclone to not do anything if more then 50% of the data has changed.
This would be an extra security measure against data loss.

[ncw (Nick Craig-Wood)]

You can't do exactly that, but rclone does have some features which can help

  --max-transfer SizeSuffix   Maximum size of data to transfer. (default off)
  --max-delete int    When synchronizing, limit the number of deletes (default -1)

If you are worried about data loss then I recommend this flag

  --backup-dir string   Make backups into hierarchy based in DIR.

[SL-Gundam]

I'm already using --backup-dir. So not really worried about the data-loss but more on the work required to restore it.

C:\rclone\rclone.exe sync --bwlimit 3M:off --fast-list --track-renames --track-renames-strategy modtime --check-first --delete-after --delete-excluded --exclude-from "C:/rclone/Configs/Exclusions.txt" --backup-dir cryptGDrive:Archive/Union --progress --log-file "C:\Users\username\Desktop\Log\Rclone Mirror_Union.log" --log-level INFO local:E:/Union/ cryptGDrive:Union

For retrieving a couple accidental deletions it's fine. But i'm syncing around ~32 TB of data in ~280000 files.
I've merged this data from 3 locations by creating a folder containing 3 symbolic links (makes Rclone detect moving files between those locations).
This works fine as long as Rclone properly detects a failed location through the symbolic link. I tested this but the result largely depends on whether or not Rclone receives proper errors from the OS or networked location through the symbolic link.
Restoring ~80000 files from the archive where older files are also archived would be a real bitch to do. Of course adding a date to the --backup-dir parameter would alleviate this. But that would mean that single file restores outside of my logging window would require more searching for the folder containing the file. Now the archive is just one folder that more or less contains the same folder structure as the original.

--max-transfer is not needed. As long as i'm uploading its good.
--max-delete is basically the same problem but in smaller increments. Something that does nothing is easier to discover then something that works partially. Doing nothing when the deletes exceeded a certain size or number would help. But like you said thats not part of the current feature set.

Luckily I'm very happy with the way Rclone works right now. The initial request would just have been a nice additional safety net.

Thank you for maintaining this excellent program and answering my question.

[ncw (Nick Craig-Wood)]

What you could do is to a --dry-run sync and parse the final stats output. (You can get this in JSON too). That tells you exactly how many files would have been transferred, etc

rclone sync --dry-run a b

Transferred:   	   48.444Ki / 48.444 KiByte, 100%, 0 Byte/s, ETA -
Checks:              1007 / 1007, 100%
Deleted:             1005 (files), 101 (dirs)
Transferred:         1002 / 1002, 100%
Elapsed time:         0.1s

Or

rclone sync --dry-run a b --use-json-log

Which produces (prettified)

{
  "level": "warning",
  "msg": "\nTransferred:   \t   48.444Ki / 48.444 KiByte, 100%, 0 Byte/s, ETA -\nChecks:              1007 / 1007, 100%\nDeleted:             1005 (files), 101 (dirs)\nTransferred:         1002 / 1002, 100%\nElapsed time:         0.1s\n\n",
  "source": "accounting/stats.go:480",
  "stats": {
    "bytes": 49607,
    "checks": 1007,
    "deletedDirs": 101,
    "deletes": 1005,
    "elapsedTime": 0.15730507,
    "errors": 0,
    "eta": null,
    "fatalError": false,
    "renames": 0,
    "retryError": false,
    "speed": 0,
    "totalBytes": 49607,
    "totalChecks": 1007,
    "totalTransfers": 1002,
    "transferTime": 0,
    "transfers": 1002
  },
  "time": "2021-09-19T15:00:36.543849+01:00"
}

[SL-Gundam]

Normal operation should be pretty much automated. So the json log would be an interesting one with which i write a script to check various values and flag anything problematic.

Thank you. Will try that.