Is it safe to rclone sync an entire 4To drive daily ?
Changes aren't so big, somewhere between 20MB to 50GB daily.
But maybe that would be stressfull for the drives ?
Or that another solution might be more appropriate ?
it would not be stressful, transferring such a small amount of data daily.
are you trying to rclone for backup of a local drive?
what provider are you using?
what is your operating system?
have you seen this flag? https://rclone.org/docs/#backup-dir-dir
The goal is indeed to have a backup of the source drive.
Both drives are local (internal sata), on a home server running Ubuntu18LTS.
Source drive is a Nextcloud data drive. Both 4To drives identical.
rclone sync is run through a crontab script daily from and to the root of both drives.
Thanks for the tip on the flag. Not sure its appropriate to my use case though.
Although the next step will be to do the same on a 3rd remote drive of the very cold deep storage type like Glacier for catastrophic emergency retrieve only (still studying the cost of that, with daily sync).
Anyhow.
I was worried about the way [which I obviously don't understand] in which rclone sync knows which files are new or changed. I imagine it has to scan the entire drive (even if there are no changes). I don't see how that can't be stressful for the drives ?
Rclone will scan the entire drive. By default it will check size+modification date on the local disk vs size+modification date on the remote and only transfer if those differ.
A scan of the disk isn't particularly intensive - it really depends on how many files you have. It only reads the directory info on the files, not the files (unless you use the --checksum flag).
If you want to simulate it, then do rclone size /path/to/disk this will scan all the files in a very similar way.
I am not planning on using the checksum option at this point in time (although that would be safer for the data I suppose, I most of all don't want to kill my drives too fast).
I'm using SMART monitoring to check on them, and am reasonnably hoping they can last about 2 years.
Both drives (source and backup) are 4TB WesternDigital Red WD40EFRX 3,5"
However, the number of file counts to 239'987 files in 23'066 directories for 2,9TB used.
Again thanks for your feedback, and the beautiful way you're maintaining this great product that is rclone.