I’m using rclone to sync borg backup repos to wasabi and so far it’s working great. I do worry about corrupt data in case of abrupt interruptions, i.e accidentally sending a kill signal to it with ctrl+c while it’s still uploading or because of a network error. My fear is that some borg repo files are uploaded and others are not, which could lead the repo to be corrupt.
Does rclone wrap the transfer in a transaction and rollbacks somehow? ISpecifically to rollback everything if a single file fails uploading? Are there any options for that? If not, are there any good practices to follow when uploading something that is constituted of multiple files, like a borg or git repo in the case of a failure when some repository files were uploaded but others not?
I am not aware about any such mechanism but in practice I can’t imagine how it would work, most of the reasons that would interrupt a transfer (starting from network trouble to API/transfer quota exceeded, the user killing the process, machine rebooting, etc) would just as well prevent any rollback.
Also for many (most) commands rclone would overwrite files so the information to roll back might not be there completely - think about it you have old_file overwritten by new_file - you might not be able to get old_file from anywhere if you change your mind.
UNLESS you are using --backup-dir with a different folder each time, for example blabla/date "+%Y-%m-%d-%H-%M-%S" ! Then you have some kind of poor man’s incremental backup and if worst comes to worst you can restore the destination to any previous state. Well, it won’t keep track of which are the new files (it will only save changed and removed files to backup-dir) but probably you can remove them by timestamp if borg really fails because it has too many files in its repository (which would frankly be a reason for me to ditch it altogether…).
If you retry the sync or copy then rclone will fill in the missing pieces. rclone will do this automatically 3 times normally. If you are concerned you can set the --retries flag higher.
Rclone will exit with an error code if something goes wrong enabling you to at least know that all was not well.
So, all in all, I can just run the same rclone command again and trust that rclone will fill in the missing pieces by checking what files are already in the remote destination vs what’s locally. There’s no risk of getting an invalid state in the remote destination at all?
rclone replaces files in the destination remote when it sees files have changed, so I think it’s pretty trustworthy in that regard. If you prefer, you can also run rclone check (or cryptcheck if your remote is encrypted) if you want to verify the integrity yourself. And in fact you should do it periodically.