Only copy if the destination file has the same checksum as last time copy

I have multiple clients working on the same files and copying changes to destination periodically. I would like to avoid lost update situation, where an update (rclone copy) is overwritten by a newer update (rclone copy). Is this already supported?

rclone copy --update works partially as it doesn't guarantee the destination file is same as the local one (last time rclone copy)

I'm not quite sure what your criteria is for updating.

If the checksums are the same, the file is the same and would not update.

  -c, --checksum                             Skip based on checksum (if available) & size, not mod-time & size

--update is when you have a newer file on the destination based on mod time and it would not update the file if the source was older.

What's your criteria for copying from the source to destination based on if you had a magic wand?

Sorry for not being clear enough, let me try it again.

My situation is below:

  1. Both client A and client B have a local copy of the destination file
  2. Client A modifies and copies to destination thus updates checksum and mod time
  3. Client B copies to destination but since destination is newer thus skipped
  4. Client B modifies and copies to destination thus overwrites destination (A's change)

I want to prevent step 4 by checking if destination checksum is still the same as in step 1, if not, then skip copying. That would imply saving a copy of step 1 checksum somewhere? Not sure it's viable with current functionality.

If it's modified though, the checksum would not be the same.

If two files have the same checksums, they are the same files.

If you two sources writing files to one spot, you have to decide who wins in conflict.

In your scenario, you have:

A->C
B->C

But C never goes back to A or B. You need to have C->A and C->B to check if something is newer and update A or B if something is newer than modify.

The real solution here is more a team drive so everyone is modifying the same C in my example rather than having local copies.

I understand what you are saying, one needs to know what's there in the destination to decide who's newer but I think I am not even there yet, I am simply suggesting if it's possible skip copying if rclone detects remote has changed since last copying.

My naive thinking is saving a copy of checksum when first copying, then when second copying, check if the destination checksum is still the same as the one we saved, if it's different, then we skip and report error. It's like optimistic lock.

I am not sure if there is truly "single" C since everything has to go over the wire for you to see, so a local copy always present, I could be wrong.

If you run a copy command, the goal is to copy something from source to destination, no? You can use any way to determine how to copy from source to destination. Checkums only tell if files are the same. You can use --update to ignore newer files.

You can use rclone check to compare source to destination and see what's different.

I'm not sure what you are getting at here as there always is a destination that you are copying to. In your example, you gave two clients going to a destination and I called that C in my example.

If you are trying to compare A<->B, that's more a sync tool between the two.

I think you want a bidirection sync tool, which is not rclone.

I don't think I really want bi-directional sync, I still prefer the simplicity of one-way sync. I will try put my thought below, maybe others interested can chime in.

The feature will be similar to rclone copy --update, except it will make sure destination has not changed by comparing the destination files with a local snapshot of checksum (since last rclone copy). If checksum is different then destination has changed (branched off), thus skip copying. It could be something like this, rclone copy --update-with-lock, the "lock" is the snapshot of checksum. If the checksum is not available, behavior will be same as rclone copy --update

Checksums are not kept anywhere and for a local file, you'd have to generate a new one to make sure you have the right check or you'd have to write a routine to see if the local modtime changed on the file and generate a new checksum, which is why I think rclone just validates the checksum of the file.

I'm still just having a hard time following the flow of what happens when something doesn't match and if there is intervention to figure out if they don't match, do you pick a side and copy up or what happens? I just can't seem to figure out your flow operationally.

Thanks for your reply, I go back and rethink the entire workflow, I started out wanting to have a single working directory tree shared by multiple computers, because I only have 2-3 computers and I am the only user so I thought it is manageable to do manual conflict resolution. Turns out there is no easy solution.

Eventually I decided to split the working directory, one for each computer and have rclone sync them to gdrive. I also have a common shared directory in gdrive and use the GDrive B&S to sync to each computer.

Not sure if there is better way to do this, since now I have multiple working directory but I am happy with the tradeoff that I don't need to think about change conflicts.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.