Copied (replaced existing) from local to s3 crypt

[Zetto]

What is the problem you are having with rclone?

Durring a sync 3 files were copied and replaced. Each of the files has not been modified so we assumed corruption that would require recovering the old version from s3. To verify this we hash the file on the source and the destination. We then hash the old version from s3 and all three files have identical hash so we don't understand why it was replaced.
2019/09/25 23:44:10 INFO : Folder/Folder/File1.ext: Copied (replaced existing)
2019/09/25 23:44:10 INFO : Folder/Folder/File2.ext: Copied (replaced existing)
2019/09/25 23:44:11 INFO : Folder/Folder/File3.ext: Copied (replaced existing)

What is your rclone version (output from rclone version)

rclone v1.46

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux

Which cloud storage system are you using? (eg Google Drive)

s3 with crypt

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --bwlimit 5M --delete-after --checkers 32 --log-file rclone.log --fast-list --stats-unit bits --transfers 8 --verbose '/Folder/' s3-crypt:'/Folder/'

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

[ncw (Nick Craig-Wood)]

If you run with -vv you will get a DEBUG message which tells you why it was copied.

It might have been the modification time that changed?

[Zetto]

The three files were replaced once and on subsequent sync they are unchanged. The modified date on the source files is from months prior so they do not appear to have changed. We even hashed the files and they all appear to be identical (The source, destination that was overwritten and the new destination).

Is there anything more we can do to determine why rclone thinks the files needed to be copied? For all future jobs we will use -vv to have additional logging.

[thestigma]

I have noticed some similar cases too where updating happens in cases where I can't immediately say why it was needed - I notice it most in my cache personally. I just haven't had the time to really dig into the details of it yet as it requires a lot of monitoring and tedious manual comparisons of metadata. I half suspect that there is some kind of intermittent error on updating modtime on the remote-side sometimes, or maybe some kind of timing error causing a non-perfect match (possibly on very large files above 1GB especially - have you noted any pattern like that?)

I think this deserves investigation, but you may just solve the problem for now by using the --checksum flag to make rclone compare on the checksum rather size+modtime . This can have some downsides in some cases, but it will garantuee that bit-identical files are not re-copied redundantly.