I’ve got about 6 TB of data stored on an encrypted Google Drive share. I’m using Rclone for FreeBSD in FreeNAS 11.1 to copy or sync these files.
The problem I’m having is that every single file, on my computer is apparently being detected as newer than the ones on the encrypted share (each file is labeled “Copied (new)” by Rclone), and so every single file is being uploaded again and again. Give that I have a bit more than 5 Mbps upload speed, this means that “updating” my backup (which is really redoing it in its entirety) takes a couple weeks!
I have already tried setting Rclone to tolerate up to 1 second of discrepancy in file times, but that’s made no difference.
I’m currently running it with -vv as you suggested.
I should have clarified that I used to back up from my laptop to my FreeNAS server (with rsync) and also from my laptop to my Google Drive encrypted share (with Rclone). But I recently streamlined this process so I’m now backing up my laptop to the FreeNAS server, and then the FreeNAS server to Google Drive using `Rclone.
Looking at system times, it seems that my FreeNAS server time is a bit over 3 seconds different from my laptop time. That might explain the problem. I’ve set both systems to use the same NTP servers and have manually synced them.
If I’m right about this being the problem, after the initial sync from FreeNAS to Google Drive the problem should go away. I’ll post here again when it’s done (probably 2 weeks) and let you know if this was indeed the problem.
I’ve had similar issues myself. There wasn’t an error though. It was just true. The remote date never changed, and the local date sometimes did change for various reasons. (For example when making a backup of a backup depending on what commands I used it was possible googledrive kept the original date of the original backup and the middleman local backup drive kept updating when new backups were performed to it).
Using something like rclone copy --checksum will not only run a checksum but will ignore the mod-time entirely, that could be a workaround, not an ideal one, but if there’s no bug here, and the dates simply are not the same, well, rclone isn’t at fault.
Even if it is just the mod time that is different, rclone shouldn’t upload the file, it should just correct the mod-time.
Good to know. Looks like I’m facing a bit bigger of a problem than I thought, then. Besides perhaps increasing the --modify-window time to maybe 5 seconds, is my only other option to use checksums?
Also, does this mean that Rclone already uses checksums in these cases? Because it would be rather daring to correct the mod time of files (i.e. declare the files identical) just based on the fact that they are the same size.
I had no idea that --checksum does nothing with crypt!
Assuming that re-uploading my files from the NAS doesn’t fix this problem by updating the modification dates on the crypt share, what are my options, besides using a bigger --modify-window value?
It is mentioned in the docs that crypt doesn’t support hashes which implies that --checksum doesn’t work but perhaps not terribly obviously.
That should work. It is a bit heavy handed though - sorry!
You can do a --size-only sync. Which checks only the size of the files and nothing else. I use this for my backup of my pictures and it works very well there. It may or may not work for you though.
