Hi all, I recently needed to move all my files off my ZFS NAS due to that I needed to create a new vdev with ashift 12 instead of 9.
Anyhow I moved all the files/directories via rsync (over ssh), and then back again on the newly created vdev/pool. Now I’m trying to run rclone again (using gdrive as backend), but it seems like it sees all the files as new even though I’ve tried to keep all the modification times, etc.
Is there a smart way to get it to skip existing files?
Thanks.
No input? No tips? Hints?
By default rclone should only check the file size and modified time.
So if you preserved the modified times it should treat the files equal.
Can you provide a short list of files with size and modified times on the source and destination that rclone tried to copy again?
This would help finding the issue.
You can also try to increase the log level and I think rclone will print the reason why it copies the files.
Thanks.
Here is excerpt from the log file:
2018/07/27 18:36:21 NOTICE: storage/tv/redacted.720p.X264/redacted.S02E10.720p.x264.mkv: Not moving as --dry-run
2018/07/27 18:36:21 NOTICE: storage/tv/redacted.720p.X264/redacted.S02E10.720p.x264.mkv: Not copying as --dry-run
And listing of both local and remote directories.
Local files:
-rwxrwxrwx 1 nobody 4294967294 1484372543 Jul 2 2016 redacted.S01E01.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1484624368 Jul 2 2016 redacted.S01E02.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1464423280 Jul 2 2016 redacted.S01E03.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1432222124 Jul 2 2016 redacted.S01E04.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1462970654 Jul 2 2016 redacted.S01E05.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1377935302 Jul 2 2016 redacted.S01E06.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1487172056 Jul 2 2016 redacted.S01E07.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1468560469 Jul 2 2016 redacted.S01E08.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1441778719 Jul 2 2016 redacted.S01E09.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1509854517 Jul 2 2016 redacted.S01E10.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1395379054 Jul 2 2016 redacted.S01E11.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1450802402 Jul 2 2016 redacted.S01E12.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1440112159 Jul 2 2016 redacted.S01E13.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1439293107 Jul 2 2016 redacted.S01E14.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1438938627 Jul 2 2016 redacted.S01E15.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1460920825 Jul 2 2016 redacted.S01E16.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1531018924 Jul 2 2016 redacted.S01E17.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1437555704 Jul 2 2016 redacted.S01E18.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1406804890 Jul 2 2016 redacted.S01E19.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1469884174 Jul 2 2016 redacted.S01E20.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1430728303 Jul 2 2016 redacted.S01E21.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1435441030 Jul 2 2016 redacted.S01E22.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1484552721 Jul 2 2016 redacted.S02E01.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1385731109 Jul 2 2016 redacted.S02E02.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1453473273 Jul 2 2016 redacted.S02E03.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1504139734 Jul 2 2016 redacted.S02E04.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1445701290 Jul 2 2016 redacted.S02E05.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1455588990 Jul 2 2016 redacted.S02E06.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1445750147 Jul 2 2016 redacted.S02E07.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1442052294 Jul 2 2016 redacted.S02E08.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1395419228 Jul 2 2016 redacted.S02E09.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1439486032 Jul 2 2016 redacted.S02E10.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1424965911 Jul 2 2016 redacted.S02E11.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1432853990 Jul 2 2016 redacted.S02E12.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1445167045 Jul 2 2016 redacted.S02E13.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1454066273 Jul 2 2016 redacted.S02E14.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1387966866 Jul 2 2016 redacted.S02E15.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1541475633 Jul 2 2016 redacted.S03E01.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1501251974 Jul 2 2016 redacted.S03E02.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1466880791 Jul 2 2016 redacted.S03E03.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1437249821 Jul 2 2016 redacted.S03E04.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1428873468 Jul 2 2016 redacted.S03E05.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1434835088 Jul 2 2016 redacted.S03E06.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1436271027 Jul 2 2016 redacted.S03E07.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1486621497 Jul 2 2016 redacted.S03E08.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1425886360 Jul 2 2016 redacted.S03E09.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1387230849 Jul 2 2016 redacted.S03E10.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1455517910 Jul 2 2016 redacted.S03E11.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1454029456 Jul 2 2016 redacted.S03E12.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1505137836 Jul 2 2016 redacted.S03E13.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1499455759 Jul 2 2016 redacted.S04E01.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1490644028 Jul 2 2016 redacted.S04E02.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1437768949 Jul 2 2016 redacted.S04E03.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1391400914 Jul 2 2016 redacted.S04E04.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1544477056 Jul 2 2016 redacted.S04E05.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1395330174 Jul 2 2016 redacted.S04E06.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1425238988 Jul 2 2016 redacted.S04E07.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1437043126 Jul 2 2016 redacted.S04E08.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1465101677 Jul 2 2016 redacted.S04E09.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1425687280 Jul 2 2016 redacted.S04E10.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1407139277 Jul 2 2016 redacted.S04E11.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1452600099 Jul 2 2016 redacted.S04E12.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1516080530 Jul 2 2016 redacted.S04E13.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1442031020 Jul 2 2016 redacted.S05E01.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1440968031 Jul 2 2016 redacted.S05E02.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1457032749 Jul 2 2016 redacted.S05E03.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1415478388 Jul 2 2016 redacted.S05E04.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1459600273 Jul 2 2016 redacted.S05E05.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1488975388 Jul 2 2016 redacted.S05E06.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1422324718 Jul 2 2016 redacted.S05E07.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1416327003 Jul 2 2016 redacted.S05E08.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1443154791 Jul 2 2016 redacted.S05E09.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1507099860 Jul 2 2016 redacted.S05E10.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1456972841 Jul 2 2016 redacted.S05E11.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1489987207 Jul 2 2016 redacted.S05E12.720p.x264.mkv
-rwxrwxrwx 1 nobody 4294967294 1994236455 Jul 2 2016 redacted.S05E13.720p.x264.mkv
Google Drive:
1484372543 2016-07-02 21:59:39.236000000 redacted.S01E01.720p.x264.mkv
1484624368 2016-07-02 21:59:39.346000000 redacted.S01E02.720p.x264.mkv
1464423280 2016-07-02 21:59:39.456000000 redacted.S01E03.720p.x264.mkv
1432222124 2016-07-02 21:59:39.583000000 redacted.S01E04.720p.x264.mkv
1462970654 2016-07-02 21:59:39.689000000 redacted.S01E05.720p.x264.mkv
1377935302 2016-07-02 21:59:39.799000000 redacted.S01E06.720p.x264.mkv
1487172056 2016-07-02 21:59:39.916000000 redacted.S01E07.720p.x264.mkv
1468560469 2016-07-02 21:59:40.036000000 redacted.S01E08.720p.x264.mkv
1441778719 2016-07-02 21:59:40.156000000 redacted.S01E09.720p.x264.mkv
1509854517 2016-07-02 21:59:40.276000000 redacted.S01E10.720p.x264.mkv
1395379054 2016-07-02 21:59:40.399000000 redacted.S01E11.720p.x264.mkv
1450802402 2016-07-02 21:59:40.529000000 redacted.S01E12.720p.x264.mkv
1440112159 2016-07-02 21:59:40.659000000 redacted.S01E13.720p.x264.mkv
1439293107 2016-07-02 21:59:40.789000000 redacted.S01E14.720p.x264.mkv
1438938627 2016-07-02 21:59:40.943000000 redacted.S01E15.720p.x264.mkv
1460920825 2016-07-02 21:59:41.079000000 redacted.S01E16.720p.x264.mkv
1531018924 2016-07-02 21:59:41.229000000 redacted.S01E17.720p.x264.mkv
1437555704 2016-07-02 21:59:41.369000000 redacted.S01E18.720p.x264.mkv
1406804890 2016-07-02 21:59:41.516000000 redacted.S01E19.720p.x264.mkv
1469884174 2016-07-02 21:59:41.669000000 redacted.S01E20.720p.x264.mkv
1430728303 2016-07-02 21:59:41.816000000 redacted.S01E21.720p.x264.mkv
1435441030 2016-07-02 21:59:41.000000000 redacted.S01E22.720p.x264.mkv
1484552721 2016-07-02 22:01:00.209000000 redacted.S02E01.720p.x264.mkv
1385731109 2016-07-02 22:01:00.329000000 redacted.S02E02.720p.x264.mkv
1453473273 2016-07-02 22:01:00.442000000 redacted.S02E03.720p.x264.mkv
1504139734 2016-07-02 22:01:00.559000000 redacted.S02E04.720p.x264.mkv
1445701290 2016-07-02 22:01:00.679000000 redacted.S02E05.720p.x264.mkv
1455588990 2016-07-02 22:01:00.802000000 redacted.S02E06.720p.x264.mkv
1445750147 2016-07-02 22:01:00.932000000 redacted.S02E07.720p.x264.mkv
1442052294 2016-07-02 22:01:01.075000000 redacted.S02E08.720p.x264.mkv
1395419228 2016-07-02 22:01:01.205000000 redacted.S02E09.720p.x264.mkv
1439486032 2016-07-02 22:01:01.352000000 redacted.S02E10.720p.x264.mkv
1424965911 2016-07-02 22:01:01.609000000 redacted.S02E11.720p.x264.mkv
1432853990 2016-07-02 22:01:01.752000000 redacted.S02E12.720p.x264.mkv
1445167045 2016-07-02 22:01:01.885000000 redacted.S02E13.720p.x264.mkv
1454066273 2016-07-02 22:01:02.015000000 redacted.S02E14.720p.x264.mkv
1387966866 2016-07-02 22:01:02.135000000 redacted.S02E15.720p.x264.mkv
1541475633 2016-07-02 22:01:02.282000000 redacted.S03E01.720p.x264.mkv
1501251974 2016-07-02 22:01:02.402000000 redacted.S03E02.720p.x264.mkv
1466880791 2016-07-02 22:01:02.522000000 redacted.S03E03.720p.x264.mkv
1437249821 2016-07-02 22:01:02.635000000 redacted.S03E04.720p.x264.mkv
1428873468 2016-07-02 22:01:02.765000000 redacted.S03E05.720p.x264.mkv
1434835088 2016-07-02 22:01:02.875000000 redacted.S03E06.720p.x264.mkv
1436271027 2016-07-02 22:01:03.012000000 redacted.S03E07.720p.x264.mkv
1486621497 2016-07-02 22:01:03.138000000 redacted.S03E08.720p.x264.mkv
1425886360 2016-07-02 22:01:03.265000000 redacted.S03E09.720p.x264.mkv
1387230849 2016-07-02 22:01:03.392000000 redacted.S03E10.720p.x264.mkv
1455517910 2016-07-02 22:01:03.525000000 redacted.S03E11.720p.x264.mkv
1454029456 2016-07-02 22:01:03.692000000 redacted.S03E12.720p.x264.mkv
1505137836 2016-07-02 22:01:03.882000000 redacted.S03E13.720p.x264.mkv
1499455759 2016-07-02 22:01:04.032000000 redacted.S04E01.720p.x264.mkv
1490644028 2016-07-02 22:01:04.172000000 redacted.S04E02.720p.x264.mkv
1437768949 2016-07-02 22:01:04.318000000 redacted.S04E03.720p.x264.mkv
1391400914 2016-07-02 22:01:04.458000000 redacted.S04E04.720p.x264.mkv
1544477056 2016-07-02 22:01:04.652000000 redacted.S04E05.720p.x264.mkv
1395330174 2016-07-02 22:01:04.708000000 redacted.S04E06.720p.x264.mkv
1425238988 2016-07-02 22:01:04.765000000 redacted.S04E07.720p.x264.mkv
1437043126 2016-07-02 22:01:04.835000000 redacted.S04E08.720p.x264.mkv
1465101677 2016-07-02 22:01:04.885000000 redacted.S04E09.720p.x264.mkv
1425687280 2016-07-02 22:01:04.938000000 redacted.S04E10.720p.x264.mkv
1407139277 2016-07-02 22:01:04.988000000 redacted.S04E11.720p.x264.mkv
1452600099 2016-07-02 22:01:05.038000000 redacted.S04E12.720p.x264.mkv
1516080530 2016-07-02 22:01:05.088000000 redacted.S04E13.720p.x264.mkv
1442031020 2016-07-02 22:01:05.145000000 redacted.S05E01.720p.x264.mkv
1440968031 2016-07-02 22:01:05.195000000 redacted.S05E02.720p.x264.mkv
1457032749 2016-07-02 22:01:05.268000000 redacted.S05E03.720p.x264.mkv
1415478388 2016-07-02 22:01:05.322000000 redacted.S05E04.720p.x264.mkv
1459600273 2016-07-02 22:01:05.375000000 redacted.S05E05.720p.x264.mkv
1488975388 2016-07-02 22:01:05.432000000 redacted.S05E06.720p.x264.mkv
1422324718 2016-07-02 22:01:05.485000000 redacted.S05E07.720p.x264.mkv
1416327003 2016-07-02 22:01:05.542000000 redacted.S05E08.720p.x264.mkv
1443154791 2016-07-02 22:01:05.595000000 redacted.S05E09.720p.x264.mkv
1507099860 2016-07-02 22:01:05.658000000 redacted.S05E10.720p.x264.mkv
1456972841 2016-07-02 22:01:05.715000000 redacted.S05E11.720p.x264.mkv
1489987207 2016-07-02 22:01:05.768000000 redacted.S05E12.720p.x264.mkv
1994236455 2016-07-02 22:01:05.828000000 redacted.S05E13.720p.x264.mkv
Does this help any?
Just copy one file with -vv output and post that.
Can I use copy instead of sync (that I normally use) to test this issue?
Sorry, I meant sync with the -vv output in the log.
Ok, I managed to test it on just one file.
I ran this:
# rclone --dry-run -vv sync /mnt/storage/tv/redacted.720p.X264/redacted.S02E10.720p.x264.mkv gdrive-crypt:last_snapshot/storage/tv/redacted.720p.X264/ --config /mnt/backup/_rclone/rclone.conf --transfers=32 --checkers=16 --drive-chunk-size=256M
2018/07/28 02:28:55 DEBUG : redacted.S02E10.720p.x264.mkv: Modification times differ by 352ms: 2016-07-02 22:01:01 +0200 CEST, 2016-07-02 20:01:01.352 +0000 UTC
2018/07/28 02:28:55 NOTICE: redacted.S02E10.720p.x264.mkv: Not copying as --dry-run
2018/07/28 02:28:55 INFO : Encrypted drive 'gdrive-crypt:last_snapshot/storage/tv/redacted.720p.X264/': Waiting for checks to finish
2018/07/28 02:28:55 INFO : Encrypted drive 'gdrive-crypt:last_snapshot/storage/tv/redacted.720p.X264/': Waiting for transfers to finish
2018/07/28 02:28:55 INFO : Waiting for deletions to finish
2018/07/28 02:28:55 INFO :
Transferred: 0 Bytes (0 Bytes/s)
Errors: 0
Checks: 1
Transferred: 1
Elapsed time: 5s
2018/07/28 02:28:55 DEBUG : 5 go routines active
2018/07/28 02:28:55 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone" "--dry-run" "-vv" "sync" "/mnt/storage/tv/redacted.720p.X264/redacted.S02E10.720p.x264.mkv" "gdrive-crypt:last_snapshot/storage/tv/redacted.720p.X264/" "--config" "/mnt/backup/_rclone/rclone.conf" "--transfers=32" "--checkers=16" "--drive-chunk-size=256M"]
It looks like the modification time differs by 352 milliseconds. I don’t even think my linux system reports file modification time in milliseconds.
Google Drive: 1439486032 2016-07-02 22:01:01.352000000 redacted.S02E10.720p.x264.mkv
Linux: -rwxrwxrwx 1 nobody 4294967294 1509854517 Jul 2 2016 redacted.S01E10.720p.x264.mkv
I guess everything should be good if I could get rclone to ignore the milliseconds. Any way to do that?
EDIT:
It seems that I can add “–checksum” flag and get the correct behaviour, but I guess that will take huge amount of time and resources. Or am I wrong?
2018/07/28 02:44:48 DEBUG : redacted.S02E10.720p.x264.mkv: Size of src and dst objects identical
2018/07/28 02:44:48 DEBUG : redacted.S02E10.720p.x264.mkv: Unchanged skipping
I still think the best thing would be to be able to skip the milliseconds from the modification time.
EDIT2:
And I just found out about --modify-window. That should help. Trying to set it at 500 ms, and try a dry run of all files.
Or is --checksum the way to go? I’ve seen a few examples of people using it. I also read somewhere that it’s not supported on encrypted drives – even though it seems to work for me?
For non crypt remotes --checksum is is the most reliable way for rclone to check for file equality. Many remotes provide some kind of hash value in the file metadata, without needing to compute the value again (see https://rclone.org/overview/#features for supported hash methods).
For local filesystems the hash value is always computed when needed.
With crypt remotes --checksum does not work and rclone will silently skip the hash comparison and only compare file sizes (using --checksum on crypt is equal to using --size-only).
--modify-window is probably your best option to only copy missing or updated files again. You just need to remember to set it for all sync, copy or move commands, since rclone will not update the modified-times on the destination.
I don’t think rclone has a method of updating modified-times during sync or move at the moment.
Doing a rclone move -c a b for non crypt remotes does not update the modified times for equal files either.
Being able to update the modified times in situations like yours would be very useful.
It would indeed and there is even an issue about it. I’m not 100% happy with how it should work though - any ideas on the user interface?
What are you not happy with? --refresh-times and --size-only seems logical?
And I guess that would work both ways i.e just reverting source/target if you for some reason want to get the modified times from the remote.
I guess probably what is needed is --refresh-times and the use of a non modtime based sync, so either --checksum or --size-only.
However I wonder if people want this to just refresh the times and not do any syncing?
If I understand correctly, I would probably use this just to refresh the times in this case. I would prefer to run a sync with --referesh-times and --size-only (due to it being encrypted) rather than using --modify-window every time from now on.
The weird thing is that I had no issues with the timestamps before I moved the files with rsync. And rsync was set to preserve modification/created data – and it did. It matches up with the files in Google Drive, except for the milliseconds. But I didn’t think Linux timestamped files with ms anyway.
So in my mind everything should be identical to “before the move”, but apparently not since I now need to use the --modify-window to make sure files are not unnecessary reuploaded.
OK here is my attempt at some docs for the new feature. It is complicated! Any thoughts?
The --refresh-times flag can be used to update modification times of existing files. This is useful if you uploaded files with the incorrect timestamps and you now wish to correct them.
This can be used any of the sync commands sync, copy or move.
When rclone comes to upload a file it will check if there is an existing file and if it matches with size (no sync flags or --size-only) or size and checksum (--checksum) then instead of re-uploading it, rclone will update the timestamp.
Note that some remotes (eg amazon drive, b2, dropbox, mega, pcloud, webdav) can’t set the modification time without re-uploading the file anyway so this flag is useless with them.
If you are doing a modification time sync (ie not using --checksum or --size-only) rclone will update modification times without --refresh-times provided that the remote supports checksums and the checksums match on the file. crypt remotes do not support checksums so to update times on a crypt remote you will need to use --refresh-times.
I did an attempt trying to be more concise – English is not my first language so be warned. 
The --refresh-times flag is used to update/sync modification times of existing files between source/remote. This is useful if you have files with incorrect timestamps and you now wish to correct them to i.e avoid re-uploading the same files.
This can be used in any of the sync commands – sync, copy or move.
Prior to rclone uploading a file it will first check if it exists on the remote and if it matches in size (no sync flags or --size-only) or both in size and checksum (--checksum). If the file exists rclone will just update the timestamp instead of re-uploading.
Note that some remotes (eg amazon drive, b2, dropbox, mega, pcloud, webdav) can’t set the modification time without re-uploading the file anyway so this flag is useless with them.
If you are doing a modification time sync (ie not using --checksum or --size-only) rclone will update modification times without --refresh-times provided that the remote supports checksums and the checksums match on the file. Crypt remotes do not support checksums so to update times on a crypt remote you will need to use --refresh-times.
I’m not totally sure if I understand this. Does this mean that as long as you are not using crypt rclone will update the modification times? Does that mean that “–refresh-times” is not needed on non-crypt remotes? Hence maybe you should just include a --size-only option in the --refresh-times flag? Since it’s enabled by default.
(if you understand what I mean)
Thank you.
--refresh-times is needed if
--size-only or --checksum and you still want to update the times.I’ve read the feature request log and this thread. It leaves me scratching my head. Who if anyone (and why) did ever say rclone would update the modified time on a file without copying it on any version? There’s nothing anywhere in the docs that eludes to this. This seems to be a myth. Throw all the stuff about crypt out please as that’s just confusing things even more.
Basically as I understand it, if rclone detects there’s a dupe on the destination (no matter what away, checksum, size, date, etc - wharever you configure it to use for dupe detection), it IGNORES that file.
IGNORES = DOES NOTHING! Which includes copying the file or modifying any files anywhere whatsoever.
Am I wrong?
The default sync mode could be better explained in the docs, I agree
https://rclone.org/docs/#c-checksum
If you look at the docs for the --checksum flag it kind of implies how it works!
Anyway the default sync routine is
Yes, I just discovered this option (I’m new to rclone)
–no-update-modtime Don’t update destination mod-time if files identical.
Here’s the question though, can you get it to update the mtime on the destination file if the mtime on the sourve is OLDER. I used this multicloud service and it copied files and set the mtime the date they were copied not the mtime of the source file!
Seems like the -c option (skip on checksum and size) would work? But I’m guessing rclones internal rules are that it won’ t modify the mtime on the dest if it’s newer than the source!
I think you probably want this issue: https://github.com/ncw/rclone/issues/1505 which isn’t done yet