RClone "SYNC" Download all files again and replace exist

Massaguana · November 30, 2019, 6:48am

is --size-only an solution for me?

What I find strange is that this problem does not occur with all files

Massaguana · November 30, 2019, 1:26pm

I use local an Linux Based NAS from Thecus, Full Crypted RAID.

ncw · November 30, 2019, 6:05pm

If you try to

rclone touch /path/to/troublesome/file/on/your/NAS

does the timestamp change to the local time?

Maybe this is a bug in the NAS?

Massaguana · February 10, 2020, 6:08pm

Yes, the timestamp changes

Massaguana · February 11, 2020, 8:58am

How do I use "touch" for whole folders?

ncw · February 11, 2020, 11:41am

touch only works on files at the moment, so you'd have to do a bit of scripting, something like

rclone lsf --files-only /path/to/dir | xargs -i rclone touch {}

Massaguana · February 11, 2020, 1:11pm

it looks like to update the timestamp don´t prevent this issue...

i have delete the complete local folder and start the rclone sync process. Alle file were downloaded

Thats what i see now? (Mac OS Terminal)

ls -la
total 73355328
drwx------ 1 massaguana staff 16384 11 Feb 11:21 .
drwx------ 1 massaguana staff 16384 11 Feb 11:04 ..
-rwx------ 1 massaguana staff 1894183065 11 Feb 11:56 Andromeda (2000) - S01E01 - Die lange Nacht.mkv
-rwx------ 1 massaguana staff 1885882806 11 Feb 11:55 Andromeda (2000) - S01E02 - Fiat Lux.mkv
-rwx------ 1 massaguana staff 1663141003 11 Feb 11:55 Andromeda (2000) - S01E03 - Die Erleuchtung.mkv
-rwx------ 1 massaguana staff 1932044655 11 Feb 12:01 Andromeda (2000) - S01E04 - D Minus Zero.mkv
-rwx------ 1 massaguana staff 1541009545 11 Feb 12:00 Andromeda (2000) - S01E05 - Doppelhelix.mkv
-rwx------ 1 massaguana staff 1712486148 11 Feb 11:57 Andromeda (2000) - S01E06 - Die grosse Schlacht.mkv
-rwx------ 1 massaguana staff 1689401982 11 Feb 11:58 Andromeda (2000) - S01E07 - Philosophie des Todes.mkv
-rwx------ 1 massaguana staff 1816033950 11 Feb 12:04 Andromeda (2000) - S01E08 - Reise in die Vergangenheit.mkv
-rwx------ 1 massaguana staff 1758147405 11 Feb 12:00 Andromeda (2000) - S01E09 - Planet der Verdammten.mkv
-rwx------ 1 massaguana staff 1723306439 11 Feb 12:01 Andromeda (2000) - S01E10 - Tödliches Gipfeltreffen.mkv
-rwx------ 1 massaguana staff 1863789981 11 Feb 12:05 Andromeda (2000) - S01E11 - Durchs Auge mitten ins Herz.mkv
-rwx------ 1 massaguana staff 1671886044 11 Feb 12:06 Andromeda (2000) - S01E12 - Pax Magellanic.mkv
-rwx------ 1 massaguana staff 1562712837 11 Feb 12:02 Andromeda (2000) - S01E13 - Der heilige Gral.mkv
-rwx------ 1 massaguana staff 1703287166 11 Feb 12:05 Andromeda (2000) - S01E14 - Genie an Bord.mkv
-rwx------ 1 massaguana staff 1531065956 11 Feb 12:04 Andromeda (2000) - S01E15 - Erzwungene Einsichten.mkv
-rwx------ 1 massaguana staff 1657447486 11 Feb 12:10 Andromeda (2000) - S01E16 - Die Summe aller Teile.mkv
-rwx------ 1 massaguana staff 1707261234 11 Feb 12:06 Andromeda (2000) - S01E17 - Das Tagebuch des Hasturi.mkv
-rwx------ 1 massaguana staff 1569985249 11 Feb 12:10 Andromeda (2000) - S01E18 - Die Schlange im Paradies.mkv
-rwx------ 1 massaguana staff 1663938145 11 Feb 12:10 Andromeda (2000) - S01E19 - Eine Frage der Ehre.mkv
-rwx------ 1 massaguana staff 1469901877 11 Feb 12:07 Andromeda (2000) - S01E20 - Erzengel Gabriel.mkv
-rwx------ 1 massaguana staff 1642057850 11 Feb 12:09 Andromeda (2000) - S01E21 - Die Odyssee.mkv
-rwx------ 1 massaguana staff 1886743647 11 Feb 12:11 Andromeda (2000) - S01E22 - Feinde an Bord.mkv

Now when i start rclone sync again, rclone will replace all files...

i have test "touch" for all files... rclone still replace all files...

EDIT: If it should be helpful so you can look Remote and test

Massaguana · February 11, 2020, 2:02pm

I don't understand why this only happens with some TV-Shows but not with all TV-Shows or Movies

ncw · February 11, 2020, 2:40pm

Try adding --modify-window 1000000000s to disable the time checking - does that help?

Massaguana · February 12, 2020, 1:28pm

Currently it seems to work

ncw · February 12, 2020, 8:51pm

That is good. I think there is something wrong with the modtime setting in your destination remote. I don't know why that is since your destination is a local disk isn't it?

Massaguana · February 13, 2020, 1:20pm

Yes and No, it is a network drive that I mount via SMB/AFP under MacOS. But I have also tested it via SSH directly on the NAS, here it is a local hard disk (RAID-6).
I can still try and report on a local hard drive from MacOS.

Massaguana · February 13, 2020, 1:43pm

okay, i test with an local drive with my MacOS Computer are done... it works... the file time are correct

looks like an issue with RClone an Thecus... to open an call with Thecus support i need more informations. Can you help me with this?

ncw · February 13, 2020, 2:00pm

Can you do an experiment?

Can you do this and post the results?

rclone touch -t "2006-01-02T15:04:05" /path/to/file/server/file.txt
rclone lsl /path/to/file/server/file.txt
stat /path/to/file/server/file.txt
rclone touch -t "2016-02-03T16:05:06" /path/to/file/server/file.txt
rclone lsl /path/to/file/server/file.txt
stat /path/to/file/server/file.txt

Massaguana · February 13, 2020, 2:48pm

Here is the result. Over SSH locally on the NAS

N10850:~# rclone --version
rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7
N10850:~# rclone lsl "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
1894183065 2019-09-16 14:33:40.000000000 Andromeda (2000) - S01E01 - Die lange Nacht.mkv
N10850:~# rclone touch -t "2006-01-02T15:04:05" "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
N10850:~# rclone lsl "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
1894183065 2006-01-02 16:04:05.000000000 Andromeda (2000) - S01E01 - Die lange Nacht.mkv
N10850:~# stat "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
  File: "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
  Size: 1894183065	Blocks: 3699592    IO Block: 4096   regular file
Device: 732h/1842d	Inode: 146178854687  Links: 1
Access: (0776/-rwxrwxrw-)  Uid: ( 1000/     AKF)   Gid: (  100/   users)
Access: 2006-01-02 16:04:05.000000000
Modify: 2006-01-02 16:04:05.000000000
Change: 2020-02-13 15:44:28.000000000

N10850:~# rclone touch -t "2016-02-03T16:05:06" "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
N10850:~# rclone lsl "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
1894183065 2016-02-03 17:05:06.000000000 Andromeda (2000) - S01E01 - Die lange Nacht.mkv
N10850:~# stat "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
  File: "/raid0/data/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht.mkv"
  Size: 1894183065	Blocks: 3699592    IO Block: 4096   regular file
Device: 732h/1842d	Inode: 146178854687  Links: 1
Access: (0776/-rwxrwxrw-)  Uid: ( 1000/     AKF)   Gid: (  100/   users)
Access: 2016-02-03 17:05:06.000000000
Modify: 2016-02-03 17:05:06.000000000
Change: 2020-02-13 15:46:00.000000000

N10850:~#

ncw · February 13, 2020, 4:47pm

That worked ok...

Did it run over SMB/AFP? If not can you try that?

What if you create a file on your normal disk and copy it with rclone copyto - does that preserve the datestamp?

echo hello > /tmp/file.txt
rclone copy /tmp/file.txt /path/to/network/share
stat /tmp/file.txt
stat /path/to/network/share/file.txt

Massaguana · February 14, 2020, 6:53am

The first test are over SSH locally on the NAS...

Here are the result over SMB/AFP:

massaguana$ rclone --version
rclone v1.51.0
- os/arch: darwin/amd64
- go version: go1.13.7
Titos-Mac-mini:~ massaguana$ rclone touch -t "2006-01-02T15:04:05" "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
Titos-Mac-mini:~ massaguana$ rclone lsl "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
1894183065 2006-01-02 16:04:05.000000000 Andromeda (2000) - S01E01 - Die lange Nacht .mkv
Titos-Mac-mini:~ massaguana$ stat "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
855638024 146178854687 -rwx------ 1 massaguana staff 0 1894183065 "Jan  2 16:04:05 2006" "Jan  2 16:04:05 2006" "Jan  2 16:04:05 2006" "Jan  2 16:04:05 2006" 4096 3699584 0 /Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv
Titos-Mac-mini:~ massaguana$ rclone touch -t "2016-02-03T16:05:06" "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
Titos-Mac-mini:~ massaguana$ rclone lsl "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
1894183065 2016-02-03 17:05:06.000000000 Andromeda (2000) - S01E01 - Die lange Nacht .mkv
Titos-Mac-mini:~ massaguana$ stat "/Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv" 
855638024 146178854687 -rwx------ 1 massaguana staff 0 1894183065 "Feb  3 17:05:06 2016" "Feb  3 17:05:06 2016" "Feb  3 17:05:06 2016" "Jan  2 16:04:05 2006" 4096 3699584 0 /Volumes/TV-Serien/Andromeda (2000)/Season 01/Andromeda (2000) - S01E01 - Die lange Nacht .mkv
Titos-Mac-mini:~ massaguana$

And the Result to create am file copy it and check the timestamps...

rclone -v copy /tmp/file.txt "/Volumes/TV-Serien/Andromeda (2000)/Season 01/file.txt" 
2020/02/14 07:59:08 INFO  : file.txt: Copied (new)
2020/02/14 07:59:08 INFO  : 
Transferred:   	         6 / 6 Bytes, 100%, 14 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         0.4s

Titos-Mac-mini:tmp massaguana$ stat /tmp/file.txt
16777220 17418919 -rw-r--r-- 1 massaguana wheel 0 6 "Feb 14 07:59:08 2020" "Feb 14 07:58:36 2020" "Feb 14 07:58:36 2020" "Feb 14 07:55:39 2020" 4096 8 0 /tmp/file.txt
Titos-Mac-mini:tmp massaguana$ stat "/Volumes/TV-Serien/Andromeda (2000)/Season 01/file.txt" 
855638024 4323328544 drwx------ 1 massaguana staff 0 16384 "Feb 14 07:59:08 2020" "Feb 14 07:59:08 2020" "Feb 14 07:59:08 2020" "Feb 14 07:59:08 2020" 4096 32 0 /Volumes/TV-Serien/Andromeda (2000)/Season 01/file.txt

ncw · February 14, 2020, 8:33am

That is where the problem is...

We asked to set the date 2016-01-02

However rclone reads it back as 2016-02-03 which is presumably the date that it was before from the previous test.

and so does stat for the modified time we are interested in.

So I think this is a bug but I'm not sure where...

Can you set the times of files using the touch command on macOS something like

touch -d YYYY-MM-DD /path/to/file
stat -x /path/to/file

and see what that does?

Massaguana · February 14, 2020, 8:55am

touch -d told me that "-d" is an illegal option...

touch -d "2006-01-02T15:04:05" /tmp/file.txt

touch: illegal option -- d
usage:
touch [-A [-][[hh]mm]SS] [-acfhm] [-r file] [-t [[CC]YY]MMDDhhmm[.SS]] file ...

ncw · February 14, 2020, 9:18am

Ah, try

touch -t 201212211111 /path
stat -x /path