Using Rclone with incrond/inotifywait fires off multiple files

[bostonmacosx (R H)]

What is the problem you are having with rclone?

So when I run this with incrond incrond fires off rclone 3 time for a "touch file.txt" in the watched folder. The result is 3 files in the folder on Google Drive not 1.

Even when I run the straight rclone below it does not corect the multiple copiies of the file int he folder...says it just ignores them? not sure why.

The below command is the incrond which use the rclone is embedded in the statement.

If I iremove the IN_ATTRIB then I get 1 file..if I add IN_CREATE I will get 3 files.....just not the expected behavioir.

/home/mounts/localdisk/LINUXBACKUP/binlog/ IN_ATTRIB,IN_CLOSE_WRITE,IN_DELETE rclone sync /home/mounts/localdisk/LINUXBACKUP/binlog/ GoogleDLinux:LINUXBACKUP/BINLOGS/ --log-level DEBUG --log-file /tmp/incron.txt

What is your rclone version (output from rclone version)

rclone v1.49.5

• os/arch: linux/amd64

• go version: go1.12.10

Which OS you are using and how many bits (eg Windows 7, 64 bit)

ubuntu 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/10/27 14:54:17 DEBUG : rclone: Version "v1.49.5" starting with parameters ["rclone" "sync" "/home/mounts/localdisk/LINUXBACKUP/binlog/" "GoogleDLinux:LINUXBACKUP/BINLOGS/" "--log-level" "DEBUG" "--log-file" "/tmp/incron.txt"]
2019/10/27 14:54:17 DEBUG : Using config file from "/home/bostonmacosx/.config/rclone/rclone.conf"
2019/10/27 14:54:18 NOTICE: test.txt: Duplicate object found in destination - ignoring
2019/10/27 14:54:18 INFO : Google drive root 'LINUXBACKUP/BINLOGS': Waiting for checks to finish
2019/10/27 14:54:18 DEBUG : mysql-bin.000002: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000002: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.000001: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000003: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000003: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.000001: Unchanged skipping
2019/10/27 14:54:18 DEBUG : test.txt: Size and modification time the same (differ by -675.131µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000004: Size and modification time the same (differ by -316.31µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.index: Size and modification time the same (differ by -582.75µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000004: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.000007: Size and modification time the same (differ by -582.75µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000007: Unchanged skipping
2019/10/27 14:54:18 DEBUG : test.txt: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.000006: Size and modification time the same (differ by -732.042µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000006: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.index: Unchanged skipping
2019/10/27 14:54:18 DEBUG : mysql-bin.000005: Size and modification time the same (differ by -991.616µs, within tolerance 1ms)
2019/10/27 14:54:18 DEBUG : mysql-bin.000005: Unchanged skipping
2019/10/27 14:54:18 INFO : Google drive root 'LINUXBACKUP/BINLOGS': Waiting for transfers to finish
2019/10/27 14:54:18 INFO : Waiting for deletions to finish
2019/10/27 14:54:18 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 9 / 9, 100%
Transferred: 0 / 0, -
Elapsed time: 0s

2019/10/27 14:54:18 DEBUG : 5 go routines active
2019/10/27 14:54:18 DEBUG : rclone: Version "v1.49.5" finishing with parameters ["rclone" "sync" "/home/mounts/localdisk/LINUXBACKUP/binlog/" "GoogleDLinux:LINUXBACKUP/BINLOGS/" "--log-level" "DEBUG" "--log-file" "/tmp/incron.txt"]

[thestigma]

Is it a shared drive (previously called Teamdrive) ?
Could you share your mounting command ?

I'm not sure I understand what you are trying to do here, but on Gdrive duplicates are allowed (by the system, though rclone tries it's best not to make them). Still it can happen sometimes, usually if there is some sort of rapid sequence of overlapping events.

Likely what is happening is that a change is being made, but before the change can be detected (and in this case registered in the mounts VFS cache) another change to the same file happens and rclone just ends up making a new file rather than modifying the existing.

In case of identical duplicates I believe the newest file is always used and the others are ignored. This is just the intended failsafe to a situation that should not have happened in the first place. (although perhaps it should not be ignoring them for a sync operation...)

You can run rclone dedupe remotename: to clean these up if they happen (usually can do it automatically but may ask the user if the right answer is not obvious, such as if the filesize is not identical).

Transfers in this case are not atomic, they rely on the information listings they gathered at the start of the transfer, and if the situation changes while it's working the result can be imperfect (such as creating dupes). A common way of inducing dupes it trying run 2 or more concurrent transfers into overlapping locations for example...

I'm not sure to what extent it is possible to prevent this further or if it's a limitation we will have to live with,

@ncw Is this relevant for you perhaps given the reproduction method is so concrete?

[bostonmacosx (R H)]

I think it was because the IN_ definitions within the inrond were firing off multiple times at the same tiime. I learned there is a way to prevent multiple calls at once....I'm playing around with these settings now.

I notw only have IN_CLOSE_WRITE,IN_DELETE,IN_NO_FOR_LOOP as the settings. that seems to be working for now.....

[ncw (Nick Craig-Wood)]

I think it was because the IN_ definitions within the inrond were firing off multiple times at the same tiime.

That would certainly explain the duplicates. Rclone is quite likely to create duplicates if there are multiple rclones all trying to copy the same thing.