RClone re-copying just synced files

Help and Support
1

People,

I only use RClone very occasionally and I have just updated to rclone-v1.36-linux-amd64. I use this command with switches that were used to solve previous problems:

rclone -v sync --drive-full-list=false --drive-formats “ods,odt” gd: /home/phil/google_drive/

I just ran it again for the first time in quite a while and of course there were quite a few new and changed files - I didn’t watch the output too closely. As an exercise, I immediately ran the script again and it seems to be copying everything again?

I see:

Copied (replaced existing)

on nearly all the files and these CAN’T need re-copying - in particular some shared .mov files take forever to copy . .

Am I stuffing something up or missing something?

Thanks,

Phil.

2

Hello,

I have the same problem. I copied from googledrive to googledrivecrypt. Now I want to user rclone to copy new files but it wants to upload all the files again

2017/07/08 09:48:02 INFO : xxxxx.mkv: Copied (replaced existing)
2017/07/08 09:48:03 INFO : xxxxx.nl.srt: Copied (replaced existing)
2017/07/08 09:48:04 INFO : xxxxx.en.srt: Copied (replaced existing)

Any ideas?

3

The first thing to do is to use rclone dedupe on the source and destination.

Next try the the latest beta and if you are still getting problems then please make a new issue on github with a full log with -vv.

4

How can I do this on source AND destination?

if I give this command :
rclone dedupe --dedupe-mode interactive /volume1/video/Media GoogleDriveCrypt:/

I get:

Unknown mode for dedupe “/volume1/video/Media”.

5

You said

So I assumed you had drive on both ends. If not just do it on the drive end.

6

Yes, sorry. I copied the files from GD to GDcrypt.

But the files are also (some of them) stored on my NAS. Now I want to upload new files every night from my NAS to my GDcrypt. But rclone wants to upload all the files. Could it be the proces GD -> GDcrypt modified the files a bit?

Update: No dupe files found:

xxxxx@DiskStation:~$ rclone dedupe --dedupe-mode interactive -v GoogleDriveCrypt:
2017/07/08 10:18:01 INFO : Encrypted Google drive root ‘PlexEncrypt’: Modify window is 1ms
2017/07/08 10:18:01 INFO : Encrypted Google drive root ‘PlexEncrypt’: Looking for duplicates using interactive mode.
xxxxx@DiskStation:~$

7

I think I see the problem. Timezones… the files it want to change has a time difference of more then 1 hour (aka 22;22 verus 23:22). Can I disable the time mod check on crypt transfer?

I added the --size-only and now I have no problems anymore.

8

As the OP - adding that switch didn’t fix my problem . . (I am in Australian EST) . .

9

@philip_rhoades - did you try rclone dedupe gd: ?

10

It deleted a few things but it does not fix the problem . . now the relevant line in my script is:

rclone -v sync --size-only --drive-full-list=false --drive-formats “ods,odt” gd:/$dir /home/phil/google_drive/$dir/

but there is still a problem with recopying:

[phil@phil ~]$ bin/rclone_gd.sh

2017/07/09 16:37:04 INFO : Local file system at /home/phil/google_drive/3D: Modify window is 1ms
2017/07/09 16:37:06 INFO : Local file system at /home/phil/google_drive/3D: Waiting for checks to finish
2017/07/09 16:37:06 INFO : Local file system at /home/phil/google_drive/3D: Waiting for transfers to finish
2017/07/09 16:37:08 INFO : BlenderHotkeysCheatsheet.odt: Copied (new)
2017/07/09 16:37:08 INFO : Waiting for deletions to finish
2017/07/09 16:37:08 INFO :
Transferred: 14.012 kBytes (2.849 kBytes/s)
Errors: 0
Checks: 0
Transferred: 1
Elapsed time: 4.9s

[phil@phil ~]$ bin/rclone_gd.sh

2017/07/09 16:37:17 INFO : Local file system at /home/phil/google_drive/3D: Modify window is 1ms
2017/07/09 16:37:19 INFO : Local file system at /home/phil/google_drive/3D: Waiting for checks to finish
2017/07/09 16:37:19 INFO : Local file system at /home/phil/google_drive/3D: Waiting for transfers to finish
2017/07/09 16:37:21 INFO : BlenderHotkeysCheatsheet.odt: Copied (replaced existing)
2017/07/09 16:37:21 INFO : Waiting for deletions to finish
2017/07/09 16:37:21 INFO :
Transferred: 14.013 kBytes (2.853 kBytes/s)
Errors: 0
Checks: 1
Transferred: 1
Elapsed time: 4.9s

11

I know what the problem with this is… When rclone comes to download a google doc it asks google what the size of the doc would be in the desired format. Eg for BlenderHotkeysCheatsheet.odt. However sometimes google returns different answers, I don’t know why, hence the re-download.

You’ll find that sometimes it does it, and sometimes it doesn’t which isn’t ideal, but I couldn’t find a way to fix it.

You can add --ignore-size to your sync but that isn’t compatible with --size-only. I think I’d probably accept that occasionally rclone will do this!

I’ve added this to the drive docs to help people in future

Duplicated files

Sometimes, for no reason I’ve been able to track down, drive will
duplicate a file that rclone uploads. Drive unlike all the other
remotes can have duplicated files.

Duplicated files cause problems with the syncing and you will see
messages in the log about duplicates.

Use rclone dedupe to fix duplicated files.

Note that this isn’t just a problem with rclone, even Google Photos on
Android duplicates files on drive sometimes.

Rclone appears to be re-copying files it shouldn’t

There are two possible reasons for rclone to recopy files which
haven’t changed to Google Drive.

The first is the duplicated file issue above - run rclone dedupe and
check your logs for duplicate object or directory messages.

The second is that sometimes Google reports different sizes for the
Google Docs exports which will cause rclone to re-download Google Docs
for no apparent reason. --ignore-size is a not very satisfactory
work-around for this if it is causing you a lot of problems.

1 Like