Some "google documents" are always re-copied, even if they have not been modified

What is the problem you are having with rclone?

Some "google documents" are always re-copied, even if they have not been modified.

Others are not.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: darwin 12.6.3 (64 bit)
- os/kernel: 21.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.19.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync src:/test/ dst:/test/ --drive-server-side-across-configs --progress -vv --log-file=/Users/vicmarto/test.txt

The rclone config contents with secrets removed.

[src]
type = drive
client_id = REMOVED
client_secret = REMOVED
scope = drive
token = REMOVED
team_drive = 
root_folder_id = REMOVED

[dst]
type = drive
client_id = REMOVED
client_secret = REMOVED
scope = drive
token = REMOVED
team_drive = 

The problem is that some "google documents" are always copied again, sync after sync. Even if they have not been modified.

Others, on the other hand, are not.

For example, the log of a first synchronization. Where of course, everything is copied, as expected:

2023/02/14 13:35:10 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "src:/test/" "dst:/test/" "--drive-server-side-across-configs" "--progress" "-vv" "--log-file=/Users/vicmarto/test.txt"]
2023/02/14 13:35:10 DEBUG : Creating backend with remote "src:/test/"
2023/02/14 13:35:10 DEBUG : Using config file from "/Users/vicmarto/.config/rclone/rclone.conf"
2023/02/14 13:35:10 DEBUG : src: detected overridden config - adding "{Db_Y9}" suffix to name
2023/02/14 13:35:10 DEBUG : fs cache: renaming cache item "src:/test/" to be canonical "src{Db_Y9}:test"
2023/02/14 13:35:10 DEBUG : Creating backend with remote "dst:/test/"
2023/02/14 13:35:10 DEBUG : dst: detected overridden config - adding "{Db_Y9}" suffix to name
2023/02/14 13:35:11 DEBUG : Google drive root 'test': 'root_folder_id = 0AJdCllWlQvjLUk9PVA' - save this in the config to speed up startup
2023/02/14 13:35:11 DEBUG : fs cache: renaming cache item "dst:/test/" to be canonical "dst{Db_Y9}:test"
2023/02/14 13:35:12 DEBUG : Google drive root 'test': Waiting for checks to finish
2023/02/14 13:35:12 DEBUG : Google drive root 'test': Waiting for transfers to finish
2023/02/14 13:35:13 DEBUG : audio.mp3: md5 = 50d9a2df0565306b23e625f3d22ddbcf OK
2023/02/14 13:35:13 INFO  : audio.mp3: Copied (server-side copy)
2023/02/14 13:35:16 DEBUG : Google drive root 'test': Sleeping for 2s before setting the modtime to work around drive bug - see #4517
2023/02/14 13:35:18 DEBUG : Google drive root 'test': Sleeping for 2s before setting the modtime to work around drive bug - see #4517
2023/02/14 13:35:18 DEBUG : doesnt_re-copy.docx: Dst hash empty - aborting Src hash check
2023/02/14 13:35:18 DEBUG : doesnt_re-copy.docx: Src hash empty - aborting Dst hash check
2023/02/14 13:35:18 INFO  : doesnt_re-copy.docx: Copied (server-side copy)
2023/02/14 13:35:21 DEBUG : always_re-copy.docx: Dst hash empty - aborting Src hash check
2023/02/14 13:35:21 DEBUG : always_re-copy.docx: Src hash empty - aborting Dst hash check
2023/02/14 13:35:21 INFO  : always_re-copy.docx: Copied (server-side copy)
2023/02/14 13:35:21 DEBUG : Waiting for deletions to finish
2023/02/14 13:35:21 INFO  : 
Transferred:   	    6.603 KiB / 6.603 KiB, 100%, 845 B/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:        10.7s

2023/02/14 13:35:21 DEBUG : 13 go routines active

In the next sync it is to be expected that nothing will be copied again. Since nothing has been synchronized, not even opened!
But that is not what happens:

2023/02/14 13:35:21 DEBUG : 13 go routines active
2023/02/14 13:37:18 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "src:/test/" "dst:/test/" "--drive-server-side-across-configs" "--progress" "-vv" "--log-file=/Users/vicmarto/test.txt"]
2023/02/14 13:37:18 DEBUG : Creating backend with remote "src:/test/"
2023/02/14 13:37:18 DEBUG : Using config file from "/Users/vicmarto/.config/rclone/rclone.conf"
2023/02/14 13:37:18 DEBUG : src: detected overridden config - adding "{Db_Y9}" suffix to name
2023/02/14 13:37:19 DEBUG : fs cache: renaming cache item "src:/test/" to be canonical "src{Db_Y9}:test"
2023/02/14 13:37:19 DEBUG : Creating backend with remote "dst:/test/"
2023/02/14 13:37:19 DEBUG : dst: detected overridden config - adding "{Db_Y9}" suffix to name
2023/02/14 13:37:19 DEBUG : Google drive root 'test': 'root_folder_id = 0AJdCllWlQvjLUk9PVA' - save this in the config to speed up startup
2023/02/14 13:37:19 DEBUG : fs cache: renaming cache item "dst:/test/" to be canonical "dst{Db_Y9}:test"
2023/02/14 13:37:20 DEBUG : Google drive root 'test': Waiting for checks to finish
2023/02/14 13:37:20 DEBUG : doesnt_re-copy.docx: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023/02/14 13:37:20 DEBUG : doesnt_re-copy.docx: Unchanged skipping
2023/02/14 13:37:20 DEBUG : audio.mp3: Size and modification time the same (differ by 0s, within tolerance 1ms)
2023/02/14 13:37:20 DEBUG : audio.mp3: Unchanged skipping
2023/02/14 13:37:20 DEBUG : always_re-copy.docx: Modification times differ by 13m48.557s: 2023-02-14 04:21:29.523 +0000 UTC, 2023-02-14 04:35:18.08 +0000 UTC
2023/02/14 13:37:20 DEBUG : always_re-copy.docx: Dst hash empty - aborting Src hash check
2023/02/14 13:37:20 DEBUG : always_re-copy.docx: Src hash empty - aborting Dst hash check
2023/02/14 13:37:20 DEBUG : Google drive root 'test': Waiting for transfers to finish
2023/02/14 13:37:25 DEBUG : Google drive root 'test': Sleeping for 2s before setting the modtime to work around drive bug - see #4517
2023/02/14 13:37:28 DEBUG : always_re-copy.docx: Dst hash empty - aborting Src hash check
2023/02/14 13:37:28 DEBUG : always_re-copy.docx: Src hash empty - aborting Dst hash check
2023/02/14 13:37:28 INFO  : always_re-copy.docx: Copied (server-side copy)
2023/02/14 13:37:28 DEBUG : Waiting for deletions to finish
2023/02/14 13:37:28 INFO  : 
Transferred:   	          off / off, -, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Transferred:            1 / 1, 100%
Elapsed time:        10.4s

2023/02/14 13:37:28 DEBUG : 9 go routines active

"always_re-copy.docx" is recopied, when it shouldn't be!

On the other hand, "doesnt_re-copy.docx" is not recopied, as it should.

Why some "google document" are re-copied and others are not? It seems to be a problem with "Modification times differ by 13m48.557s". But if you look at the sync time, this modification time is impossible.

I would appreciate any help, thanks.

The modification time appears to be not what rclone is expecting here. I suspect Google is reading the modification time from the document itself rather than in the metadata where rclone sets it.

You can try using the --update flag - that might help in this case.

Thanks @ncw for your answer.

Using --update, it does work.

On the subject of the modification time, would it be possible to update also the modification time of the document itself, apart from the metadata?

There is no option to do this in the drive API. I guess it might be possible in the Google docs API but rclone doesn't implement that.

I see. Thank you very much ncw!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.