Copying/syncing from GDrive to local disk with files that are identical in size and name results in many GB of transfer

What is the problem you are having with rclone?

I did an rclone sync from crypt-GDrive to a local ext4 disk while connected to faster internet than my home (which is just an unlimited cellphone plan hooked to a router). I then used rclone to sync that ext4 drive to another larger btrfs ssd where my local copy of the data will live. I did this a second time to make sure I had everything and pulled down another 300GB.
I'm trying to do a daily sync to keep the daily data transfer low, but somewhere in the sync chain above the mtimes of many files were off by a few ms and now unless I use --size-only , it will try to download 100GB or more. Is there a way to fix the mtime of files with an identical name and size? I want to use --track-renames --track-rename-strategy modtime, since local and remote checksum does not match?

--modify-window 1s is a work around, but I also want to "true-up" between remote and local times.

Alternatively, is there a way for --track-renames to rehash each local file with a gdrive compatible?

Run the command 'rclone version' and share the full output of the command.

rclone --version
rclone v1.61.1
- os/version: debian 11.6 (64 bit)
- os/kernel: 5.10.0-20-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.19.4
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

GDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --transfers=1 --track-renames --track-renames-strategy modtime --max-duration 4h --cutoff-mode=SOFT --log-level=INFO --progress  --dscp LE content-crypt: /mnt/content/```

The rclone config contents with secrets removed.

[content]
type = drive
client_id = xx
client_secret = xx
scope = drive
auth_owner_only = false
use_trash = false
skip_gdocs = false
skip_checksum_gphotos = true
shared_with_me = false
trashed_only = false
allow_import_name_change = false
alternate_export = false
chunk_size = 128M
acknowledge_abuse = true
keep_revision_forever = false
server_side_across_configs = true
stop_on_upload_limit = false
token = xx
team_drive = xx
root_folder_id = 

[content-crypt]
type = crypt
remote = content-music:
filename_encryption = standard
directory_name_encryption = true
password = x-xx-xx
password2 = xx
server_side_across_configs = true

A log from the command with the -vv flag

2023/01/13 05:19:06 DEBUG : Media/Music/Kendrick Lamar/To Pimp a Butterfly (2015)/disc.png: Modification times differ by -784ms: 2018-08-11 19:36:30.784 +0000 UTC, 2018-08-11 11:36:30 -0800 AKDT
2023/01/13 05:19:06 DEBUG : Media/Music/Kendrick Lamar/To Pimp a Butterfly (2015)/discart.png: Modification times differ by -596ms: 2019-11-21 05:36:15.596 +0000 UTC, 2019-11-20 20:36:15 -0900 AKST
2023/01/13 05:19:06 DEBUG : Media/Music/Kesha/Children of the Revolution (2020)/album.nfo: Modification times differ by -615ms: 2022-01-23 23:16:23.615 +0000 UTC, 2022-01-23 14:16:23 -0900 AKST
2023/01/13 05:19:06 DEBUG : Media/Music/Kesha/Children of the Revolution (2020)/folder.jpg: Modification times differ by -688ms: 2022-01-23 23:16:23.688 +0000 UTC, 2022-01-23 14:16:23 -0900 AKST

Rclone would normally fix them for you if the source supported checksums.

--modify-window is the correct fix for this problem. Some filing systems (BTFS maybe?) only store timestamps to a precision of 1 second so you'll see differences in modtime of < 1s.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.