Unable to to utilize modtime and size instead of checksums during sync.Very slow checksum checking uploading to Google Drive

Help and Support
1

What is the problem you are having with rclone?

I have some very big files (each one > 150GB) to sync with Google Drive. I need to sync checking only modtime and size of the files, skipping checksum, because checksum test takes very long time. I've tried with differents flags, but it seems that rclone always test for cheksum, even if the modtime is different. The checksum test takes over 3 hours, and the copy 2:30 hours.

Run the command 'rclone version' and share the full output of the command.

Version "v1.62.2"

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -vv "e:\Backup" gdrive:Backup\xxxx --drive-chunk-size 1024M  --transfers 16 --check-first --modify-window 10s --ignore-checksum

The rclone config contents with secrets removed.

- type: drive
- client_id: xxx
- client_secret: xxx
- scope: drive
- token: {"access_token":}
- team_drive:

A log from the command with the -vv flag

2023/05/17 21:12:47 INFO  : Google drive root 'Backup/ELCA': Running all checks before starting transfers
2023/05/17 21:13:47 INFO  : 
Transferred:   	          0 B / 8.451 MiB, 0%, 0 B/s, ETA -
Checks:                16 / 20, 80%
Transferred:            0 / 9, 0%
Elapsed time:       1m1.3s
Checking:
 * ELCA-SQL/FULL_C2977EA4…8986-4092FD2540EB.VMRS: checking
 * ELCA-SQL/FULL_C2977EA4…rd Disks/ELCA-SQL.vhdx: checking
 * ELCA-TERMINAL/FULL_9A5…sks/ELCA-TERMINAL.vhdx: checking
 * ELCA-TERMINAL/FULL_9A5…B97C-D0F8E34EE7E5.VMRS: checking

2023/05/18 00:21:47 INFO  : 
Transferred:   	          0 B / 214.012 GiB, 0%, 0 B/s, ETA -
Checks:                19 / 20, 95%
Transferred:            0 / 12, 0%
Elapsed time:     3h9m1.3s
Checking:
 * ELCA-TERMINAL/FULL_9A5…sks/ELCA-TERMINAL.vhdx: checking

2023/05/18 00:22:42 INFO  : Google drive root 'Backup/ELCA': Checks finished, now starting transfers
2023/05/18 00:22:46 INFO  : ELCA-TERMINAL/FULL_9A5C0CA6-8BA6-484C-B97C-D0F8E34EE7E5/Virtual Machines/9A5C0CA6-8BA6-484C-B97C-D0F8E34EE7E5.vmgs: Copied (replaced existing)
2023/05/18 00:22:47 INFO  : 
Transferred:   	  185.435 MiB / 414.016 GiB, 0%, 34.479 MiB/s, ETA 3h24m50s
Checks:                20 / 20, 100%
Transferred:            1 / 13, 8%
Elapsed time:    3h10m1.3s

2023/05/18 02:52:44 INFO  : ELCA-TERMINAL/FULL_9A5C0CA6-8BA6-484C-B97C-D0F8E34EE7E5/Virtual Hard Disks/ELCA-TERMINAL.vhdx: Copied (replaced existing)
2023/05/18 02:52:44 INFO  : 
Transferred:   	  414.016 GiB / 414.016 GiB, 100%, 52.524 MiB/s, ETA 0s
Checks:                20 / 20, 100%
Transferred:           13 / 13, 100%
Elapsed time:   5h39m58.1s
3

There should have been a bit of DEBUG earlier in the log saying why rclone wanted to upload the file - can you post that please?

4

Here is the full log

E:\rclone>e:\rclone\rclone sync -vv -n "e:\Iperius Backup\ELCA-SQL\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\Virtual Hard Disks" "gdrive:Backup\ELCA\ELCA-SQL\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\Virtual Hard Disks" --drive-chunk-size 1024M  --transfers 16 --ignore-checksum
2023/05/19 15:03:36 DEBUG : rclone: Version "v1.62.2" starting with parameters ["e:\\rclone\\rclone" "sync" "-vv" "-n" "e:\\Iperius Backup\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks" "gdrive:Backup\\ELCA\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks" "--drive-chunk-size" "1024M" "--transfers" "16" "--ignore-checksum"]
2023/05/19 15:03:36 DEBUG : Creating backend with remote "e:\\Iperius Backup\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks"
2023/05/19 15:03:36 DEBUG : Using config file from "e:\\rclone\\rclone.conf"
2023/05/19 15:03:36 DEBUG : fs cache: renaming cache item "e:\\Iperius Backup\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks" to be canonical "//?/e:/Iperius Backup/ELCA-SQL/FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB/Virtual Hard Disks"
2023/05/19 15:03:36 DEBUG : Creating backend with remote "gdrive:Backup\\ELCA\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks"
2023/05/19 15:03:36 DEBUG : gdrive: detected overridden config - adding "{kysdd}" suffix to name
2023/05/19 15:03:38 DEBUG : fs cache: renaming cache item "gdrive:Backup\\ELCA\\ELCA-SQL\\FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB\\Virtual Hard Disks" to be canonical "gdrive{kysdd}:Backup/ELCA/ELCA-SQL/FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB/Virtual Hard Disks"
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.mrt: Modification times differ by -395.279ms: 2023-05-18 19:37:52.395279 +0200 CEST, 2023-05-18 17:37:52 +0000 UTC
2023/05/19 15:03:39 DEBUG : B9E738BC-0EDD-4595-98E6-7E96BD89C3A9.ictm: Modification times differ by -114.0182ms: 2023-05-18 19:37:52.1140182 +0200 CEST, 2023-05-18 17:37:52 +0000 UTC
2023/05/19 15:03:39 DEBUG : 8D430BA2-CE2C-485F-873E-5796BB13F0F6.ictm: Modification times differ by -851.6955ms: 2023-05-18 19:37:54.8516955 +0200 CEST, 2023-05-18 17:37:54 +0000 UTC
2023/05/19 15:03:39 DEBUG : Google drive root 'Backup/ELCA/ELCA-SQL/FULL_C2977EA4-BCF0-416B-8986-4092FD2540EB/Virtual Hard Disks': Waiting for checks to finish
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.mrt: Modification times differ by -276.2938ms: 2023-05-18 19:37:00.2762938 +0200 CEST, 2023-05-18 17:37:00 +0000 UTC
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx: Modification times differ by -851.6955ms: 2023-05-18 19:37:54.8516955 +0200 CEST, 2023-05-18 17:37:54 +0000 UTC
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx: Modification times differ by -129.6411ms: 2023-05-18 19:37:52.1296411 +0200 CEST, 2023-05-18 17:37:52 +0000 UTC
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.rct: Modification times differ by -379.6541ms: 2023-05-18 19:37:52.3796541 +0200 CEST, 2023-05-18 17:37:52 +0000 UTC
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.rct: Modification times differ by -276.2938ms: 2023-05-18 19:37:00.2762938 +0200 CEST, 2023-05-18 17:37:00 +0000 UTC
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.mrt: md5 = 49c67b498c2d8294c254232e9fb45c7f OK
2023/05/19 15:03:39 NOTICE: ELCA-SQL-DATA.vhdx.mrt: Skipped update modification time as --dry-run is set (size 76Ki)
2023/05/19 15:03:39 DEBUG : B9E738BC-0EDD-4595-98E6-7E96BD89C3A9.ictm: md5 = be98c416209715dd6b984ff873e5535b OK
2023/05/19 15:03:39 DEBUG : 8D430BA2-CE2C-485F-873E-5796BB13F0F6.ictm: md5 = c4f949c0bf678287d7ff579d4f8ee70e OK
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.mrt: md5 = 1b697175024e8d8bac6a77bcd076a849 OK
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.mrt: Unchanged skipping
2023/05/19 15:03:39 NOTICE: B9E738BC-0EDD-4595-98E6-7E96BD89C3A9.ictm: Skipped update modification time as --dry-run is set (size 176.060Ki)
2023/05/19 15:03:39 DEBUG : B9E738BC-0EDD-4595-98E6-7E96BD89C3A9.ictm: Unchanged skipping
2023/05/19 15:03:39 NOTICE: 8D430BA2-CE2C-485F-873E-5796BB13F0F6.ictm: Skipped update modification time as --dry-run is set (size 23.616Ki)
2023/05/19 15:03:39 NOTICE: ELCA-SQL.vhdx.mrt: Skipped update modification time as --dry-run is set (size 76Ki)
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.rct: md5 = 383eaf0dad79b53bdd5614afe3f5803f OK
2023/05/19 15:03:39 DEBUG : 8D430BA2-CE2C-485F-873E-5796BB13F0F6.ictm: Unchanged skipping
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.mrt: Unchanged skipping
2023/05/19 15:03:39 NOTICE: ELCA-SQL-DATA.vhdx.rct: Skipped update modification time as --dry-run is set (size 1.391Mi)
2023/05/19 15:03:39 DEBUG : ELCA-SQL-DATA.vhdx.rct: Unchanged skipping
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.rct: md5 = cd79a7bf44f4749056277480b23964e9 OK
2023/05/19 15:03:39 NOTICE: ELCA-SQL.vhdx.rct: Skipped update modification time as --dry-run is set (size 4.875Mi)
2023/05/19 15:03:39 DEBUG : ELCA-SQL.vhdx.rct: Unchanged skipping
2023/05/19 15:04:38 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 6 / 8, 75%
Elapsed time:       1m2.3s
Checking:
 *                            ELCA-SQL-DATA.vhdx: checking
 *                                 ELCA-SQL.vhdx: checking

2023/05/19 15:05:38 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 6 / 8, 75%
Elapsed time:       2m2.3s
Checking:
 *                            ELCA-SQL-DATA.vhdx: checking
 *                                 ELCA-SQL.vhdx: checking
5

This is rclone saying the modification times differ by 114 ms so likely the file needs uploading.

If the file is the same size then rclone will checksum the source and destination to check they are identical or not before uploading. This usually saves time as uploading is a lot slower than checksumming a local file.

There is no way to override this at the moment.

I could perhaps make this flag do it

  --ignore-checksum   Skip post copy check of checksums

Though that flag is quite specific so there probably needs to be a new one.

There was a suggestion that rclone should have a more configurable checking method like

  --track-renames-strategy string   Strategies to use when synchronizing using track-renames hash|modtime|leaf (default "hash")

By default this is something like size,modtime,hash but what you want is size,modtime with no hash.

6

Hi David,

those files are disk image of virtual disk, so the size is always fixed, and only the date/time change. I do not need to check cheksum on those files, because I already know the files need to be synced.

A flag like the following one should be very usefull.

--new-flag   Skip check of checksums, check only modtime and/or size

You'll be very kind if you could put this in your roadmap.
Thank you.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.