Google Drive sync checking gets huge amount of time

[jarek]

What is the problem you are having with rclone?

Not understand how Wasabi sync differ from my google drive syncand need help or expanation on that.
When syncing my borg backup to wasabi it s matter of seconds (few files about 300MB).
Full archive is about 135GB.
When syncing to my mounted-rclone-union-google-drive every file is 'checked' and this take huge amount of time (and transfer), 'checking' of already synced files take at least 30 minutes (part of log below).

Assumed, that only files that differ in size and time will be touched (uploaded/deleted) and that will be done as fast as with wasabi. But it only is when use ---size-only flag with wasabi.

So would it mean, wasabi is super fast compared to gdrive, or it use kind of 'size only' as default, or I clearly missed something here (and missed some configuration options) ?

What is your rclone version (output from rclone version)

rclone v1.55.1-DEV

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux centos 8

Which cloud storage system are you using? (eg Google Drive)

Google Drive and Wasabi

The command you were trying to run (eg rclone copy /tmp remote:tmp)

with wasabi:

rclone sync -v /media/usb/backup/bck.borg wasabi:bck

with google drive:
mount:

rclone mount -v gdrive: /mnt/gdrive \
--config=rclone.conf \
--allow-other \
--dir-cache-time 1000h \
--attr-timeout 1000h \
--poll-interval 1m \
--umask 002 \
--user-agent aasd1\
--drive-use-trash=false \
--rc \
--rc-addr129.168.0.202:4321 \
--rc-web-gui \
--rc-enable-metrics \
--rc-user=**** \
--rc-pass=**** \
--rc-web-gui-no-open-browser \
--rc-web-gui-force-update \
--union-create-policy=mfs \
--union-action-policy=all \
--vfs-read-chunk-size 32M \
--drive-acknowledge-abuse=true \
--fast-list

command:

rclone -v sync /media/usb/backup/bck.borg /mnt/gmultidrive --transfers 2 --checkers 2  --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10

The rclone config contents with secrets removed.

[wasabi]
type = s3
provider = Wasabi
env_auth = false
access_key_id = ****
secret_access_key = ****
endpoint = s3.eu-central-1.wasabisys.com
acl = private

[c1]
root_folder_id = ****
type = drive
scope = drive
service_account_file = /a/b/d.json
impersonate = c1@me.you

[c2]
root_folder_id = ****
type = drive
scope = drive
service_account_file =  /a/b/d.json
impersonate = c2@me.you

[c3]
root_folder_id = ****
type = drive
scope = drive
service_account_file =  /a/b/d.json
impersonate = c3@me.you

[c4]
root_folder_id = ****
type = drive
scope = drive
service_account_file =  /a/b/d.json
impersonate = c4@me.you

[gmultidrive]
type = union
upstreams = c1: c2: c3: c4:

A log from the command with the -vv flag

2021/08/22 23:35:14 INFO  : data/1/1874: Updated modification time in destination
2021/08/22 23:35:14 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:               511 / 1210, 42%
Elapsed time:      11m1.9s
Checking:
 *                                   data/1/1853: checking
 *                                   data/1/1876: checking

2021/08/22 23:35:14 INFO  : data/1/1853: Updated modification time in destination
2021/08/22 23:35:14 INFO  : data/1/1876: Updated modification time in destination
2021/08/22 23:35:15 INFO  : data/1/1878: Updated modification time in destination
2021/08/22 23:35:16 INFO  : data/1/1880: Updated modification time in destination
2021/08/22 23:35:16 INFO  : data/1/1882: Updated modification time in destination
2021/08/22 23:35:16 INFO  : data/1/1884: Updated modification time in destination
2021/08/22 23:35:17 INFO  : data/1/1886: Updated modification time in destination
2021/08/22 23:35:17 INFO  : data/1/1888: Updated modification time in destination
2021/08/22 23:35:18 INFO  : data/1/1890: Updated modification time in destination
2021/08/22 23:35:19 INFO  : data/1/1892: Updated modification time in destination
2021/08/22 23:35:19 INFO  : data/1/1894: Updated modification time in destination
2021/08/22 23:35:20 INFO  : data/1/1896: Updated modification time in destination
2021/08/22 23:35:20 INFO  : data/1/1898: Updated modification time in destination
2021/08/22 23:35:21 INFO  : data/1/1900: Updated modification time in destination
2021/08/22 23:35:21 INFO  : data/1/1902: Updated modification time in destination
2021/08/22 23:35:22 INFO  : data/1/1877: Updated modification time in destination

[asdffdsa (jojothehumanmonkey)]

hello and welcome to the forum,

not sure exactly what you are asking, given your setup.

• sync with wasabi
• sync with gdrive
• rclone mount
• rclone union

are you comparing rclone sync on wasabi with a rclone mount using a union of gdrive?
if true, that might not be a good comparison

rclone sync does testing by size and modification time
that applies to both gdrive and wasabi.

gdrive has lots of limitations, that wasabi does not have.

• too many api calls per second, rclone gets throttled.
• "lots of small files can take a long time."

@jarek

'checking' of already synced files take at least 30 minutes

by way of example, in wasabi, i have a bucket with over 1,000,000 files.
let's say that the local source and that bucket have already been synced, so no new files to be transferred.
the rclone sync command, which needs to check every file, takes only 30 seconds, not 30 minutes.

that log is just a snippet without the top part of the log, hard to know what is going on...

you may be able to tweak the gdrive flags and perhaps make changes at google api website.

about the rclone mount command

• --fast-list does nothing on a mount....

• --rc-addr129.168.0.202:4321 does not look valid, perhaps it should be
  --rc-addr=129.168.0.202:4321

[jarek]

Thought I was clear in my question

My files:
It is borg archive folder with 1210 files there are only two kind of files in it - small (1KB), and big 250MB -500 MB max, whole folder have 139GB now.

Wasabi:
I'm using wasabi from year now and have no problem at all with it, my daily syncs - like I said - get around 30 seconds, there are about 250MB - 500MB of changed files.

Gdrive:
Now I'm trying to create another cloud store with 4 gdrive accounts unioned with rclone union (config above) and mounted in my linux box with command listed above.
First sync taken around 10 hours because my upload is limited to ~40Mbit/s so I understand this.
But when I'm syncing again with already synced remote it takes over 30 minutes to complete, no file transfer, only checks.

Now i have no idea why checks in gdrive soluton take so long to complete.

Thanks fora a tip on --fast-list , IP typo is due my obfuscation of data - it is with equality sign.

[UPDATE]

You gave me idea to look to verbose output (should do it in first place):

2021/08/23 09:38:30 DEBUG : data/1/1009: Modification times differ by -362.334µs: 2021-04-28 01:10:26.242362334 +0200 CEST, 2021-04-28 01:10:26.242 +0200 CEST

As I understand this forces to checksum check and that takes huge amount of time.
When I set --modify-window 1s it started work as expected. Im syncing data daily so this looks safe.