Bisync takes hours to sync big folders

Help and Support
1

What is the problem you are having with rclone?

I try to sync a folder in Google Drive (roughly 2TB, 600 subfolders and each has 1-10 files) with a local folder (on a ssd, so should not be the bottleneck). With --resync it works, but takes a lot of time (last time 6h), even when all the files are already there. After I run this command last evening I did not touch any file, but it takes even longer and then failes when I tried to run the command again without the --resync flag.

I could not find the log I got from running the command the first time without --resync. But for now it gives me Bisync critical error: cannot find prior Path1 or Path2 listings, likely due to critical error on prior run. I will make another resync and another normal run and reply to this thread.

So for now my question is: Is it normal to take 3+3h to index/hash 2TB data, because this is what I assume is happening? CPU is only 1 core ~70% occupied and I haven't catched any network traffic.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.19.0-45-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linkig: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive and a local ssd

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone bisync --resync --verbose --progress  gd:Filme /mnt/ssd/Filme

The rclone config contents with secrets removed.

[gd]
type = drive
client_id = XXXX
client_secret = XXXX
scope = drive
token = {"access_token":"XXXX..."
team_drive =

A log from the command with the -vv flag

This was yesterday:

2023/07/02 20:25:58 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023-07-02 20:25:58 INFO  : Synching Path1 "gd:Filme/" with Path2 "/mnt/ssd/Filme/"
2023-07-02 20:25:58 INFO  : Copying unique Path2 files to Path1
2023-07-02 23:30:39 INFO  : Resynching Path1 to Path2
2023-07-02 23:30:45 NOTICE: XXXX: Duplicate directory found in source - ignoring
2023-07-02 23:32:21 INFO  : There was nothing to transfer
2023-07-02 23:32:21 INFO  : Resync updating listings
2023-07-03 02:28:24 INFO  : Bisync successful
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              4881 / 4881, 100%
Elapsed time:    6h2m27.0s
2023/07/03 02:28:24 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:              4881 / 4881, 100%
Elapsed time:    6h2m27.0s

And this is with -vv flag:

rclone bisync --resync -vv --progress  gd:Filme /mnt/ssd/Filme
2023/07/03 16:20:51 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "bisync" "--resync" "-vv" "--progress" "gd:Filme" "/mnt/ssd/Filme"]
2023/07/03 16:20:51 DEBUG : Creating backend with remote "gd:Filme"
2023/07/03 16:20:51 DEBUG : Using config file from "/home/USER/.config/rclone/rclone.conf"
2023/07/03 16:20:51 DEBUG : Google drive root 'Filme': 'root_folder_id = XXXX - save this in the config to speed up startup
2023/07/03 16:20:52 DEBUG : Creating backend with remote "/mnt/ssd/Filme"
2023/07/03 16:20:52 NOTICE: bisync is EXPERIMENTAL. Don't use in production!
2023-07-03 16:20:52 DEBUG : Lock file created: /home/USER/.cache/rclone/bisync/gd_Filme..mnt_ssd_Filme.lck
2023-07-03 16:20:52 INFO  : Synching Path1 "gd:Filme/" with Path2 "/mnt/ssd/Filme/"
2023-07-03 16:20:52 INFO  : Copying unique Path2 files to Path1
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
2

You'd want to dedupe the source as there's no telling what would get sync'ed.

3

ARM - 1 core CPU system... There could be many bottlenecks on such hardware.

To get some estimate how fast things are

create 10GB file on your SSD:

dd if=/dev/urandom of=10GB.bin bs=64M count=160

and as google drive is using md5 hash run:

time md5sum 10GB.bin

It will give you good feeling how long it takes to hash 2TB of data

4

No I have 4 cores, but only one is active during resync.

the md5 also just used one core

|real|0m33,757s|
|user|0m24,315s|
|sys|0m8,581s|
5

Check this thread:

It suggests to use --ignore-checksum to speed things up

1 Like
6

FYI. There is substantial amount of changed and new code in the process with milestone set for the next rclone release (v1.64):

7

Seems the Duplicate directory found in source was the problem for bisync to fail when not in resync mode.

Also thanks for the --ignore-checksum, this reduced the sync time to a few minutes.

8

I think the key is that you are trying something what is not ready yet - so problems are expected. It is experimental feature with still a lot of ongoing work.

Beta testers results are always welcomed:) Just do not expect that it will be trouble free and it will work without data lose. I would not use it for "production".

9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.