Rclone Swift sync very slow

Help and Support
1

What is the problem you are having with rclone?

Rclone sync of Swift container consisting of docker files taking long time. The below command is set to sync data for last 7 hrs on Monday evening and its been running since then. The sync is happening between US Data Center and data center in Europe.
The initial sync of all the data took around 2 weeks. I believe its only the delta (daily code updates will not be more that few GBs). Appreciate if someone can help with the right flags to speed up the sync.

What is your rclone version (output from rclone version)

rclone v1.53.1

  • os/arch: linux/amd64
  • go version: go1.15

Which OS you are using and how many bits (eg Windows 7, 64 bit)

CentOS 7.8.2003

Which cloud storage system are you using? (eg Google Drive)

Swift Stack

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync Swift1:registry Swift2:dockerregistry --fast-list --checksum --ignore-existing --max-age 7h --log-file /var/log/rclone/rclonesync.log --log-level DEBUG

The rclone config contents with secrets removed.

[Swift1]
type = swift
user = repo
key = ********
auth = http://url/auth/v1.0
domain =
tenant =
tenant_domain =
region =
storage_url =
auth_version = 1

[Swift2]
type = swift
user = svc.prod
key = ******
auth = http://url/auth/v1.0
domain =
tenant =
tenant_domain =
region =
storage_url =
auth_version = 1

A log from the command with the -vv flag

2020/10/08 21:02:09 DEBUG : images/files/docker/v2/repositories/copart/ycsdmvsmus/_layers/sha256/eff7a10a26da26c34360c78b6dff31cb179c2cdad454tre4yg/link: Excluded from sync (and deletion)
2020/10/08 21:02:09 DEBUG : images/files/docker/v2/repositories/copart/ycsdmvsmus/_layers/sha256/f0fd78c1cca8d49323a5910a6b7a7a5ac541af721ee4188f23345544fe/link: Excluded from sync (and deletion)
2

I see this in the logs:
its still running :frowning: but nothing copied I guess.
2020/10/09 04:41:12 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 44h22m3.6s

3

How many objects/files are there?

You can increase --checkers and --transfers and if you are mostly checking files, the first one would help.

4

You added --fast-list which means rclone will do all the listing to memory before attempting to transfer any files. If there are millions of file this listing can take some time!

You may find that not using --fast-list but increasing --checkers is faster. This will use more transactions but get the job done quicker.

5

Thank you ncw. I will remove --fast-list next time and check.
The job hasn't completed and its been more than 5 days. I am wondering why the --max-age 1d is not working - its just going on and keeps rolling into the next day.
Ideally what I want is the job to check if any uploads happened the previous day and sync to destination.

Ofcourse - the transfer is going between continents.. over the internet. Maybe it will take time.

2020/10/15 05:31:35 INFO :
Transferred: 46.818G / 46.818 GBytes, 100%, 39.413 MBytes/s, ETA 0s
Transferred: 6247 / 6247, 100%
Elapsed time: 141h46m0.6s

Also the log file is around 2 GB. Is there a way for rclone to auto rotate it? I tried the linux log rotate, it worked but for some reason I found rclone now is writing the same output to two files (the rotated one and new one).

Thanks in advance.

6

Thanks Animisity022 - I have added --checkers and --transfers but its been running for a while now. Will monitor for couple more. days and remove --fast-list as mentioned by ncw,

7

You need to add the copytruncate option to logrotate to get it to work with rclone.

8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.