Sync Wasabi -> Hetzner Very slow

Help and Support
1

What is the problem you are having with rclone?

I run sync Wasabi to Hetzner
and it goes very slowly when I use --checksum
Here I can't use more than 4 checkers at a time.
I don't have a lot of transfer data (100gb)daily, but over 1 million files.
I use hetzner which has a limit of 10 Concurrent connections

What will be smart for me?? --check-first??

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.4.0-144-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Wasabi -> Hetzner

The command you were trying to run (eg rclone copy /tmp remote:tmp)

RC_FLAGS="--checksum --progress --create-empty-src-dirs --checkers 4  --transfers 4"

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
2

Very slow

sorry, no idea what that means?

can you answer the other questions?

3

Forgot my question
I have updated
Sorry

4

again, still no idea what Very slow means?
in my case, from wasabi->hetzner, for a single file, i average approx 50MiBs.

--checksum forces rclone to hash every dest file, that takes time to calculate.
unless you have an unusual use-case, not sure why you added that flag?

try adding --fast-list

can you post the exact command, not just the flags.

???

5

[INFO] Source: W:bucket1 -> Destination: H2:bucket1
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Checks: 16009 / 16009, 100%
Elapsed time: 17m57.9s
This is slow when i got 1 mill files

rclone sync W:bucket1 H1:bucket1 --checksum --progress --create-empty-src-dirs --checkers 4 --transfers 4

I added --checksum to make sure the files I sync are 100% OK
Is it not necessary?

you suggest "--fast-list" what will it do here?

Thanks for helping

START OF CONFIG

6

tl;dir, up above, we have been requested basic information, all forum members are asked to post.
so please, post all the requested information. do not want to have to keep asking for it, ok?

after posting that,
in addition, now, run a command that copies a single file, add -vv and post the entire output.

can you read the docs, and if that is not clear, let me know.
run the command with that flag and see if it makes a difference.

7

"H1": {
"host": "XXXXX.your-storagebox.de",
"md5sum_command": "md5 -r",
"pass": "XXXXXXXXXX",
"port": "23",
"sha1sum_command": "sha1 -r",
"shell_type": "unix",
"type": "sftp",
"user": "xxxxxx"
},
"W": {
"access_key_id": "XXXXXXX",
"endpoint": "s3.eu-central-1.wasabisys.com",
"env_auth": "false",
"provider": "Wasabi",
"secret_access_key": "XXXXXXX",
"type": "s3"
},

8

and where are you running rclone, local or hetzner cloud vm?

sftp is a very old/slow protocol, does not work well with many small files.

now, run a command and post the full output.
--- that copies a single file
--- add -vv --fast-list
--- remove --checksum

9

This i with --checksum

2023/03/14 19:15:39 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "W:serverbackup/3g.jpg" "H2:serverbackup/" "--checksum" "-vv" "--progress" "--create-empty-src-dirs" "--checkers" "4" "--transfers" "4"]

2023/03/14 19:15:39 DEBUG : Creating backend with remote "W:serverbackup/3g.jpg"

2023/03/14 19:15:39 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"

2023/03/14 19:15:39 DEBUG : fs cache: adding new entry for parent of "W:serverbackup/3g.jpg", "W:serverbackup"

2023/03/14 19:15:39 DEBUG : Creating backend with remote "H2:serverbackup/"

2023/03/14 19:15:39 DEBUG : sftp://XXXXXX@XXXXXX.your-storagebox.de:23/serverbackup/: New connection xxx.xxx.xxx.xxx:44948->xxx.xxx.xxx.xxx:xxx to "SSH-2.0-OpenSSH_8.8 FreeBSD-20211221"

2023/03/14 19:15:39 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/serverbackup/: Shell type "unix" from config

2023/03/14 19:15:39 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/serverbackup/: Relative path resolved to "/home/serverbackup"

2023/03/14 19:15:39 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/serverbackup/: Using root directory "/home/serverbackup"

2023-03-14 19:15:39 DEBUG : 3g.jpg: Need to transfer - File not found at Destination

2023-03-14 19:15:40 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/Serverbackup/: Shell path "/home/serverbackup/3g.jpg"

2023-03-14 19:15:40 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/serverbackup/: Running remote command: md5 -r /home/serverbackup/3g.jpg

2023-03-14 19:15:40 DEBUG : sftp://xxxxx@xxxxx.your-storagebox.de:23/serverbackup/: Remote command result: 23bc4b82be36caf0eead1b2986xxxxxx /home/serverbackup/3g.jpg

2023-03-14 19:15:40 DEBUG : 3g.jpg: Parsed hash: 23bc4b82be36caf0eead1b2986xxxxxx

2023-03-14 19:15:40 DEBUG : 3g.jpg: md5 = 23bc4b82be36caf0eead1b2986xxxxxx OK

2023-03-14 19:15:40 INFO : 3g.jpg: Copied (new)

Transferred: 763.928 KiB / 763.928 KiB, 100%, 0 B/s, ETA -

Transferred: 1 / 1, 100%
Elapsed time: 0.7s
2023/03/14 19:15:40 INFO :
Transferred: 763.928 KiB / 763.928 KiB, 100%, 0 B/s, ETA -
Transferred: 1 / 1, 100%
Elapsed time: 0.7s

10

post the complete output, not just a snippet

11

i would establish a baseline.

  1. run the sync using the exact command you first posted
    make sure all the files were transfered.

  2. run the exact same command again, this time no files should be transferred.
    make sure to add -vv
    post the full output.

and you did not answer this question???, please answer all questions the first time, so i do not have to keep asking. thanks

12

i run it from Hetzner

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.