Files corrupted with Hetzner storage box

Help and Support
1

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I'm trying to crontab a backup from my linux server with rclone to a Hetzner storage box.
As I would like to back up the full /opt directory, I used that command:

rclone sync --progress --skip-links --fast-list /opt/ backup_1:opt

But I am getting spammed with the following errors:

ERROR : gitlab/rh/data/postgresql/data/pg_stat_tmp/db_0.stat: Failed to calculate dst hash: failed to calculate md5 hash: failed to run "md5sum -r /home/opt/gitlab/rh/data/postgresql/data/pg_stat_tmp/db_0.stat": md5: /home/opt/gitlab/rh/data/postgresql/data/pg_stat_tmp/db_0.stat: No such file or directory: Process exited with status 1

This not only occurs with .stat files, also with .tar.gz and some others.
Funny enough, if I use --ignore-checksum, I am still getting that error.

Also, I noted that many files were erroring with this error:

corrupted on transfer: MD5 hash differ "53a0e5a5db26301f97d390a51155cae7" vs ""

So I thought that has to do with the data transfer itself.

So I googled and found that Hetzner only supports 10 connections at a time.
So, I added --transfers=3 and --checksrs=6 to the command.
But still, the same issue.

Any ideas what I am doing wrong?

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: almalinux 8.8 (64 bit)
  • os/kernel: 4.18.0-348.12.2.el8_5.x86_64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Hetzner storage box

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync --progress --skip-links --fast-list /opt/ backup_1:opt

The rclone config contents with secrets removed.

[backup_1]
type = sftp
host = redacted
user = redacted
port = 23
pass = redacted
md5sum_command = md5sum -r
sha1sum_command = sha1 -r
shell_type = unix

A log from the command with the -vv flag

Would be ways to long - rclone is trying to clone 10GB.

2

That looks like when rclone tried to make the MD5SUM on the remote that the connection was banned probably because of connection limits.

That is a direct consequence of the first error.

Try --checkers 4 --transfers 4 --check-first which will mean rclone isn't checking and transferring at the same time.

If that doesn't help try lowering the numbers some more.

If that doesn't help then please post a full log somewhere.

3

Alright, thank you for your answer!
I just tried with that command to transfer only one specific folder and not the whole buch of data:

rclone sync --skip-links --fast-list /opt/gitlab/rh/ backup_1:opt/gitlab/rh --ignore-checksum --checkers 3 --transfers 3 --check-first

Here's what has been logged (default log, no debug):

Log 
2023/06/14 12:51:06 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 12:51:06 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 12:51:06 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 12:51:06 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 12:51:06 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 12:51:06 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 12:51:06 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 12:51:08 ERROR : data/prometheus/data/wal/00001768: corrupted on transfer: sizes differ 80004366 vs 80004612
2023/06/14 12:51:08 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 12:51:08 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 12:51:08 ERROR : Attempt 1/3 failed with 2 errors and: corrupted on transfer: sizes differ 80004366 vs 80004612
2023/06/14 12:51:08 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 12:51:08 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 12:51:08 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 12:51:08 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 12:51:08 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 12:51:08 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 12:51:08 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 12:51:10 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 12:51:10 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 12:51:10 ERROR : Attempt 2/3 failed with 1 errors and: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 12:51:10 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 12:51:10 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 12:51:10 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 12:51:10 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 12:51:10 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 12:51:10 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 12:51:10 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 12:51:11 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 12:51:11 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 12:51:11 ERROR : Attempt 3/3 failed with 1 errors and: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 12:51:11 Failed to sync: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
4

repeat with DEBUG log

5

Ok, here you go:

6

no single ERROR....

7

This is the sftp transfer failing for some reason. Unfortunately the error messages on sftp are very limited!

Are you out of quota maybe?

8

Nope - ~800GB left.
Is rclone maybe caching the data to send locally on the source server? But shouldn't matter anyways as I have more than 30 GB left on the source system too.

9

No it shouldn't be.

I suspect this is rate limiting on the ssh connections, try reducing to --checkers 1 --transfers 1 --check-first to see if you can get it to run clean. If it does you can try increasing those numbers.

10

Already done, but still those errors... (without debug again):

2023/06/14 16:18:00 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 16:18:00 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 16:18:00 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 16:18:00 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 16:18:00 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 16:18:00 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 16:18:01 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 16:18:02 ERROR : logs/gitlab-workhorse/current: corrupted on transfer: sizes differ 2558650 vs 2559120
2023/06/14 16:18:02 ERROR : logs/nginx/gitlab_access.log: corrupted on transfer: sizes differ 2756830 vs 2756982
2023/06/14 16:18:04 ERROR : data/prometheus/data/wal/00001770: corrupted on transfer: sizes differ 56248572 vs 56256674
2023/06/14 16:18:05 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 16:18:05 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 16:18:05 ERROR : Attempt 1/3 failed with 4 errors and: corrupted on transfer: sizes differ 56248572 vs 56256674
2023/06/14 16:18:06 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 16:18:06 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 16:18:06 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 16:18:06 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 16:18:06 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 16:18:06 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 16:18:07 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 16:18:08 ERROR : data/prometheus/data/wal/00001770: corrupted on transfer: sizes differ 56268378 vs 56268904
2023/06/14 16:18:08 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 16:18:08 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 16:18:08 ERROR : Attempt 2/3 failed with 2 errors and: corrupted on transfer: sizes differ 56268378 vs 56268904
2023/06/14 16:18:08 NOTICE: data/gitaly/gitaly.socket: Can't transfer non file/directory
2023/06/14 16:18:08 NOTICE: data/gitlab-rails/sockets/gitlab.socket: Can't transfer non file/directory
2023/06/14 16:18:08 NOTICE: data/gitlab-workhorse/sockets/socket: Can't transfer non file/directory
2023/06/14 16:18:08 NOTICE: data/gitaly/run/gitaly-300/sock.d/intern: Can't transfer non file/directory
2023/06/14 16:18:08 NOTICE: data/postgresql/.s.PGSQL.5432: Can't transfer non file/directory
2023/06/14 16:18:08 NOTICE: data/redis/redis.socket: Can't transfer non file/directory
2023/06/14 16:18:09 ERROR : logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023/06/14 16:18:11 ERROR : data/prometheus/data/wal/00001770: corrupted on transfer: sizes differ 56325508 vs 56354649
2023/06/14 16:18:11 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting files as there were IO errors
2023/06/14 16:18:11 ERROR : sftp://my-host-redacted:23/opt/gitlab/rh: not deleting directories as there were IO errors
2023/06/14 16:18:11 ERROR : Attempt 3/3 failed with 2 errors and: corrupted on transfer: sizes differ 56325508 vs 56354649
2023/06/14 16:18:11 Failed to sync with 2 errors: last error was: corrupted on transfer: sizes differ 56325508 vs 56354649```
11

Shot in the dark:

Are you 100% certain those files aren't changing while you are backing up? I am seeing log files, socket (which won't transfer), write-ahead-logs, etc.

Make a local copy so that these aren't touched and try again. And/or make some local test files.

Personally, I moved away from storage boxes since SFTP is noticeably slower than HTTP based protocols, trying to stay in the connection limit was hard, and downloads were painfully slow. I would believe it was user error on my part and would test again if I ever need but for now, I am using B2. (I could use webDAV which is faster but I lose ModTime support. Not the end of the world but size-based change detection is fraught with false negatives)

12

Hi there,
thank you for your suggestion.
I already tried copying the files away and then rclone from there. Same issues again.

With
rclone sync --skip-links --fast-list /tmp/rclone_backup_cache/gitlab/ backup_1:opt/gitlab/ --ignore-checksum --checkers 1 --transfers 1 --check-first --progress
Most errors seems to be those:

rh/logs/gitlab-rails/auth.log.16.gz: Failed to copy: Update Create failed: sftp: "Failure" (SSH_FX_FAILURE)
2023-06-19 13:56:24 ERROR : sftp://my-host-redacted:23/opt/gitlab/: not deleting files as there were IO errors
2023-06-19 13:56:24 ERROR : sftp://my-host-redacted:23/opt/gitlab/: not deleting directories as there were IO errors

I ensured that no more devices are having a SFTP connection at the same time.
Seems like it would happen only to .gz files...

13

Does it work on retry? I don't see errors with size changes so that is good.

As others have said, SFTP doesn't give great error messages but I have found that I tend to get that error when the (usually encrypted) path name is too long.

14

Alright, thank you for your idea.
I just tried manually rcloning such a .gz file (the errors only come with the .gz files!). It works like a charm without any errors.
So I would assume the path length cannot be the problem, can it?

15

I would agree. Maybe try the full sync again? Or increase the number of retries?

Like I said, I moved off of storage boxes because it was too finicky with this kind of stuff.

16

I already tried multiple times - the same issue again.
Can you recommend some providers where I could switch to alternative technologies? But: They has to be in EU, as the GDPR in germany (my location) is requiring that.

Also, maybe I could switch away from rclone and just go with a default rsync... what's your opinon with that?

17

The hetzner storage box also supports webdav and smb

A webdav config looks like this.

[hbox-webdav]
type = webdav
url = https://uXXX.your-storagebox.de
user = uXXX
pass = XXX

Webdav doesn't support modification times or hashes so it isn't a full features as sftp but it is very likely to be faster and might work better.

18

Alright, I will try, thank you!

19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.