yes
rclone is aware of it, based on the values returned by hetzner.
the Total: value from rclone is total free space available.
so, the larger the total size of all snapshots, the smaller Total free will be
the number are a bit off, since hetzner is using GB, rclone is using TiB
5120 - 4607 = 513, which is approx size of snapshots
from hetzner website - used/snapshots/total
from rclone about
Total: 4.607 TiB
Used: 2.945 TiB
Free: 1.661 TiB
Ta. Good to know. I think I will start using Hetzner myself. Looks like it is perfect balance of features and price for my needs.
yeah, i recommend hetzner. the filesystem is zfs with automated and manual snapshots.
using port = 23, enables file verification when using sftp and a simple ssh command line.
and not well documented, but as seen from terminal, there is support for rclone + restic, which i know you use
| Available as server side backend: |
| borg |
| rsync |
| scp |
| sftp |
| rclone serve restic --stdio |
then combine that with a hetzner cloud vm in the same region.
i have a cheap linux vm, which is setup as a veeam backup and replication backup repository.
which used the smb feature of the storagebox.
This is cool. Sold to me.
This was the issue, incredible, thanks.
I still get some errors (always the same) :
Animes/Gintama/Season 7/Gintama - S07E27 - Style Goes Out of Fashion the Moment It's Put Into Words + There Are Two Types of People In This World - Those Who Yell Out Their Attack Names, and Those Who Don't.mp4: Failed to copy: Update Create failed: sftp: "Bad message" (SSH_FX_BAD_MESSAGE)
I think it's because of the text length, how can I correct that ?
Well... such issues could be prevented before you started. By optimizing your configuration. Now it is too late.
You would have to change crypt names encoding. But now it means starting from scratch and uploading everything again.
So just live with it. And do not use long names.
With Hetzner and in my country in Canada I get the 0% VAT so it's basically 2€/TO so it's incredible, fast uploads/downloads and unlimited transfers if you use their machines + sub-users so, never seen better.
I can start from scratch not a problem, tell what I need to change for that if you can please !
Or maybe I can see in logs (or verbose) the names so I can change them manually ?
Unless server side move is supported then names re-encoding can be done in place. But this is another story. I think with pretty much full remotes I would think twice before going this route.
if yes it will make all setup a bit simpler
rclone test info --check-length --check-base32768 HetznerCloud:test
and post results
yes this is the same
sudo rclone test info --check-length --check-base32768 HetznerCloud:test
Password:
2025/01/30 17:25:55 EME operates on 1 to 128 block-cipher blocks, you passed 513
2025/01/30 17:25:55 EME operates on 1 to 128 block-cipher blocks, you passed 513
panic: EME operates on 1 to 128 block-cipher blocks, you passed 513
goroutine 298 [running]:
log.Panicf({0x27fe881?, 0x470b25?}, {0xc000a93cf0?, 0x1010000006f?, 0x7f2f14a68e98?})
log/log.go:439 +0x65
github.com/rfjakob/eme.Transform({0x2bd0cc8, 0xc000514000}, {0xc000ac4ee0, 0x10, 0x10}, {0xc000ad8000, 0x2010, 0x2a80}, 0x1)
github.com/rfjakob/eme@v1.1.2/eme.go:128 +0x1e5
github.com/rclone/rclone/backend/crypt.(*Cipher).encryptSegment(0xc000ac4ea0, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/cipher.go:283 +0xd0
github.com/rclone/rclone/backend/crypt.(*Cipher).encryptFileName(0xc000ac4ea0, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/cipher.go:513 +0x27f
github.com/rclone/rclone/backend/crypt.(*Cipher).EncryptFileName(...)
github.com/rclone/rclone/backend/crypt/cipher.go:532
github.com/rclone/rclone/backend/crypt.(*Fs).NewObject(0xc0005f8c30, {0x2bd6750, 0x40458a0}, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/crypt.go:454 +0x65
github.com/rclone/rclone/backend/union.(*Fs).NewObject.func1(0x5)
github.com/rclone/rclone/backend/union/union.go:784 +0x8b
github.com/rclone/rclone/backend/union.multithread.func1()
github.com/rclone/rclone/backend/union/union.go:1053 +0x4e
created by github.com/rclone/rclone/backend/union.multithread in goroutine 1
github.com/rclone/rclone/backend/union/union.go:1051 +0x47
panic: EME operates on 1 to 128 block-cipher blocks, you passed 513
goroutine 295 [running]:
log.Panicf({0x27fe881?, 0x470b25?}, {0xc0008f5cf0?, 0x1010000006f?, 0x7f2f146ea158?})
log/log.go:439 +0x65
github.com/rfjakob/eme.Transform({0x2bd0cc8, 0xc000b30400}, {0xc007e00790, 0x10, 0x10}, {0xc000b12000, 0x2010, 0x2a80}, 0x1)
github.com/rfjakob/eme@v1.1.2/eme.go:128 +0x1e5
github.com/rclone/rclone/backend/crypt.(*Cipher).encryptSegment(0xc007e00750, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/cipher.go:283 +0xd0
github.com/rclone/rclone/backend/crypt.(*Cipher).encryptFileName(0xc007e00750, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/cipher.go:513 +0x27f
github.com/rclone/rclone/backend/crypt.(*Cipher).EncryptFileName(...)
github.com/rclone/rclone/backend/crypt/cipher.go:532
github.com/rclone/rclone/backend/crypt.(*Fs).NewObject(0xc0005f8ea0, {0x2bd6750, 0x40458a0}, {0xc000b20000?, 0x0?})
github.com/rclone/rclone/backend/crypt/crypt.go:454 +0x65
github.com/rclone/rclone/backend/union.(*Fs).NewObject.func1(0x2)
github.com/rclone/rclone/backend/union/union.go:784 +0x8b
github.com/rclone/rclone/backend/union.multithread.func1()
github.com/rclone/rclone/backend/union/union.go:1053 +0x4e
created by github.com/rclone/rclone/backend/union.multithread in goroutine 1
github.com/rclone/rclone/backend/union/union.go:1051 +0x47
Brilliant.
So we change all setup and then we re-encode all names in place (no need to start fresh)
[Hetzner]
[Hetzner1]
[Hetzner2]
[Hetzner3]
[Hetzner4]
[Hetzner5]
[HetznerUnion]
type = union
upstreams = Hetzner5:PMS Hetzner4:PMS Hetzner3:PMS Hetzner2:PMS Hetzner1:PMS Hetzner:PMS
action_policy = epall
create_policy = mfs
search_policy = ff
[HetznerCloud]
type = crypt
remote = HetznerUnion:
password = XXX
Make this change and test if all works. It should the same as before but we have now only one crypt to worry about.
EDIT - made mistake in Union. See now corrected
WOW
Ok. There is some issue. But let's ignore it and instead run:
rclone test info --check-length --check-base32768 Hetzner5:test
After you can delete these test directories if they are left behind. These rclone tests are not perfect:)
sudo rclone test info --check-length --check-base32768 Hetzner5:test
2025/01/30 17:42:35 NOTICE: sftp://hetzner5@hetzner5.your-storagebox.de:23/test/rclone-test-info-secuses5/test-base32768: 0 differences found
2025/01/30 17:42:35 NOTICE: sftp://hetzner5@hetzner5.your-storagebox.de:23/test/rclone-test-info-secuses5/test-base32768: 1028 matching files
// Hetzner5
maxFileLength = 255 // for 1 byte unicode characters
maxFileLength = 127 // for 2 byte unicode characters
maxFileLength = 85 // for 3 byte unicode characters
maxFileLength = 63 // for 4 byte unicode characters
base32768isOK = true // make sure maxFileLength for 2 byte unicode chars is the same as for 1 byte characters
I changed the configuration to the new one (after edit) :
[Hetzner]
type = sftp
host = XXX
user = XXX
port = 23
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
[Hetzner1]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner1Encrypted]
type = crypt
remote = Hetzner1:PMS
password = XXX
[Hetzner2]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner2Encrypted]
type = crypt
remote = Hetzner2:PMS
password = XXX
[Hetzner3]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner3Encrypted]
type = crypt
remote = Hetzner3:PMS
password = XXX
[Hetzner4]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner4Encrypted]
type = crypt
remote = Hetzner4:PMS
password = XXX
[Hetzner5]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner5Encrypted]
type = crypt
remote = Hetzner5:PMS
password = XXX
[HetznerCloud]
type = crypt
remote = HetznerUnion:
password = XXX
[HetznerEncrypted]
type = crypt
remote = Hetzner:PMS
password = XXX
[HetznerUnion]
type = union
upstreams = Hetzner5:PMS Hetzner4:PMS Hetzner3:PMS Hetzner2:PMS Hetzner1:PMS Hetzner:PMS
create_policy = mfs
shoud be:
[HetznerUnion]
type = union
upstreams = Hetzner5:PMS Hetzner4:PMS Hetzner3:PMS Hetzner2:PMS Hetzner1:PMS Hetzner:PMS
action_policy = epall
create_policy = mfs
search_policy = ff
after test if you can see all data using HetznerCloud etc. It should be the same as before
action_policy = epall and search_policy = ff are the default so I think the rclone config redacted don't show them as it's correct on the config edit (I checked).
I can see all my files as before when running rclone ls HetznerCloud:
yeap. Still I always prefer to be explicit and future proof (if somebody changes it one day)
So now scary part. You can use Hetzner snapshots to create option to roll back everything if something goes wrong. Should not but who knows... I have never done it with union before.
Create PMSNew dir in every Hetzner remote
Add new union and crypt setup.
[HetznerUnionNew]
type = union
upstreams = Hetzner5:PMSNew Hetzner4:PMSNew Hetzner3:PMSNew Hetzner2:PMSNew Hetzner1:PMSNew Hetzner:PMSNew
action_policy = epall
create_policy = mfs
search_policy = ff
[HetznerCloudNew]
type = crypt
remote = HetznerUnionNew:
password = XXX
filename_encoding = base32768
and run:
rclone move --server-side-across-configs -vv HetznerCloud: HetznerCloudNew:
After it finishes HetznerCloudNew will allow names up to ≈ 240 characters vs ≈ 160 before.
Run it first with --dry-run to see if all looks right.
I'll do it without snapshot as I think I don't have enough space to store it.
I don't quite understand the --dry-run but as I see it just say that he can move it whithout doing it :
2025/01/30 18:40:49 NOTICE: Animes/Concrete Revolutio - Superhuman Phantasmagoria - {tvdb=299491}/Season 1/Concrete Revolutio S01E13.mkv: Skipped move as --dry-run is set (size 925.894Mi)
2025/01/30 18:40:49 NOTICE: Animes/Concrete Revolutio - Superhuman Phantasmagoria - {tvdb=299491}/Season 1/Concrete Revolutio S01E13.fr.srt: Skipped move as --dry-run is set (size 23.790Ki)
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 1", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 2", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 3", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 4", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 5", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Added delayed dir = "Séries/Community/Season 6", newDst=<nil>
2025/01/30 18:40:49 DEBUG : Animes/Code Lyoko/Season 3/Code Lyoko - S03E01 - Straight to Heart.mp4: Need to transfer - File not found at Destination
I think it's all good to go now without it.
rclone config redacted
[Hetzner]
type = sftp
host = XXX
user = XXX
port = 23
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
[Hetzner1]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner2]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner3]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner4]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[Hetzner5]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23
[HetznerCloud]
type = crypt
remote = HetznerUnion:
password = XXX
[HetznerCloudNew]
type = crypt
remote = HetznerUnionNew:
password = XXX
filename_encoding = base32768
[HetznerUnion]
type = union
upstreams = Hetzner5:PMS Hetzner4:PMS Hetzner3:PMS Hetzner2:PMS Hetzner1:PMS Hetzner:PMS
create_policy = mfs
[HetznerUnionNew]
type = union
upstreams = Hetzner5:PMSNew Hetzner4:PMSNew Hetzner3:PMSNew Hetzner2:PMSNew Hetzner1:PMSNew Hetzner:PMSNew
create_policy = mfs