Sync with Union Remotes exceed the quota

PortX_Charge · January 15, 2025, 3:29am

What is the problem you are having with rclone?

I'm using rClone to sync my NAS to a Storage Server (20TB storage server maximum) so I have 5x20TB, each storage server is accessible by ssh, webdav or samba so I created 5 configurations for each and used rClone Union to upload all my files to all of them BUT it seem's that rClone or Union protocol can't get the quota of my storage server as I get the same errorserrors :
"Failed to copy: 1 error: HetznerCloud: mkdir "/xxx/xxx.xxx" failed: sftp: "Quota exceeded" (SSH_FX_FAILURE)"

Run the command 'rclone version' and share the full output of the command.

rclone v1.67.0

os/version: qts 5.2.0 (64 bit)
os/kernel: 5.10.60-qnap (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.22.4
go/linking: static
go/tags: none

I know that it's not the latest version but I don't think it's related to version of rClone, I just don't know how to set manually the limit as all my storage servers have the same size.

Which cloud storage system are you using? (eg Google Drive)

Hetzner Storage Server

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

sudo ./rclone sync /NAS/ HetznerCloud: --progress --exclude="/Downloads/**" --exclude="/.*/" --ignore-errors

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[Hetzner]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none

[Hetzner1]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none

[Hetzner1Encrypted]
type = crypt
remote = Hetzner1:/PMS
password = XXX

[Hetzner2]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none

[Hetzner2Encrypted]
type = crypt
remote = Hetzner2:/PMS
password = XXX

[Hetzner3]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = none
sha1sum_command = none

[Hetzner3Encrypted]
type = crypt
remote = Hetzner3:/PMS
password = XXX

[HetznerCloud]
type = union
upstreams = Hetzner3Encrypted: Hetzner2Encrypted: Hetzner1Encrypted: HetznerEncrypted:

[HetznerEncrypted]
type = crypt
remote = Hetzner:/PMS
password = XXX

A log from the command that you were trying to run with the `-vv` flag

kapitainsky · January 15, 2025, 8:58am

Can you run:

rclone about Hetzner3Encrypted:

PortX_Charge · January 15, 2025, 2:48pm

sudo ./rclone about Hetzner3Encrypted:
Password:
Total: 20 TiB
Used: 5.925 TiB
Free: 14.075 TiB

It seems that he knows finally so why is he sending theses errors

kapitainsky · January 15, 2025, 3:22pm

This is good sign:) We will get to the bottom of it and fix it.

Now could you please share all log file with your command with -vv option?

Try to copy some small data subset only so log is not massive.

Also bump your rclone version - to the latest one. There were multiple SFTP fixes in the meantime.

PortX_Charge · January 20, 2025, 4:48am

Sorry for my late reply.

I tried to do the exact same command using Windows PowerShell connecting throught SSH to my NAS and ran the command without any issues. But when running it from my Raspberry Pi, I get theses errors, very strange and so it's not a rClone issue anymore I think !

asdffdsa · January 20, 2025, 2:03pm

with hetzner sftp, might want to use port = 23
that enables file transfer verification using hash.
and the remote would look like

[sbox_sftp_02]
type = sftp
host = redacted.your-storagebox.de
port = 23
key_file = c:\data\c\combined\hetzner\keys\id_ed25519
user = redacted
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r

PortX_Charge · January 22, 2025, 4:20am

Hi, thanks for informations but why should I enable file transfer using hash ?
Btw can you explain the "md5 -r" & "sha1 -r" ?

kapitainsky · January 22, 2025, 5:44am

Hash is not used for transfer but for validation if two files are the same - for example source and destination to check if nothing was corrupted during transfers.

asdffdsa · January 22, 2025, 2:48pm

well, after rclone completes the transfer a file, rclone can compute and compare the hash of the dest file and compare it to the hash of the source file.

those are the command, on the dest server, that computes the hash.
you can find those values from the debug log.

as i mentioned, up above, need to use port = 23
https://docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/#rclone

PortX_Charge · January 26, 2025, 5:10pm

It looks like it wasn't a problem with Raspberry finally as I get the errors again, less than on the Raspberry but still :

PortX_Charge · January 28, 2025, 5:51am

I'm now using the @asdffdsa configuration with port 23 and it seems to be better as I get less errors but I still get some and the files with theses errors are not uploaded at all on the remote (except some with "Bad message") :

And last thing after running the command I get "Failed to sync with 933 errors: last error was: failed to set directory modtime: 173 errors: last error: 3 errors: Hetzner2Encrypted: SetModTime failed: sftp: "Failure" (SSH_FX_FAILURE); Hetzner1Encrypted: SetModTime failed: sftp: "Failure" (SSH_FX_FAILURE); HetznerEncrypted: SetModTime failed: sftp: "Failure" (SSH_FX_FAILURE)"

PortX_Charge · January 28, 2025, 7:10am

I'm talking alone I know but I really need help as I'm stuck and I want to help other people in my case so I write all my findings.
After searching more informations on the forum I've seen that maybe it can be a storage space issue so just to remind the configuration, now I have 6 Hetzner Storage Box (Hetzner:, Hetzner1:, ..., Hetzner5:).
I configure them on rClone using SFTP on port 23 then I use encrypt to encrypt all files on Hetzner (HetznerEncrypt:, Hetzner1Encrypt:, ..., Hetzner5Encrypt:).
Because of all theses Storage Boxes, I use rClone Union to get something that looks like 1 Storage (HetznerCloud) so I have this :

Maybe it's trying to upload on Hetzner1 but it's full and not switching to the one that it's not ?
If you can help me again @kapitainsky please !

kapitainsky · January 28, 2025, 7:51am

You do not help by posting random screenshots which are difficult to read. Also you mentioned that now you changed configuration.

Post rclone config redacted and full debug log as a text. To limit its size try to transfer some small subdirectory only.

PortX_Charge · January 29, 2025, 5:31am

Redacted config :

[Hetzner]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner1]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner1Encrypted]
type = crypt
remote = Hetzner1:PMS
password = XXX

[Hetzner2]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner2Encrypted]
type = crypt
remote = Hetzner2:PMS
password = XXX

[Hetzner3]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner3Encrypted]
type = crypt
remote = Hetzner3:PMS
password = XXX

[Hetzner4]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner4Encrypted]
type = crypt
remote = Hetzner4:PMS
password = XXX

[Hetzner5]
type = sftp
host = XXX
user = XXX
pass = XXX
shell_type = unix
md5sum_command = md5 -r
sha1sum_command = sha1 -r
port = 23

[Hetzner5Encrypted]
type = crypt
remote = Hetzner5:PMS
password = XXX

[HetznerCloud]
type = union
upstreams = Hetzner5Encrypted: Hetzner4Encrypted: Hetzner3Encrypted: Hetzner2Encrypted: Hetzner1Encrypted: HetznerEncrypted:

[HetznerEncrypted]
type = crypt
remote = Hetzner:PMS
password = XXX

And here are the logs : rrrrrCloneee Deboguuae · GitHub

kapitainsky · January 29, 2025, 7:58am

Unfortunately it is not obvious for me what the issue is.

First you can try some simple tweaks - do not use IPV6 (often causes problems) and lower number of connections (Hetzner allows only 10). Add --bind 0.0.0.0 --checkers 2 --transfers 2.

If it fails then you have to try to isolate what is causing the issue. Stop using union. Test sync with every single Hetzenr account - maybe one is misconfigured (wrong user, wrong permissions)?

Hetzner is not some exotic provider and there are multiple users here using it without any issues (or we would see more issues similar to yours). It points into your setup specific thing.

PortX_Charge · January 30, 2025, 1:03pm

I tried to do with the options you sent me, same issue.
All my accounts are working fine as I tested all rclone ls HetznerX and rclone ls HetznerXEncrypted, they show me my files.
I can't try to sync each Hetzner account as all the Hetzner are full except the last one (Hetzner5):

sudo rclone about HetznerCloud:
Total:   120 TiB
Used:    110.620 TiB
Free:    9.380 TiB

I contacted Hetzner about the issue and they told me that maybe it's because the Storage Box is full and maybe creating the error but I can't get it as Union should be doing the job and send the files to the one that is not right ?

kapitainsky · January 30, 2025, 1:19pm

Try one (H5encrypted) with free space. Post sync and about results.

Use some test data and directory to limit log file size.

We will get to the bottom of it. I have a feeling that it is something obvious but can’t see it yet:(

Post also about of all other crypt remotes.

@asdffdsa do you know if snapshots are included in Hetzner account quotas? Maybe they are and about is not aware of it.

PortX_Charge · January 30, 2025, 1:52pm

I'm in the same case as you, I just don't understand why, I sent 100To days before using Union but the few last datas that I need to send are making me crazy.

All good when I sync to the Hetzner5Encrypted : Logs · GitHub
Theses exact same files are in Failed status when using Union.

If I'm right, Snapshot are included in quotas but should be seen by the rclone about if activated on Hetzner panel, I just don't use it (and no snapshot created) :

sudo rclone about HetznerEncrypted:
Total:   20 TiB
Used:    20.000 TiB
Free:    395.222 MiB

sudo rclone about Hetzner1Encrypted:
Total:   20 TiB
Used:    20.000 TiB
Free:    281.564 MiB

sudo rclone about Hetzner2Encrypted:
Total:   20 TiB
Used:    20.000 TiB
Free:    316.985 MiB

sudo rclone about Hetzner3Encrypted:
Total:   20 TiB
Used:    20.000 TiB
Free:    233.604 MiB

sudo rclone about Hetzner4Encrypted:
Total:   20 TiB
Used:    19.999 TiB
Free:    1.002 GiB

sudo rclone about Hetzner5Encrypted:
Total:   20 TiB
Used:    10.627 TiB
Free:    9.373 TiB

kapitainsky · January 30, 2025, 2:49pm

Until we find real reason you can mark other upstream remotes as no create

[HetznerCloud]
type = union
upstreams = Hetzner5Encrypted: Hetzner4Encrypted::nc Hetzner3Encrypted::nc Hetzner2Encrypted::nc Hetzner1Encrypted::nc HetznerEncrypted::nc

This should do the trick temporarily

BTW - are all crypts the same? I mean are they all using the same password?

kapitainsky · January 30, 2025, 2:57pm

Eureka! - I think... but close to sure.

Default union's create policy is epmfs which means that new content is written to a remote where the path already exist. Which is not necessary remote with the most free space (it was when it was written first time to a new path).

Change your union to:

[HetznerCloud]
type = union
upstreams = Hetzner5Encrypted: Hetzner4Encrypted: Hetzner3Encrypted: Hetzner2Encrypted: Hetzner1Encrypted: HetznerEncrypted:
action_policy = epall
create_policy = mfs
search_policy = ff

Default epmfs works only when content is nicely distributed between many dirs in root. E.g. some backup software using files sharding. IMO it is not the best choice for default. But it is too late to change it as it would break many people's setup (even I am sure that most do not even realise that what they've got is not what they wanted).