Yandex Union not working as Expected

Help and Support
1

What is the problem you are having with rclone?

The Yandex disk union isn't working as expected. I keep getting an error saying the disk space is full even though the union has disks/remotes that are not full.

Run the command 'rclone version' and share the full output of the command.

rclone v1.65.2

  • os/version: debian 12.5 (64 bit)
  • os/kernel: 6.1.0-18-cloud-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.21.6
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Yandex Disk

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy HAE: yandexunionc:Google --filter-from text.txt --transfers 4 -P --no-check-dest

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[HA]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX
root_folder_id = XXX

[HAE]
type = crypt
remote = HA:E
filename_encryption = standard
directory_name_encryption = true
password = XXX
password2 = XXX

[yandex]
type = yandex
token = XXX

[yandexe]
type = crypt
remote = yandex:Shared
password = XXX
password2 = XXX

[yandexunion]
type = union
upstreams = yandex:Shared2 yandex:Shared3 yandex:Shared4 yandex:Shared5 yandex:Shared6

[yandexunionc]
type = crypt
remote = yandexunion:
password = XXX
password2 = XXX

A log from the command that you were trying to run with the -vv flag

Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.1sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.6s
2024/02/28 16:11:23 DEBUG : DL-sep-1: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DL-aug-6: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DL-augg-5: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DL-aug-4: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : Aug-DL-4: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : Aug-DL-3: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DL-Aug-2: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : Aug-DL-2: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : Aug-DL: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DD-JULY-19: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : GT: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : July-DD-New-9th: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : July-DD-0: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : TG: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : June-DD-24: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NEWMFEW: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NewFMay: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NEW_DD_Happens: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NEW-NEW-04-23: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NDGNewTr: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : DRNEW-TR: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : G-Backup-NotImportant: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-DD-New-DD: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-DD-jan-23: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-D-jan-23: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-B-Jan: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-D-jan-jan: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-D-Jan: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : D-New-Jan: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : D-DRIVE: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : Newww-D: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : NEW-T: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : New-D: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : G1: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : ASU: Excluded
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : H's Resume.pdf: Need to transfer - File not found at Destination
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         7.9s
2024/02/28 16:11:23 DEBUG : favorites_2_9_21.html: Need to transfer - File not found at Destination
Transferred:   	          0 B / 21.942 KiB, 0%, 0 B/s, ETA -
Transferred:            0 / 1, 0%
Elapsed time:         7.9sTransferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 2, 0%
Elapsed time:         8.1s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
2024/02/28 16:11:24 DEBUG : H/.inside: Need to transfer - File not found at Destination
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 2, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
2024/02/28 16:11:24 DEBUG : H/.outside: Need to transfer - File not found at Destination
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 3, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
2024/02/28 16:11:24 DEBUG : H/VID_35820102_011141_607.mp4: Need to transfer - File not found at Destination
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Transferred:            0 / 3, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
2024/02/28 16:11:24 DEBUG : H/recording_17112021_014820.mp4: Need to transfer - File not found at Destination
Transferred:   	          0 B / 1.021 MiB, 0%, 0 B/s, ETA -
Transferred:            0 / 5, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
 *                                  H/.inside: transferring
2024/02/28 16:11:24 DEBUG : H/recording_17112021_015448.mp4: Need to transfer - File not found at Destination
Transferred:   	          0 B / 22.172 MiB, 0%, 0 B/s, ETA -
Transferred:            0 / 6, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
 *                                  H/.inside: transferring
2024/02/28 16:11:24 DEBUG : H/recording_17112021_020119.mp4: Need to transfer - File not found at Destination
Transferred:   	          0 B / 48.996 MiB, 0%, 0 B/s, ETA -
Transferred:            0 / 6, 0%
Elapsed time:         8.3s
Transferring:
 *                          H's Resume.pdf: transferring
 *                         favorites_2_9_21.html: transferring
 *                                  H/.inside: transferringTransferred:   	          0 B / 84.656 MiB, 0%, 0 B/s, ETA -

Transferred:   	          0 B / 10.670 GiB, 0%, 0 B/s, ETA -
Transferred:            0 / 753, 0%
Elapsed time:        13.8s
Transferring:
 *                          H's Resume.pdf:  0% /21.942Ki, 0/s, -
 *                         favorites_2_9_21.html:  0% /4.651Mi, 0/s, -
 *                                  H/.inside:  0% /0, 0/s, -
 *                                 H/.outside:  0% /0, 0/s, -
2024/02/28 16:11:29 ERROR : H's Resume.pdf: Failed to copy: [507 - DiskOwnerStorageQuotaExhaustedError] Owner of shared folder has exhausted storage quota. (У владельца общей папки недостаточно свободного места.)
2

run rclone about yandex and post results here

3

Here you go:

rclone about yandexunion:
Total:   5.024 TiB
Used:    248.527 GiB
Free:    4.782 TiB

This is the weird thing, there's enough space in other drives but as soon as the first drive fills up, rclone starts returning an error even though the other drives are completely empty.

4

I'm not sure I'm following the config.

That all points back to the same 1 remote? What do you mean other ones?

5

Good point. And it makes sense now. The same rclone about value is returned for all union members so the first one is always picked up.

It must me some yandex story. Shared folders are from other accounts most likely. But rclone can not determine free space individually.

@jxmese can you create individual accounts? Not try to access it using some yandex sharing over one account only?

6

@Animosity022, @kapitainsky, ahh yes, that could be it.

I'll give it a try by adding the other accounts to rclone and test it.

7

start with two for test - should be enough to prove/disapprove this theory

8

@kapitainsky @Animosity022, oh wait, I just remembered why I did things this way in the first place.

So, the reason I'm trying to access all shared folders from one account is because for every account you make on Yandex, you need a new phone number.

As a workaround, what I've done is, I've created one main account and the rest are all "child accounts". Child accounts (accounts for children) don't need a number to sign up and can use the main account (parent's account) number while getting the same benefits for mail and disk space.

However, the caveat is that "child accounts" don't support third-party apps on Yandex:

image
image652×634 52.5 KB

This is why I tried creating shared folders and accessing them through the main account. does this make sense?

Is there any workaround for this?

Again, thanks for your help. I appreciate it.

9

You could try different union policies.

Instead of default create epmfs policy use rand. It won't be perfect and you might have to run your copy few times to copy all content. Especially when close to filling in all space. But if your start with all shares empty or equally full it should work very well.

As long as minimum any share free space is less than maximum file size you copy it will work 100% fine.

Depending on what you need it for it might be good enough. It will include some "randomness" in its logic you might have to take into account.

10

@kapitainsky, thanks. Let me try that and get back to you right now.

11

To improve efficiency of such approach you might consider using chunker overlay. But it all depends what you use it for. Maybe no need

12

Yeah, I tried using chunker before and it didn't help. I've also just tried using rand and it's giving the same error - owner of the disk has reached quota. I've even made sure that the main account I'm using to access is not part of the union. Only the child accounts are part of the union. So, theoretically, the owner account has 100% free disk space as rclone is only using the other accounts for the union.

13

Different policies won't matter if the one account is the problem.

You need different upstreams to support about for free space detection and your setup doesn't do that.

Anything else you managed to get working I think would be pure luck at that point.

14

I would still try. Create different remotes - all effectively pointing into the same account:

[yandex]
type = yandex
token = XXX

[yandex1]
type = yandex
token = XXX

...

and then:

[yandexunion]
type = union
upstreams = yandex:Shared2 yandex1:Shared3  ...

It should trick rclone to believe that they are different.

And use rand.

15

That's just RNG for putting in the right spot as all the remotes report the same free space.

May as well just put rand in the first one if that's the case.

16

okay, I've just tried it and it still gives the same error. Is there no way to actually make rclone treat the folders as separate remotes?

Here's what I used for the config:


[yandex]
type = yandex
token = XXX

[yandex1]
type = yandex
token = XXX

[yandex2]
type = yandex
token = XXX

[yandex3]
type = yandex
token = XXX

[yandex4]
type = yandex
token = XXX

[yandexunion]
type = union
upstreams = yandex:Shared2 yandex1:Shared3 yandex2:Shared4 yandex3:Shared5 yandex4:Shared6

[yandexunionc]
type = crypt
remote = yandexunion:
password = XXX
password2 = XXX
17

Can you upload anything using yandex web UI? As other possibility is that they anticipated such "clever" workarounds:)

18

Yep, that's the thing. I can upload using the web UI. To all the shared folders. Using the main account. Just not through rclone. Which is what prompted me to make this post because if I can still upload to any folder using the web UI then theoretically, I should be able to somehow do that using rclone.

19

Can you upload using rclone to share where there is still free space? Without using union.

20

Another interesting thing - Yes I can. Using rclone, I can upload to any of the "child" shared folders through the main account.

The reason I wanted to use Union was because:

  1. To combine all the shared folders and use them as one drive instead of figuring out which files to keep where.
  2. To use crypt on the combined folders.

So, in summary:

  1. The web UI works
  2. Rclone works for uploading to all folders using any account as long as the folders are shared.
  3. Rclone Union doesn't seem to work even though all folders are shared between accounts.