Union remote doesn’t switch when first remote is full (update)

Help and Support
1

What is the problem you are having with rclone?

Receiving error "The user's Drive storage quota has been exceeded" on a Union drive which is not full.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0

  • os/version: ubuntu 20.04 (64 bit)
  • os/kernel: 5.11.0-41-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.17.8
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync <local path> gdrive-union:

I am confident the issue I'm having is the same as this forum topic found here. However the topic is closed and I cannot comment.
Basically doing a sync on a UNION gdrive where the first drive is full causes it to fail and not move to the next available disk. Nick Craig-Wood (don't want to ping) suggested the possibility of implementing a minfreespace option before moving to the next drive.
I believe this option would solve my problem but wondered if this has been implemented or if someone found a way around this yet. I searched the global tags for both "free" and "min" reading tags and didn't find one matching this.

This is my first post sorry if missing something relevant.

2

Can you post your redacted config please so we can see how the union is configured?

3 
{
    "gdrive-union": {
        "type": "union",
        "upstreams": "gdrive1: gdrive2: gdrive3: gdrive4:"
    },
    "gdrive1": {
        "client_id": "1.apps.googleusercontent.com",
        "client_secret": "G",
        "scope": "drive",
        "team_drive": "",
        "token": "{\"access_token\":\"ya2dg\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//04Qt8\",\"expiry\":\"2022-03-27T20:23:05.388418026-04:00\"}",
        "type": "drive"
    },
    "gdrive2": {
        "client_id": "1.apps.googleusercontent.com",
        "client_secret": "G",
        "scope": "drive",
        "team_drive": "",
        "token": "{\"access_token\":\"yoQ\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//040e-L2rE\",\"expiry\":\"2022-03-27T20:23:05.362010657-04:00\"}",
        "type": "drive"
    },
    "gdrive3": {
        "client_id": "1.apps.googleusercontent.com",
        "client_secret": "G",
        "scope": "drive",
        "team_drive": "",
        "token": "{\"access_token\":\"yWQ\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//04UE\",\"expiry\":\"2022-03-27T20:23:05.368428927-04:00\"}",
        "type": "drive"
    },
    "gdrive4": {
        "client_id": "1.apps.googleusercontent.com",
        "client_secret": "G",
        "scope": "drive",
        "team_drive": "",
        "token": "{\"access_token\":\"yw\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//04c\",\"expiry\":\"2022-03-27T20:23:05.367419975-04:00\"}",
        "type": "drive"
    },
    "sparedrive": {
        "client_id": "1.apps.googleusercontent.com",
        "client_secret": "G",
        "scope": "drive",
        "team_drive": "",
        "token": "{\"access_token\":\"yaE\",\"token_type\":\"Bearer\",\"refresh_token\":\"1//04dA\",\"expiry\":\"2022-03-25T16:08:22.957979-04:00\"}",
        "type": "drive"
    }
}

rclone about gdrive-union:
Total:   60 GiB
Used:    41.560 GiB
Free:    18.440 GiB
Trashed: 716 B
Other:   0 B
4

Are all your remotes shared drives? Or normal drives? Or a mixture? I suspect they are a mixture.

Unfortunately shared drives don't know how much space they have as there is no API for that so they will never report as being full.

5

During configuration I set all drives up as normal drives. The 4 drives are all just free google accounts with random email addresses so I'm not sure how I could have made them shared drives but I'm not sure how to check that.
Is the API for reporting as being full different than the API used in the rclone about drive:? At minimum I'm able to see the storage state using that command on all 4 drives.
So to answer your question, I believe they're all normal drives with no shared drives in the union.

6

OK that makes sense.

Can you work out which drive is giving that error? Try rclone about on it to see if the output looks sensible.

7

I suppose I forgot to mention one important fact in my original post. When I first did a sync to these drives everything worked as expected. One drive filled up, then it moved onto a second drive and filled it up as expected. However trying to write anything NEW to the union causes the disk full error. My assumption is that on the initial check it sees the first disk is full and therefore errors out. However I don't know how it works so it's speculation. I think it was discussed in the other forum I linked which is why I forgot to mention.

All of the drives report correctly. Two of them are full from the initial sync. The about can be seen below:

rclone about gdrive1:
Total:   15 GiB   
Used:    9.101 GiB
Free:    5.899 GiB
Trashed: 0 B      
Other:   0 B 
rclone about gdrive2:
Total:   15 GiB    
Used:    14.968 GiB
Free:    33.111 MiB
Trashed: 0 B       
Other:   0 B  
rclone about gdrive3:
Total:   15 GiB    
Used:    2.525 GiB 
Free:    12.475 GiB
Trashed: 358 B     
Other:   0 B 
rclone about gdrive4:
Total:   15 GiB    
Used:    14.967 GiB
Free:    33.767 MiB
Trashed: 358 B     
Other:   0 B

All files I'm syncing are 50MiB so it's likely failing when it hits one of the 33MiB Free drives.
In the other forum you made a post about why determining no free space is difficult but had an idea to implement on a minfreespace flag where if the disk has less than the min configured free space to move to the next disk. Here is that post.

8

As a workaround you can set all full drives as read-only. For example if only gdrive4 had free space the relevant line in rclone.conf would look like this:

upstreams = gdrive1::ro gdrive2::ro gdrive3::ro gdrive4:

As you can guess the ':ro' seals all 3 upstreams so rclone is forced to fill gdrive4.

That's what I did with upstreams that can't report free space. I'm already used to working like this. Just toggle the ':ro's as needed.

9

Ah, that makes sense now!

Yes that would be a useful flag wouldn't it!

Rclone could try a bit harder to figure out if a file would actually fit on a remote also.

We don't always know the size of a file when we start the upload though - in particular if you are using rclone mount without --vfs-cache-mode writes or above we don't know the size.

Because there are often multiple files being uploaded at once, this would need rclone to keep track of free space commited but not reported yet. Also the free space measurements often lag just to add further complications...

A minfreespace option would be easier to implement!

As a first step, please could you open a new issue on Github about minfreespace for the union backend.

Do you want to implement this? I can give you hints if you want?

That is a fantastic workaround - thank you.

10

As a first step, please could you open a new issue on Github about minfreespace for the union backend.

I will do this today and try to put appropriate details.

Do you want to implement this? I can give you hints if you want?

I've never coded in GO and never done much beyond basic python scripting but I would be willing to take a look.
Thank you for all your help with working through this with me!

1 Like
11

I can give you hints if you want?

I didn't specifically ask for it but I will take some hits on where to get started. I'm not familiar with the code base at all. Thanks again!

12

I will reply on the Github issue: Add minfreespace flag for creation policies from mergerfs · Issue #6071 · rclone/rclone · GitHub

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.