Union mount space calculation

What is the problem you are having with rclone?

Mounting a union of a google drive and a local SMB backed filesystem prevents uploading files.

Run the command 'rclone version' and share the full output of the command.

rclone version

rclone v1.63.0

  • os/version: arch "rolling" (64 bit)
  • os/kernel: 6.4.3-arch1-1 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.5
  • go/linking: dynamic
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

union between Google Drive & alias backed systemd automounted SMB folder

The command you were trying to run (eg rclone copy /tmp remote:tmp)

The google drive, 197TB over a 5TB allocation due to googles recent policy changes
Mounted as read only

# rclone about gd: -vv
2023/08/27 14:13:29 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "about" "gd:" "-vv"]
2023/08/27 14:13:29 DEBUG : Creating backend with remote "gd:"
2023/08/27 14:13:29 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
Total:   5 TiB
Used:    203.931 TiB
Free:    off
Trashed: 773.763 GiB
Other:   0 B
2023/08/27 14:13:30 DEBUG : 6 go routines active

The alias backed SMB filesystem - 133TB free

# rclone about mtankmedia: -vv
2023/08/27 14:14:16 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "about" "mtankmedia:" "-vv"]
2023/08/27 14:14:16 DEBUG : Creating backend with remote "mtankmedia:"
2023/08/27 14:14:16 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/08/27 14:14:16 DEBUG : Creating backend with remote "/mnt/mtankmedia"
2023/08/27 14:14:16 DEBUG : fs cache: renaming cache item "mtankmedia:" to be canonical "/mnt/mtankmedia"
Total:   133.285 TiB
Used:    0 B
Free:    133.285 TiB
2023/08/27 14:14:16 DEBUG : 4 go routines active

and the union mount which I expected to show

  • capacity = sum of google + local
  • free as just SMB share due to google being marked read only
### rclone about allmedia: -vv
2023/08/27 14:14:49 DEBUG : rclone: Version "v1.63.0" starting with parameters ["rclone" "about" "allmedia:" "-vv"]
2023/08/27 14:14:49 DEBUG : Creating backend with remote "allmedia:"
2023/08/27 14:14:49 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/08/27 14:14:49 DEBUG : Creating backend with remote "mtankmedia:"
2023/08/27 14:14:49 DEBUG : Creating backend with remote "gd:Media"
2023/08/27 14:14:49 DEBUG : Creating backend with remote "/mnt/mtankmedia"
2023/08/27 14:14:49 DEBUG : fs cache: renaming cache item "mtankmedia:" to be canonical "/mnt/mtankmedia"
2023/08/27 14:14:49 DEBUG : fs cache: switching user supplied name "mtankmedia:" for canonical name "/mnt/mtankmedia"
2023/08/27 14:14:49 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
Total:   138.285 TiB
Used:    203.931 TiB
Free:    off
2023/08/27 14:14:50 DEBUG : 6 go routines active

rclone mount google drive command

rclone mount gd:Media /opt/gd-media \
                            --gid 1001 --uid 1001 \
                            --daemon-wait=5m \
                            --fast-list \
                            --dir-cache-time 5000h \
                            --poll-interval 10s \
                            --drive-pacer-min-sleep 10ms \
                            --drive-pacer-burst 200 \
                            --config /root/.config/rclone/rclone.conf \
                            --log-file=/var/log/rclone.log \
                            --log-level=INFO \
                            --allow-other \
                            --drive-chunk-size=128M \
                            --cache-dir=/mnt/Cache \
                            --vfs-cache-mode full \
                            --vfs-cache-max-size 256G \
                            --vfs-cache-max-age 5000h \
                            --vfs-read-ahead 1G

rclone alias local SMB share from NAS (not optimised because of testing SMB vs alias + union)

/rclone mount allmedia: /opt/all-media \
                            --gid 1001 --uid 1001 \
                            --config /root/.config/rclone/rclone.conf \
                            --log-file=/var/log/rclone-allmedia.log \
                            --log-level=DEBUG

The rclone config contents with secrets removed.

[gd]
type = drive
client_id = <SNIP>.apps.googleusercontent.com
client_secret = <SNIP>
scope = drive
root_folder_id = <SNIP>
service_account_file =
token = {"access_token":"<SNIP>","expiry":"2023-08-27T15:30:25.131088687-07:00"}


[mtankmedia]
type = alias
remote = /mnt/mtankmedia

[allmedia]
type = union
upstreams = gd:Media:ro mtankmedia:

A log from the command with the -vv flag

2023/08/27 15:02:41 DEBUG : rclone: Version "v1.63.0" starting with parameters ["/usr/bin/rclone" "mount" "allmedia:" "/opt/all-media" "--gid" "1001" "--uid" "1001" "--config" "/root/.config/rclone/rclone.conf" "--log-file=/var/log/rclone-allmedia.log" "--log-level=DEBUG"]
2023/08/27 15:02:41 DEBUG : Creating backend with remote "allmedia:"
2023/08/27 15:02:41 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/08/27 15:02:41 DEBUG : Creating backend with remote "mtankmedia:"
2023/08/27 15:02:41 DEBUG : Creating backend with remote "/mnt/mtankmedia"
2023/08/27 15:02:41 DEBUG : Creating backend with remote "gd:Media"
2023/08/27 15:02:41 DEBUG : fs cache: renaming cache item "mtankmedia:" to be canonical "/mnt/mtankmedia"
2023/08/27 15:02:41 DEBUG : fs cache: switching user supplied name "mtankmedia:" for canonical name "/mnt/mtankmedia"
2023/08/27 15:02:41 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2023/08/27 15:02:41 INFO  : union root '': poll-interval is not supported by this remote
2023/08/27 15:02:41 DEBUG : union root '': Mounting on "/opt/all-media"
2023/08/27 15:02:41 DEBUG : : Root:
2023/08/27 15:02:41 DEBUG : : >Root: node=/, err=<nil>
2023/08/27 15:02:48 DEBUG : /: Lookup: name="TV"
2023/08/27 15:02:49 DEBUG : /: >Lookup: node=TV/, err=<nil>
2023/08/27 15:02:49 DEBUG : TV/: Attr:
2023/08/27 15:02:49 DEBUG : TV/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/08/27 15:05:11 DEBUG : : Statfs:
2023/08/27 15:05:11 DEBUG : : >Statfs: stat={Blocks:37120675328 Bfree:4503582005646051 Bavail:4503582005646051 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

and from sonarr

Aug 27 15:04:51 dockerhost docker[916260]: [Warn] FreeSpaceSpecification: Not enough free space (0) to import: /downloads/completed/Series/<snip>.mkv (45692457)

EDIT: Also tried using mfs (most free space) for action and create poliicies but no difference.

As per docs:

Quota Relevant Policies

Some policies rely on quota information. These policies should be used only if your upstreams support the respective quota fields.

Policy Required Field
lfs, eplfs Free
mfs, epmfs Free
lus, eplus Used
lno, eplno Objects

Default policies:

action - epall
create - epmfs
search - ff

In your case as Free space is not available you should try lus or eplus for create policy.

And this is what you see:

Total capacity is total Google drive + total SMB = 5TiB + 133TiB = 138TiB
Used is 0TiB + 203 TiB = 203 TiB
Free is not available as it requires all remotes to support it.

The problem is Google reporting Total < Used.

Also probably good point that read only remotes should not be included in Free Space calculations including when this value is not available. Feel free to change it and submit PR - would be nice improvement.

Thanks @kapitainsky , you confirmed my suspicions.
I'm actually surprised other folks haven't attempted something similar, leaving 100's of TB of data in google drive and supplementing with local storage seems like a handy use case for rclone. The union mount would enable sonarr/radarr to leverage the combined space without needing to go to the hassle of splitting library management for local and remote space.

I tried with the 'lus' mode as well but no luck, Google's used space (even though :ro) is skewing calculations.

2023/08/28 06:19:11 DEBUG : rclone: Version "v1.63.0" starting with parameters ["/usr/bin/rclone" "mount" "allmedia:" "/opt/
all-media" "--gid" "1001" "--uid" "1001" "--config" "/root/.config/rclone/rclone.conf" "--log-file=/var/log/rclone-allmedia.
log" "--log-level=DEBUG" "--allow-non-empty"]
2023/08/28 06:19:11 DEBUG : Creating backend with remote "allmedia:"
2023/08/28 06:19:11 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/08/28 06:19:11 DEBUG : Creating backend with remote "mtankmedia:"
2023/08/28 06:19:11 DEBUG : Creating backend with remote "/mnt/mtankmedia"
2023/08/28 06:19:11 DEBUG : Creating backend with remote "gd:Media"
2023/08/28 06:19:11 DEBUG : fs cache: renaming cache item "mtankmedia:" to be canonical "/mnt/mtankmedia"
2023/08/28 06:19:11 DEBUG : fs cache: switching user supplied name "mtankmedia:" for canonical name "/mnt/mtankmedia"
2023/08/28 06:19:11 DEBUG : union root '': actionPolicy = *policy.EpLus, createPolicy = *policy.EpLus, searchPolicy = *policy.EpFF
2023/08/28 06:19:11 INFO  : union root '': poll-interval is not supported by this remote
2023/08/28 06:19:11 DEBUG : union root '': Mounting on "/opt/all-media"
2023/08/28 06:19:11 DEBUG : : Root:
2023/08/28 06:19:11 DEBUG : : >Root: node=/, err=<nil>
2023/08/28 06:19:37 DEBUG : /: Lookup: name="TV"
2023/08/28 06:19:37 DEBUG : /: >Lookup: node=TV/, err=<nil>
2023/08/28 06:19:37 DEBUG : TV/: Attr:
2023/08/28 06:19:37 DEBUG : TV/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/08/28 06:20:08 DEBUG : : Statfs:
2023/08/28 06:20:08 DEBUG : : >Statfs: stat={Blocks:37120675584 Bfree:4503582005646051 Bavail:4503582005646051 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2023/08/28 06:22:58 DEBUG : /: Lookup: name="Movies"
2023/08/28 06:22:58 DEBUG : /: >Lookup: node=Movies/, err=<nil>
2023/08/28 06:22:58 DEBUG : Movies/: Attr:
2023/08/28 06:22:58 DEBUG : Movies/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/08/28 06:25:09 DEBUG : : Statfs:
2023/08/28 06:25:09 DEBUG : : >Statfs: stat={Blocks:37120675584 Bfree:4503582005646051 Bavail:4503582005646051 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
Aug 28 06:26:09 dockerhost docker[2026140]: [Warn] FreeSpaceSpecification: Not enough free space (0) to import: /downloads/completed/Series/<snip>.mkv (2942333853)
Aug 28 06:26:31 dockerhost docker[2031497]: [Warn] FreeSpaceSpecification: Not enough free space (0) to import: /downloads/completed/Movies/<snip>.mp4 (935486639)

Sadly a code change is beyond my abilities right now. I'll enquire with @ncw to see if funded support is an option here.

For anyone else hitting this issue, mergerfs was able to work around the problem.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.