Google Drive mount works, does not as a Union remote

o1o1oo1 · July 12, 2023, 12:34pm

What is the problem you are having with rclone?

I have a Google Drive repository of media files. Read only access is enough. Big flat structure (>100k files/1 dir). Rclone works fine on this remote. Rclone mount is slow to populate (~30 min) but then works fine. The same setup as part of a union remote (Google Drive + local storage) only lists local storage, never gets populated with Google Drive content.

Run the command 'rclone version' and share the full output of the command.

rclone v1.63.0

os/version: darwin 13.4.1 (64 bit)
os/kernel: 22.5.0 (x86_64)
os/type: darwin
os/arch: amd64
go/version: go1.20.5
go/linking: dynamic
go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive, local storage, union remote

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount media-union: '/Users/user/placeholder' \
--config /Users/user/.rclone.conf \
-vvv \
--volname Media \
--noappledouble \
--noapplexattr \
--allow-non-empty \
--allow-other \
--dir-cache-time 4800h \
--umask 002 \
--use-mmap \
--cache-info-age 2400h \
--buffer-size 0 \
--poll-interval 9m \
--timeout 20m \
--cache-read-retries 15 \
--cache-db-purge \
--no-checksum \
--use-server-modtime \
--no-gzip-encoding \
--no-update-modtime \
--no-seek \
--poll-interval 30s \
--vfs-cache-mode full

As you can see I've tried many switches. The same command with a different remote (Azure) works fine. Direct rclone access (ls, led, etc.) with encrypted and raw google drive remotes works as expected.

The rclone config contents with secrets removed.

[media-union]
type = union
upstreams = "/Users/user/Pictures/media" googledrive_media_encrypted:ro

[googledrive_media_encrypted]
type = crypt
remote = googledrive:media/
password = redacted
password2 = redacted

A log from the command with the `-vv` flag

2023/07/12 08:25:03 DEBUG : Setting --exclude-from "[/Users/user/TimeMachineExclusionList.txt]" from environment variable RCLONE_EXCLUDE_FROM="/Users/user/TimeMachineExclusionList.txt"
2023/07/12 08:25:03 DEBUG : rclone: Version "v1.63.0" starting with parameters ["/usr/local/bin/rclone" "mount" "media-union:" "/Users/user/placeholder" "--config" "/Users/user/.rclone.conf" "-vvv" "--volname" "media" "--noappledouble" "--noapplexattr" "--allow-non-empty" "--allow-other" "--dir-cache-time" "4800h" "--umask" "002" "--use-mmap" "--cache-info-age" "2400h" "--buffer-size" "0" "--poll-interval" "9m" "--timeout" "20m" "--cache-read-retries" "15" "--cache-db-purge" "--no-checksum" "--use-server-modtime" "--no-gzip-encoding" "--no-update-modtime" "--no-seek" "--poll-interval" "30s" "--vfs-cache-mode" "full"]
2023/07/12 08:25:03 DEBUG : Creating backend with remote "media-union:"
2023/07/12 08:25:03 DEBUG : Using config file from "/Users/user/.rclone.conf"
2023/07/12 08:25:03 DEBUG : Creating backend with remote "googledrive_media_encrypted:ro"
2023/07/12 08:25:03 DEBUG : Creating backend with remote "/Users/user/Pictures/Photos local"
2023/07/12 08:25:03 DEBUG : Creating backend with remote "googledrive:media/6s8nkctj3kppb3gbim6s8q1nds"
2023/07/12 08:25:04 DEBUG : Google drive root 'RAWs/6s8nkctj3kppb3gbim6s8q1nds': 'root_folder_id = 0ANeGUGkJGJCCUk9PVA' - save this in the config to speed up startup
2023/07/12 08:25:05 DEBUG : union root '': actionPolicy = *policy.EpAll, createPolicy = *policy.EpMfs, searchPolicy = *policy.FF
2023/07/12 08:25:05 INFO  : union root '': poll-interval is not supported by this remote
2023/07/12 08:25:05 DEBUG : vfs cache: root is "/Users/user/Library/Caches/rclone"
2023/07/12 08:25:05 DEBUG : vfs cache: data root is "/Users/user/Library/Caches/rclone/vfs/media-union"
2023/07/12 08:25:05 DEBUG : vfs cache: metadata root is "/Users/user/Library/Caches/rclone/vfsMeta/media-union"
2023/07/12 08:25:05 DEBUG : Creating backend with remote "/Users/user/Library/Caches/rclone/vfs/media-union/"
2023/07/12 08:25:05 DEBUG : fs cache: renaming cache item "/Users/user/Library/Caches/rclone/vfs/media-union/" to be canonical "/Users/user/Library/Caches/rclone/vfs/media-union"
2023/07/12 08:25:05 DEBUG : Creating backend with remote "/Users/user/Library/Caches/rclone/vfsMeta/media-union/"
2023/07/12 08:25:05 DEBUG : fs cache: renaming cache item "/Users/user/Library/Caches/rclone/vfsMeta/media-union/" to be canonical "/Users/user/Library/Caches/rclone/vfsMeta/media-union"
2023/07/12 08:25:05 DEBUG : Mounting on "/Users/user/placeholder" ("media")
2023/07/12 08:25:05 DEBUG : Adding "-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC" for macOS
2023/07/12 08:25:05 DEBUG : union root '': Mounting with options: ["-o" "attr_timeout=1" "-o" "fsname=media-union:" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "daemon_timeout=600" "-o" "allow_other" "-o" "volname=media" "-o" "noappledouble" "-o" "noapplexattr" "-o" "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"]
2023/07/12 08:25:05 INFO  : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
2023/07/12 08:25:05 DEBUG : union root '': Init: 
2023/07/12 08:25:05 DEBUG : union root '': >Init: 
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Getattr: fh=0xFFFFFFFFFFFFFFFF
2023/07/12 08:25:05 DEBUG : /: >Getattr: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: Statfs: 
2023/07/12 08:25:05 DEBUG : /: >Statfs: stat={Bsize:4096 Frsize:4096 Blocks:4294967295 Bfree:4294967295 Bavail:4294967295 Files:1000000000 Ffree:1000000000 Favail:0 Fsid:0 Flag:0 Namemax:255}, errc=0
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
2023/07/12 08:25:05 DEBUG : /: Access: mask=00
2023/07/12 08:25:05 DEBUG : /: >Access: errc=0
(...)
2023/07/12 08:25:06 DEBUG : /filename-2-2.dng: >Getattr: errc=0
2023/07/12 08:25:06 DEBUG : /filename-2-2.dng: Access: mask=00
2023/07/12 08:25:06 DEBUG : /filename-2-2.dng: >Access: errc=0
2023/07/12 08:25:06 DEBUG : /filename-3.dng: Getattr: fh=0xFFFFFFFFFFFFFFFF
(...)

kapitainsky · July 12, 2023, 12:37pm

It is not a bug but your mistake:

You have union of local and ro folder on google. Probably you do not have such folder so nothing is listed...

read only should be specified as:

googledrive_media_encrypted::ro

kapitainsky · July 12, 2023, 12:42pm

And on a side note I see that you are using vfs cache - which is always great - but you do not limit cache size... It can potentially eat all your computer free disk space:)

I would add:

--vfs-cache-max-size 100G

adjust it to what you can afford on your system.

o1o1oo1 · July 12, 2023, 3:36pm

Thank you. My bad, my face is all red

system · August 11, 2023, 3:37pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.