1.57 seems to cause "Directory already mounted, use --allow-non-empty to mount anyway" error under docker

martin · November 8, 2021, 1:31pm

What is the problem you are having with rclone?

When upgrading from 1.56.2 to 1.57.0 docker image that mounts a remote to be shared with other containers errors with

2021/11/08 13:10:09 Fatal error: Directory already mounted, use --allow-non-empty to mount anyway: /mnt

Presumably related to pull #5593

What is your rclone version (output from `rclone version`)

rclone/rclone:1.56.2 -> rclone/rclone:1.57.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Linux 64bit running docker (Docker version 20.10.7, build 20.10.7-0ubuntu1~20.04.1)

Which cloud storage system are you using? (eg Google Drive)

Google drive (although this wouldn't be specific to particular backend)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

docker-compose.yaml

version: "3.8"
services:
  rclone-gsuite:
    #image: rclone/rclone:1.56.2
    image: rclone/rclone:1.57.0
    container_name: rclone-gsuite-tmp
    cap_add:
      - SYS_ADMIN
    devices:
      - /dev/fuse
    security_opt:
      - "apparmor:unconfined"
    environment:
      - RCLONE_ALLOW_OTHER=true
      - RCLONE_VERBOSE=2
    volumes:
      - ~/.config/rclone:/config/rclone
      - type: bind
        source: ${PWD}/mnt
        target: /mnt
        bind:
          propagation: shared
    command: ["mount","Encrypt_gsuite:","/mnt"]

Command
docker-compose up

The rclone config contents with secrets removed.

[gsuite]
type = drive
client_id = ***secret***.apps.googleusercontent.com
client_secret = ***secret***
scope = drive
token = {"access_token":"***secret***","token_type":"Bearer","refresh_token":"***secret***","expiry":"2021-11-08T14:14:02.909276049Z"}
root_folder_id = ***secret***

[Encrypt_gsuite]
type = crypt
remote = gsuite:jottacloud
filename_encryption = standard
directory_name_encryption = true
password = ***secret***

A log from the command with the `-vv` flag

Attaching to rclone-gsuite-tmp
rclone-gsuite-tmp | 2021/11/08 13:20:31 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "mount" "Encrypt_gsuite:" "/mnt"]
rclone-gsuite-tmp | 2021/11/08 13:20:31 DEBUG : Creating backend with remote "Encrypt_gsuite:"
rclone-gsuite-tmp | 2021/11/08 13:20:31 DEBUG : Using config file from "/config/rclone/rclone.conf"
rclone-gsuite-tmp | 2021/11/08 13:20:31 DEBUG : Creating backend with remote "gsuite:jottacloud"
rclone-gsuite-tmp | 2021/11/08 13:20:31 Fatal error: Directory already mounted, use --allow-non-empty to mount anyway: /mnt

I think this is related to how host bind mounts are presented to the container.
For example

# df -h .
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/test--vg   62G   17G   43G  29% /
# docker run -it --rm --mount type=bind,source=${PWD}/mnt,target=/mnt,bind-propagation=shared ubuntu
root@cfe8a41ddd17:/# grep "/mnt" /proc/mounts
/dev/mapper/test--vg /mnt ext4 rw,relatime,errors=remount-ro 0 0

ivandeex · November 8, 2021, 2:15pm

rclone in 1.57 started to check /proc/mounts on linux.
For a quick fix just use --allow-non-empty.
If you want me to investigate, please replace command: in the descriptor by entrypoint: ["cat", "/proc/mounts"] rerun your container and paste here the output.

martin · November 8, 2021, 2:45pm

Well, almost certain that it'll be that entry in /proc/mounts that I mentioned above (which looks like a normal mount when in the container)

/dev/mapper/test--vg /mnt ext4 rw,relatime,errors=remount-ro 0 0

Full output

rclone-gsuite-tmp | overlay / overlay rw,relatime,lowerdir=/var/lib/docker/overlay2/l/NJ7CFN673BWIFV4GUHYFMS4J4P:/var/lib/docker/overlay2/l/RG6GBQ27POPHZ2UA47UU4ZWERZ:/var/lib/docker/overlay2/l/5XXGX5CT2RXTIWVFBIFIDRBTQW:/var/lib/docker/overlay2/l/BL5GGQLBFOSZQ3GQWA5AABOJW2:/var/lib/docker/overlay2/l/JIGKGGORJKQYPZZJRK3MEAJCC6:/var/lib/docker/overlay2/l/BNPWFYYDTVJHJKBV6YFA2ZG47G,upperdir=/var/lib/docker/overlay2/d73c4ae2355e76f3dcc006c8a7bd79a591108fa2cf2d685cdc533e9389528e60/diff,workdir=/var/lib/docker/overlay2/d73c4ae2355e76f3dcc006c8a7bd79a591108fa2cf2d685cdc533e9389528e60/work,xino=off 0 0
rclone-gsuite-tmp | proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | tmpfs /dev tmpfs rw,nosuid,size=65536k,mode=755 0 0
rclone-gsuite-tmp | devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=666 0 0
rclone-gsuite-tmp | sysfs /sys sysfs ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | tmpfs /sys/fs/cgroup tmpfs rw,nosuid,nodev,noexec,relatime,mode=755 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/systemd cgroup ro,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/blkio cgroup ro,nosuid,nodev,noexec,relatime,blkio 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/pids cgroup ro,nosuid,nodev,noexec,relatime,pids 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/memory cgroup ro,nosuid,nodev,noexec,relatime,memory 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/freezer cgroup ro,nosuid,nodev,noexec,relatime,freezer 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/rdma cgroup ro,nosuid,nodev,noexec,relatime,rdma 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/net_cls,net_prio cgroup ro,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/cpuset cgroup ro,nosuid,nodev,noexec,relatime,cpuset 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/cpu,cpuacct cgroup ro,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/perf_event cgroup ro,nosuid,nodev,noexec,relatime,perf_event 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/devices cgroup ro,nosuid,nodev,noexec,relatime,devices 0 0
rclone-gsuite-tmp | cgroup /sys/fs/cgroup/hugetlb cgroup ro,nosuid,nodev,noexec,relatime,hugetlb 0 0
rclone-gsuite-tmp | mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | shm /dev/shm tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0
rclone-gsuite-tmp | /dev/mapper/test--vg /mnt ext4 rw,relatime,errors=remount-ro 0 0
rclone-gsuite-tmp | /dev/mapper/test--vg /config/rclone ext4 rw,relatime,errors=remount-ro 0 0
rclone-gsuite-tmp | /dev/mapper/test--vg /etc/resolv.conf ext4 rw,relatime,errors=remount-ro 0 0
rclone-gsuite-tmp | /dev/mapper/test--vg /etc/hostname ext4 rw,relatime,errors=remount-ro 0 0
rclone-gsuite-tmp | /dev/mapper/test--vg /etc/hosts ext4 rw,relatime,errors=remount-ro 0 0
rclone-gsuite-tmp | proc /proc/bus proc ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | proc /proc/fs proc ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | proc /proc/irq proc ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | proc /proc/sys proc ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | proc /proc/sysrq-trigger proc ro,nosuid,nodev,noexec,relatime 0 0
rclone-gsuite-tmp | tmpfs /proc/acpi tmpfs ro,relatime 0 0
rclone-gsuite-tmp | tmpfs /proc/kcore tmpfs rw,nosuid,size=65536k,mode=755 0 0
rclone-gsuite-tmp | tmpfs /proc/keys tmpfs rw,nosuid,size=65536k,mode=755 0 0
rclone-gsuite-tmp | tmpfs /proc/timer_list tmpfs rw,nosuid,size=65536k,mode=755 0 0
rclone-gsuite-tmp | tmpfs /proc/sched_debug tmpfs rw,nosuid,size=65536k,mode=755 0 0
rclone-gsuite-tmp | tmpfs /proc/scsi tmpfs ro,relatime 0 0
rclone-gsuite-tmp | tmpfs /sys/firmware tmpfs ro,relatime 0 0
rclone-gsuite-tmp exited with code 0

At the moment, I'm just keeping on 1.56.2, I'm not sure about using the --allow-non-empty as the directory is empty but I would want to catch an issue if it wasn't (which this presumably wouldn't)

ivandeex · November 8, 2021, 3:34pm

@ncw

My recent mount related change caused backwards incompatibility for OP.

Previously rclone used non-empty mountpoint directory listing as an indicator that directory is already mounted. In the error message rclone suggested --allow-non-empty to override.

This behavior caused problems when mounted remote folder was empty. I kept it on BSD because I don't know a better way.

I changed the behavior on Linux which reveals kernel mount table to user space via /proc/mounts pseudo-file. Now rclone on Linux can look there and tell for sure whether directory is mounted or not, be it empty or stuffed with files.

I thought I'd better keep the name of the override flag as --allow-non-empty for backwards compatibility and compatibility with BSD.

However in this case the OP uses rclone in docker to do the mount and deliberately bind-mounts host directory into container with shared flag to later export in-container mount back on host. Rclone in container detects the previous bind-mount, rejects remount and suggests --allow-non-empty to override.

User is confused by the flag name as the host directory is in fact empty.

@ncw
I want your advise.
Should we go for extreme semantic truth and add differently named flags on different OSes with single function "mount override" and adjust error message accordingly?
Or documenting the flag name quirk is enough?

ncw · November 8, 2021, 4:27pm

There is of course an XKCD for this

Every bug fixed introduces an potential backwards incompatibility

There is an example we can explore in Install

# config on host at ~/.config/rclone/rclone.conf
# data on host at ~/data

# make sure the config is ok by listing the remotes
docker run --rm \
    --volume ~/.config/rclone:/config/rclone \
    --volume ~/data:/data:shared \
    --user $(id -u):$(id -g) \
    rclone/rclone \
    listremotes

# perform mount inside Docker container, expose result to host
mkdir -p ~/data/mount
docker run --rm \
    --volume ~/.config/rclone:/config/rclone \
    --volume ~/data:/data:shared \
    --user $(id -u):$(id -g) \
    --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \
    --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \
    rclone/rclone \
    mount dropbox:Photos /data/mount &
ls ~/data/mount

This works fine with v1.57 however if we change it slightly to this then it fails with v1.57 but works with v1.56


# perform mount inside Docker container, expose result to host
mkdir -p ~/data
docker run --rm \
    --volume ~/.config/rclone:/config/rclone \
    --volume ~/data:/data:shared \
    --user $(id -u):$(id -g) \
    --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \
    --device /dev/fuse --cap-add SYS_ADMIN --security-opt apparmor:unconfined \
    rclone/rclone \
    mount dropbox:Photos /data &
ls ~/data

This is is exactly equivalent to the OPs case and seems quite likely people might run into it.

Perhaps we should tighten the /proc/mounts check to refuse only to mount on a fuse filing system, or even maybe just on an rclone fuse filing system?

Rclone mounts look like this

s3:rclone /mnt/tmp fuse.rclone rw,nosuid,nodev,relatime,user_id=1000,group_id=1000 0 0

so perhaps looking for a fuse.rclone might be enough?

I think the error message is perfectly clear - it says the problem and it says which flag to use, so I'm not too worried about that.

martin · November 8, 2021, 4:42pm

Ah, of course, use a sub directory off of the mount. I think that's probably a better work around for me than using the --allow-non-empty flag.

I think checking for rclone fuse mount might be better as I'm not sure my use case is overly esoteric.

ivandeex · November 8, 2021, 5:07pm

Is everyone okay with that?

in the long past rclone failed to mount if it detected any contents in the directory
yesterday it failed if the directory was someone's mountpoint
soon it will accept both cases (and much more) and fail only if it was an rclone mount

If you're ok than I'll submit a patch (and mark it for 1.57.1)
ok?

p.s. it will be a oneliner for me. just want to be sure that it makes you all happy.

martin · November 8, 2021, 8:36pm

Sounds good to me

ncw · November 8, 2021, 10:52pm

Yes I think that works for me

system · November 11, 2021, 10:52pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.