I want to use the rclone docker container in userspace mode. So, on Ubuntu 18.04 I have a regular account, say uid 100, which I've added to the docker group, to allow sudo-less docker commands. My rclone config file sits at ~/.rclone.d/rclone.conf and is owned by 100:100. The rclone binary inside the rclone docker image is at /root/rclone, with permissions 0755, but /root has permissions 0700 (both owned by 0:0).
If I run the container with uid 0 (the default), whatever files rclone writes (data files, and also the rclone.conf with refreshed tokens) get uid 0, which is not what I want.
If I attempt to run the container with uid 100 (docker run --user 100 ...), I cannot get to /root/rclone because of the current permissions on /root.
I could build my own image off rclone/rclone:latest which includes a chmod 0755 /root, but I'm wondering if there's a supported way to do what I want (or perhaps the chmod could go in the official image)
What is your rclone version (output from rclone version)
rclone v1.49.2-heads/v1.49.2-beta
os/arch: linux/amd64
go version: go1.13
Which OS you are using and how many bits (eg Windows 7, 64 bit)
Ubuntu 18.04, 64.
Which cloud storage system are you using? (eg Google Drive)
Google drive.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
$ docker run --env RCLONE_CONFIG=/rclone/rclone.conf --volume ~/.rclone.d:/rclone rclone/rclone listremotes
...
gd:
$ docker run --env RCLONE_CONFIG=/rclone/rclone.conf --volume ~/.rclone.d:/rclone rclone/rclone lsd gd:
...
(ok, but now ~/.rclone.d/rclone.conf is owned by 0:100, because the container updated the token)
$ docker run --user 100:100 --env RCLONE_CONFIG=/rclone/rclone.conf --volume ~/.rclone.d:/rclone rclone/rclone listremotes
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "chdir to cwd (\"/root\") set in config.json failed: permission denied": unknown.
$ docker run --user 100:100 --env RCLONE_CONFIG=/rclone/rclone.conf --volume ~/.rclone.d:/rclone --workdir /rclone --entrypoint /root/rclone rclone/rclone listremotes
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/root/rclone\": stat /root/rclone: permission denied": unknown.
A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)
The error is from docker not from rclone, and its cause is quite clear.
My experience with docker is informal, not professional, so I cannot exclude the possibility that somebody out there has some elegant solution to all this, hence the post.
I know I don't know several things:
Why does /root have permissions 0700? Does it come like this from Alpine? Is there some security angle to this (I doubt it).
Why does the rclone binary get placed in /root, as opposed to e.g. /usr/local/bin?
Modulo all of the above, my feelings are that:
It is desirable to allow userspace operation of the image (docker run --user 100 ...) to avoid the issue with files being created with uid 0.
Docker is meant to host the rclone binary, not the user files: neither data nor configs. Those would normally reside on the host disk, and they would be mounted as a volume.
There is something I don't yet understand about how rclone handles the config file. I understand it needs write access to update tokens. What I don't understand is if under usermode docker, it is enough to mount a single host config file, versus a host dir containing the config file.
Concrete suggestions:
Put the rclone binary at /usr/local/bin, make sure it's accessible by non-root users.
Declare the ENTRYPOINT as /usr/local/bin/rclone or just rclone (it should be in the path), but not the relative ./rclone of today (because then customizing WORKDIR requires also customizing ENTRYPOINT).
Create an empty directory /data in the image (not sure if this step is necessary).
Declare /data as the WORKDIR in the image definition.
Explain in the docs that the user of the image is expected to mount the host data folder for the transfers at /data, i.e. docker run [--user UID:GID] --volume /home/joe/data:/data ....
The above suggestions cover the binary and the data, but not the config. I would have to play with the fixed image running in usermode to understand if it is enough to mount a single host config file versus a host config dir containing a config file.
It is traditional on unix systems for root to be 0700 to keep the sysadmins stuff private
That is how the PR came
Agree
I don't think mounting a single config file works due to the way rclone updates its config file
$ docker run --rm -v ~/.rclone.conf:/root/.rclone.conf rclone/rclone config update z test 1
Remote config
2019/09/12 12:29:12 Failed to save config after 10 tries: Failed to move previous config to backup location: rename /root/.rclone.conf /root/.rclone.conf.old: device or resource busy
Rclone keeps its config in $USER/.config/rclone/ normally so that would be the obvious thing to mount.
docker run --rm -v ~/.config/rclone:/root/.config/rclone rclone/rclone config update z test 1
If you are running rclone as root then that is root/.config/rclone/.
Those all look good to me Though I'd probably put the binary in /bin
I wonder if we should also create a /config directory then set XDG_CONFIG_HOME in the image to be /config. This will cause rclone to make its config in /config/rclone and that would then make it easy to mount your own config -v ~/.config/rclone:/config/rclone regardless of the user settings.
I would suggest /usr/local/bin because it's something built locally (inside the image) not coming from standard Alpine packages.
Yes, that's exactly what I was thinking to suggest. I saw the error about 10 tries as well, but that one I didn't fully understand- Why does the move fail exactly? Clearly the container with uid=0 has all the permissions it would ever want. My best guess is it has to do with how Docker mounts single files, so that /root/.rclone.conf is marked as "special" somehow, but I'm not sure about this. If that's the case, yes, I would agree to make a /config, set the relevant environment variable to point there, and tell the user to mount their config dir there.
Yes please - I like the way the conversation is turning out!
There is one more thing I was thinking needs addressing is the fuse package to enable rclone mount in the container. I'm not 100% sure what needs to be added. I don't know if you could access the fuse mount from outside the container either!
I believe the purpose of "AS builder" is to prevent build products from being packaged in the final image.
I created a PR that fixes userspace operation, enables Dockerized mount, plus some documentation. I haven't seen any CI/CD checking the Docker image, is there one? (Perhaps that would be a separate task.)
I would think of the Docker image as yet another product, e.g. Linux, OSX, Windows, and Docker image. I'm not sure which CI/CD checks which product, but I believe none checks the Docker image.
(Since I can't post the link...) Look for "mateidavid zstr" to find an example .travis.yml that builds a Docker image in the before_install step, and uses it for further checks. (I wrote this way back when, probably things changed a bit, but the idea is that Travis can build and use Docker images.)
There are 2 types of test you might want to do when in a Travis build for a Docker image:
Tests of rclone executable inside the Docker image. I understand these are performed during the Docker build.
Tests of the rclone image layout. E.g. at the absolute minimum, one could check that docker run rclone --version correctly prints some valid version. The various rearrangements I did, such as, can you run the image in usermode, also only affect the image itself, not the executable, and one could write tests for that to lock in expected behavior.
I could do another PR on this subject, but I would have to research if e.g. Travis would accept 2 completely different configs (the existing one, plus one that involves Docker image tests).
The CI/CD runs the same go tests that are run when building the docker image, so in that respect they are quite similar.
Rclone has a whole suite of integration tests which are too expensive to run on every CI event - I check those manually before tagging a release.
I don't currently have any tests on the rclone executable itself, my assumption is that the go tests are "good enough". This has been problematic once or twice in the past usually when I break the command line flag parsing, but not often.
Tests for the expected behaviour of the docker image would be useful I think. How would you write this? Given that the rest of the project is written in Go I'd probably use the go testing framework to do that.
The rclone travis config is pretty manual so you could easily add another build type I think. Rclone also has a duplicated CI in azure pipelines as I'm seeing whether I want to switch over...
I'll look at the Dockerhub automated testing framework and Travis, but offline, meaning when and if I get a chance. If I find something useful I'll start a new thread.
.