What is the problem you are having with rclone?
Hello all,
I encounter a problem when using the rclone docker volume plugin on linux. As part of a project for the PUNCH4NFDI consortium I am trying to have Docker to create a volume for containers which is linked to a dCache instance. The goal is to have Docker containers to write data to this volume while mounted to the dCache instance so that files are written there directly. Access to the dCache is protected and authentication is provided via an OpenID Connect token.
I managed to successfully configure the rclone.config file to have rclone work with the dCache. If I try to directly mount the dCache instance to a folder on my computer it is successful. Rclone obtains the necessary authentication via the line bearer_token_command = oidc-token <remote_short_name> in the rclone.conf file.
I have installed the rclone docker volume plugin as a managed plugin, the installation was successful (following instructions at this link: Docker Volume Plugin). I have created the /var/lib/docker-plugins/rclone/config and /var/lib/docker-plugins/rclone/cache folders. I have copied the rclone.config file in the config folder.
The next step in the workflow is represented by the creation of a docker volume (see the list of commands I give below). I do this using the rclone plugin as a device. I obtain an error claiming that the oidc-token executable is not in $PATH. This means that the rclone docker plugin is working correctly, it reads the correct configuration file (if I had not copied the rclone.config file in /var/lib/docker-plugins/rclone/config I would have gotten a different error claiming that the remote configuration was not found, so clearly this is not the source of the problem). Also, I am using the Docker engine and not the Docker app (as I know that the Docker app uses its own virtual machine to interact with the host system which prevents some paths to be visible by Docker).
I modified by hand the rclone.config file, substituting the line bearer_token_command = oidc-token <remote_short_name> with bearer_token_command = /usr/bin/oidc-token <remote_short_name> as /usr/bin is the path of the oidc-token executable (this path was contained in $PATH). I have tried several other combination of paths. I have tried copying the oidc-token executable in the /var/lib/docker-plugins/rclone/config and cache folders. I have tried specifying as paths for the oidc-token command as /data/config and /data/cache as these are the supposed mount points I get for the /var/lib/docker-plugins/rclone/config and cache folders when I inspect the rclone docker volume plugin settings. Everyone of these tries returns with the error that the oidc-token executable is not found.
If instead of trying to have rclone access the dCache by obtaining a token with the oidc-token command I manually copy the token in the rclone.config file (substituting the line bearer_token_command = oidc-token <remote_short_name> with bearer_token = <token_value>) everything works properly. Volume gets created and containers can access the dCache normally. Only problem with this is that the token has to be refreshed about every hour and it is not the solution I am looking for.
In short, why is the oidc-token executable not visible from the rclone docker volume plugin and how do I solve this problem?
Thank you in advance for your help and constructive feedback.
Run the command 'rclone version' and share the full output of the command.
#rclone v1.58.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.13.0-40-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.9
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
webdav dCache, instructions for configuring the rclone.config file are here: WebDAV
The command you were trying to run (eg rclone copy /tmp remote:tmp)
docker volume create -d rclone -o remote=<remote_name_and_path> --name testvol2
The rclone config contents with secrets removed.
[<remote_name>]
type = webdav
url = <remote_url>
vendor = other
bearer_token_command = oidc-token <remote_short_name>
A log from the command with the -vv flag
This is the error message obtained with the rclone.config file reported above:
Error response from daemon: create testvol2: VolumeDriver.Create: failed to get bearer token using "oidc-token punch": : exec: "oidc-token": executable file not found in $PATH
And this the error message if I replace the last line of the rclone.config file reported above with bearer_token_command = /usr/bin/oidc-token <remote_short_name>
Error response from daemon: create testvol2: VolumeDriver.Create: failed to get bearer token using "/usr/bin/oidc-token punch": : fork/exec /usr/bin/oidc-token: no such file or directory