Rclone works fine local, fails in docker with same config

What is the problem you are having with rclone?

I am attempting to get rclone working against gdrive in a docker container that will expose the gdrive to the local file system. The use case is for a plex library.

I am trying to resolve an issue where my system locks occasionally and needs to be hard reset by putting both plex and my rclone implementation into containers.

I can run rclone locally with no issues with the following command: rclone Video: /gdrive

The docker commands do not throw any errors, however I when I try to ls either /gdrive or do a docker exec and try to ls /data I get nothing.

I am using the same exact config locally as I do in the container.

What is your rclone version (output from rclone version)

XXXXXX@XXXXXXXXX:/# rclone version
rclone v1.51.0

  • os/arch: linux/amd64
  • go version: go1.13.7

I am also using the official docker container

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 16.04.6 LTS
4.4.0-137-generic #163-Ubuntu SMP Mon Sep 24 13:14:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Docker:
XXXXXXXXXX@XXXXXXXXXX:/# docker version
Client:
Version: 17.05.0-ce
API version: 1.29
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64

Server:
Version: 17.05.0-ce
API version: 1.29 (minimum version 1.12)
Go version: go1.7.5
Git commit: 89658be
Built: Thu May 4 22:10:54 2017
OS/Arch: linux/amd64
Experimental: false

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I have tried several permutations with and without -privileged

docker run --rm \
--name=gdrive \
--volume /etc/gdrive:/config/rclone \
--volume /gdrive:/data:shared \
--user $(id -u):$(id -g) \
--volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro \
--cap-add MKNOD --cap-add SYS_ADMIN --device /dev/fuse --security-opt apparmor=unconfined \
--privileged \
rclone/rclone \
mount Video: /data



docker run --name gdrive \
    --cap-add SYS_ADMIN \
    --device /dev/fuse \
    --security-opt apparmor=unconfined \
    -v /etc/gdrive:/config/rclone \
    -v /gdrive:/data:shared \
    rclone/rclone \
    mount Video: /data

Here is the config I am using:

[drive]
type = drive
client_id = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
client_secret = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
scope = drive
root_folder_id = xxxxxxxxxxxxxxxxxxxxxxxx
service_account_file =
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","token_type":"Bearer","refresh_token":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx","expiry":"2020-05-02T01:20:47.011206178Z"}

[Video]
type = cache
remote = drive:/Video/
plex_url = xxxxxxxxxxxxx:32400
plex_username =
plex_password =
plex_token =

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2020/05/02 00:43:14 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "mount" "Video:" "/data"]
2020/05/02 00:43:14 DEBUG : Using config file from "/config/rclone/rclone.conf"
2020/05/02 00:43:14 DEBUG : Video: wrapped drive:Video at root
2020/05/02 00:43:14 INFO  : Video: Cache DB path: /root/.cache/rclone/cache-backend/Video.db
2020/05/02 00:43:14 INFO  : Video: Cache chunk path: /root/.cache/rclone/cache-backend/Video
2020/05/02 00:43:14 INFO  : Video: Chunk Memory: true
2020/05/02 00:43:14 INFO  : Video: Chunk Size: 5M
2020/05/02 00:43:14 INFO  : Video: Chunk Total Size: 10G
2020/05/02 00:43:14 INFO  : Video: Chunk Clean Interval: 1m0s
2020/05/02 00:43:14 INFO  : Video: Workers: 4
2020/05/02 00:43:14 INFO  : Video: File Age: 6h0m0s
2020/05/02 00:43:14 DEBUG : Adding path "cache/expire" to remote control registry
2020/05/02 00:43:14 DEBUG : Adding path "cache/stats" to remote control registry
2020/05/02 00:43:14 DEBUG : Adding path "cache/fetch" to remote control registry
2020/05/02 00:43:14 DEBUG : Cache remote Video:: Mounting on "/data"
2020/05/02 00:43:14 DEBUG : Cache remote Video:: subscribing to ChangeNotify
2020/05/02 00:43:14 DEBUG : Adding path "vfs/forget" to remote control registry
2020/05/02 00:43:14 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/05/02 00:43:14 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2020/05/02 00:43:14 DEBUG : : Root:
2020/05/02 00:43:14 DEBUG : : >Root: node=/, err=<nil>
2020/05/02 00:43:40 DEBUG : /: Attr:
2020/05/02 00:43:40 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/05/02 00:43:40 DEBUG : /: ReadDirAll:
2020/05/02 00:43:40 DEBUG : Cache remote Video:: list ''
2020/05/02 00:43:40 DEBUG : : list: error: missing cached dir:
2020/05/02 00:43:40 DEBUG : /: >ReadDirAll: item=0, err=<nil>
2020/05/02 00:43:47 DEBUG : /: Attr:
2020/05/02 00:43:47 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/05/02 00:43:47 DEBUG : /: ReadDirAll:
2020/05/02 00:43:47 DEBUG : /: >ReadDirAll: item=0, err=<nil>
2020/05/02 00:44:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:44:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:45:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:45:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:46:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:46:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:47:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:47:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:48:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:48:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:49:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:49:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:50:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:50:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:51:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:51:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:52:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:52:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:53:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:53:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:54:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:54:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:55:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:55:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:56:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:56:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:57:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:57:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:58:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:58:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 00:59:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 00:59:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:00:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:00:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:01:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:01:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:02:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:02:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:03:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:03:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:04:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:04:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:05:14 DEBUG : Cache remote Video:: starting cleanup
2020/05/02 01:05:14 DEBUG : Google drive root 'Video': Checking for changes on remote
2020/05/02 01:05:36 DEBUG : /: Attr:
2020/05/02 01:05:36 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/05/02 01:05:36 DEBUG : /: ReadDirAll:
2020/05/02 01:05:36 DEBUG : : Re-reading directory (21m55.448533132s old)
2020/05/02 01:05:36 DEBUG : Cache remote Video:: list ''
2020/05/02 01:05:36 DEBUG : : list: error: missing cached dir:
2020/05/02 01:05:36 DEBUG : /: >ReadDirAll: item=0, err=<nil>
2020/05/02 01:06:14 DEBUG : Cache remote Video:: starting cleanup

What happens if you ssh in or docker exec "rclone lsf Video:" in the container.

The result of that would tell us whether it is mounting problems or config problems.

I was being stupid before.

Here is a better output. So it looks like there might be an issue with the cached dir, but my attempts to google "list: error: missing cached dir:" have not yielded any useful results.

XXXXXXXXXXXXXXXXX@XXXXXXXXXXXXXXX:/etc/gdrive# docker run --rm --name=gdrive --volume /etc/gdrive:/config/rclone --volume /gdrive:/data:shared --user $(id -u):$(id -g) --volume /etc/passwd:/etc/passwd:ro --volume /etc/group:/etc/group:ro --cap-add MKNOD --cap-add SYS_ADMIN --device /dev/fuse --security-opt apparmor=unconfined --privileged rclone/rclone -vv lsf Video:
2020/05/02 13:59:16 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "-vv" "lsf" "Video:"]
2020/05/02 13:59:16 DEBUG : Using config file from "/config/rclone/rclone.conf"
2020/05/02 13:59:16 DEBUG : Video: wrapped drive:Video at root
2020/05/02 13:59:16 INFO  : Video: Cache DB path: /root/.cache/rclone/cache-backend/Video.db
2020/05/02 13:59:16 INFO  : Video: Cache chunk path: /root/.cache/rclone/cache-backend/Video
2020/05/02 13:59:16 INFO  : Video: Chunk Memory: true
2020/05/02 13:59:16 INFO  : Video: Chunk Size: 5M
2020/05/02 13:59:16 INFO  : Video: Chunk Total Size: 10G
2020/05/02 13:59:16 INFO  : Video: Chunk Clean Interval: 1m0s
2020/05/02 13:59:16 INFO  : Video: Workers: 4
2020/05/02 13:59:16 INFO  : Video: File Age: 6h0m0s
2020/05/02 13:59:16 DEBUG : Adding path "cache/expire" to remote control registry
2020/05/02 13:59:16 DEBUG : Adding path "cache/stats" to remote control registry
2020/05/02 13:59:16 DEBUG : Adding path "cache/fetch" to remote control registry
2020/05/02 13:59:16 DEBUG : Cache remote Video:: list ''
2020/05/02 13:59:16 DEBUG : : list: error: missing cached dir:
2020/05/02 13:59:16 ERROR : : error listing: directory not found
2020/05/02 13:59:16 Failed to lsf with 2 errors: last error was: error in ListJSON: directory not found
2020/05/02 13:59:16 INFO  : plex: stopped Plex watcher
2020/05/02 13:59:16 DEBUG : Cache remote Video:: Services stopped

A friend figured it out.

The root id is going to be different in the container. Since I cloned the config from my local install into my container, it was trying to use the wrong root id.

I blanked it in the container rclone conf and it was autopopulated the next time I ran the container.

Weird.

Problem solved though.

Great - glad you solved the problem. Yes the root_id caching is a little annoying - I need to figure out a way of improving that.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.