Using rclonefs to mount a folder with --rc activated

Marc_Lamarine · September 5, 2023, 2:14pm

STOP and READ USE THIS TEMPLATE NO EXCEPTIONS - By not using this, you waste your time, our time and really hate puppies. Please remove these two lines and that will confirm you have read them.

What is the problem you are having with rclone?

I have a different behaviour when I use rclonefs to mount with or without the --rc option. When rc mode is activated, the mounting process failed. rclonefs and rclone is the same. I reduced the problem to this but ultimately I goes in automount and fuse.

ll /usr/bin/rclonefs 
lrwxrwxrwx 1 root root 21 Mar  9 11:32 /usr/bin/rclonefs -> /usr/local/bin/rclone*

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

os/version: ubuntu 20.04 (64 bit)
os/kernel: 5.15.0-79-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.19.4
go/linking: static
go/tags: none

Which cloud storage system are you using? (eg Google Drive)

My ultimate goal is to use any storage. I have tried with S3 but the reproducible example is using a local mount.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

First I create a user config file to mount the local /tmp to mount point /mnt/toto

# The mount works as expected
 rclone --config=myrclone.conf config create mytmp local
$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
ll /mnt/toto/
total 679
drwxr-xr-x 1 docker host_user_group      0 Sep  5 15:43 ./
drwxr-xr-x 1 root   root              4096 Sep  5 11:21 ../
drwxr-xr-x 1 docker host_user_group      0 Sep  1 09:03 .X11-unix/
drwxr-xr-x 1 docker host_user_group      0 Oct 31  2022 RtmpIP04VL/
.....

$ ps -eaf 
UID          PID    PPID  C STIME TTY          TIME CMD
docker         1       0  0 10:33 pts/0    00:00:01 bash
docker      1230       0  0 15:37 pts/1    00:00:00 bash
root        1378       1  0 15:43 pts/0    00:00:00 /usr/local/rclone/rclone mount mytmp:/tmp /mnt/toto --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000 --daemon

Then I try to do the same with the --rc option ( I have killed the previous mounting process) it fails

$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o rw,rc,rc-no-auth,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 15:51:34 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/09/05 15:52:34 Fatal error: mount not ready

# Waiting for the error I had this processus listing
2023/09/05 15:51:34 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/09/05 15:52:34 Fatal error: mount not ready

$ ps -eaf
UID          PID    PPID  C STIME TTY          TIME CMD
docker         1       0  0 10:33 pts/0    00:00:01 bash
docker      1230       0  0 15:37 pts/1    00:00:00 bash
root        1614       1  0 15:49 pts/0    00:00:00 sudo -E rclonefs mytmp:/tmp /mnt/toto -o rw,rc,rc-no-auth,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
root        1615    1614 18 15:49 pts/0    00:00:03 rclonefs mytmp:/tmp /mnt/toto -o rw,rc,rc-no-auth,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
root        1628    1615  0 15:49 pts/0    00:00:00 [rclone] <defunct>
docker      1647    1230  0 15:50 pts/1    00:00:00 ps -eaf

The rclone config contents with secrets removed.

[mytmp]
type = local

A log from the command that you were trying to run with the `-vv` flag

$ sudo -E rclonefs -vv mytmp:/tmp /mnt/toto -o rw,rc,rc-no-auth,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 16:11:39 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 16:11:39 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 16:11:39 DEBUG : Setting --config "/home/docker/.config/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/home/docker/.config/rclone/rclone.conf"
2023/09/05 16:11:39 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclonefs" "mount" "mytmp:/tmp" "/mnt/toto" "--rc" "--rc-no-auth" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--verbose=2" "--daemon"]
2023/09/05 16:11:39 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/09/05 16:11:39 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 16:11:39 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 16:12:39 DEBUG : Daemon timed out. Terminating daemon pid 1755
2023/09/05 16:12:39 Fatal error: mount not ready

kapitainsky · September 5, 2023, 3:37pm

First update your rclone version - it is now v1.63.1 - and see if problem persist.

No point to investigate old releases.

Marc_Lamarine · September 5, 2023, 5:24pm

Hello,

Sorry about that, I checked on the one we use.
I upgraded the version

$ rclone --version
rclone v1.63.1
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.15.0-79-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.6
- go/linking: static
- go/tags: none

Now even the non --rc command does not work anymore.

$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 19:20:48 Fatal error: mount not ready

Same defunct

$ ps -eaf 
UID          PID    PPID  C STIME TTY          TIME CMD
docker         1       0  0 19:05 pts/0    00:00:01 bash
docker       519       0  0 19:19 pts/1    00:00:00 bash
root         528       1  0 19:19 pts/0    00:00:00 sudo -E rclonefs mytmp:/tmp /mnt/toto -o rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
root         529     528 14 19:19 pts/0    00:00:00 rclonefs mytmp:/tmp /mnt/toto -o rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
root         541     529  1 19:19 pts/0    00:00:00 [rclone] <defunct>
docker       552     519  0 19:19 pts/1    00:00:00 ps -eaf

kapitainsky · September 5, 2023, 5:39pm

could you enable DEBUG level logging and capture all log?

Marc_Lamarine · September 5, 2023, 6:40pm

Hello

20:37:48-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o  log-level=DEBUG,rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 20:38:01 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclonefs" "mount" "mytmp:/tmp" "/mnt/toto" "--log-level=DEBUG" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--daemon"]
2023/09/05 20:38:01 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 20:38:01 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 20:39:01 DEBUG : Daemon timed out. Terminating daemon pid 615
2023/09/05 20:39:01 Fatal error: mount not ready

/mnt/toto exists

20:39:01-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ more myrclone.conf 
[mytmp]
type = local
20:38:50-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ ll /mnt/toto/
total 8
drwxr-xr-x 2 root root 4096 Sep  5 19:16 ./
drwxr-xr-x 1 root root 4096 Sep  5 19:16 ../

kapitainsky · September 5, 2023, 6:44pm

Because you run it with --daemon flag we can not see full output.

Could you run it without it?

or redirect full log output to log file --log-file /path/to/rclone.log

Marc_Lamarine · September 5, 2023, 7:25pm

I am not a super specialist but I thought the whole point was to run it as a daemon
The rclonefs "automatically" adds the daemon mode.
If I were to run it in non daemon mode, my naive understanding is that then I should rewrite the command line using rclone directly and not rclonefs ?

My last try is that one, that fails. I will try with a rclone command call.

21:23:37-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ sudo -E rclonefs --log-file=tmp/rclone.conf --log-level=DEBUG mytmp:/tmp /mnt/toto -o rw,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 21:24:02 Failed to parse command line: flag "--log-file=tmp/rclone.conf" is not supported in mount mode

Marc_Lamarine · September 5, 2023, 7:34pm

Hello

I did some progress converting the call to an rclone one
There is apparently a problem with fusermount
as it looks for a fusermount3 and I have only 2.99

21:28:10-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ rclone mount mytmp:/tmp /mnt/toto --log-level=DEBUG --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000
2023/09/05 21:28:44 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --allow-other "true" from environment variable RCLONE_ALLOW_OTHER="true"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --dir-cache-time "5m0s" from environment variable RCLONE_DIR_CACHE_TIME="300s"
2023/09/05 21:28:44 DEBUG : Setting --vfs-cache-mode "full" from environment variable RCLONE_VFS_CACHE_MODE="full"
2023/09/05 21:28:44 DEBUG : Setting --config "/home/docker/.config/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/home/docker/.config/rclone/rclone.conf"
2023/09/05 21:28:44 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "mount" "mytmp:/tmp" "/mnt/toto" "--log-level=DEBUG" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000"]
2023/09/05 21:28:44 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 21:28:44 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 21:28:44 INFO  : Local file system at /tmp: poll-interval is not supported by this remote
2023/09/05 21:28:44 DEBUG : Local file system at /tmp: Mounting on "/mnt/toto"
2023/09/05 21:28:44 Fatal error: failed to mount FUSE fs: fusermount: exec: "fusermount3": executable file not found in $PATH

21:28:44-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ echo $PATH
/home/docker/.local/bin:/usr/local/bin:/usr/bin:/bin:/sbin:/usr/sbin
21:28:54-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ type fusermount 
fusermount is /usr/bin/fusermount
21:28:58-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ /usr/bin/fusermount --version
fusermount version: 2.9.9

I had a try by doing a symbolic link on my current fusermount version
and it worked that time

sudo ln -s /usr/bin/fusermount /usr/bin/fusermount3
sudo rclone mount mytmp:/tmp /mnt/toto --log-level=DEBUG --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000
2023/09/05 21:33:19 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclone" "mount" "mytmp:/tmp" "/mnt/toto" "--log-level=DEBUG" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000"]
2023/09/05 21:33:19 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 21:33:19 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 21:33:19 INFO  : Local file system at /tmp: poll-interval is not supported by this remote
2023/09/05 21:33:19 DEBUG : Local file system at /tmp: Mounting on "/mnt/toto"
2023/09/05 21:33:19 DEBUG : : Root: 
2023/09/05 21:33:19 DEBUG : : >Root: node=/, err=<nil>
2023/09/05 21:33:53 DEBUG : /: Attr: 
2023/09/05 21:33:53 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2023/09/05 21:33:53 DEBUG : /: ReadDirAll: 
2023/09/05 21:33:53 DEBUG : /: >ReadDirAll: item=25, err=<nil>
2023/09/05 21:33:53 DEBUG : /: Lookup: name="RtmpIP04VL"
2023/09/05 21:33:53 DEBUG : /: >Lookup: node=RtmpIP04VL/, err=<nil>

sudo ls -l /mnt/toto 
total 674
drwxr-xr-x 1 docker host_user_group      0 Oct 31  2022 RtmpIP04VL
drwxr-xr-x 1 docker host_user_group      0 Aug 29 17:41 downloaded_packages
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:39 file6213c097f
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:40 file678bd0f91
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:40 file722fbd91
drwxr-xr-x 1 docker host_user_group      0 Mar  9 10:49 file72b5cacd7
-rw-r--r-- 1 docker host_user_group   2800 Mar  9 11:40 foo.o

kapitainsky · September 5, 2023, 7:49pm

Yeah - as I remember Ubuntu 20.04 did not use fuse3 by default - it can though. But I am not sure if there are no other dependencies - so if link method works it should be enough.

Marc_Lamarine · September 5, 2023, 7:49pm

EUREKA
These two commands worked

rclone mount mytmp:/tmp /mnt/toto --rc --log-level=DEBUG  --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000
rclone mount mytmp:/tmp /mnt/toto  --log-level=DEBUG  --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000

I had to change /etc/fuse.conf - to add user_allow_other and modify permission on the mountpoint /mnt/toto

# /etc/fuse.conf - Configuration file for Filesystem in Userspace (FUSE)

# Set the maximum number of FUSE mounts allowed to non-root users.
# The default is 1000.
#mount_max = 1000

# Allow non-root users to specify the allow_other or allow_root mount options.
user_allow_other

asdffdsa · September 5, 2023, 7:54pm

hi, check out this post

Marc_Lamarine · September 5, 2023, 8:17pm

Hello @asdffdsa , thanks for the tip. I had not seen it, will have a look.

Now I have rclonefs that works in non rc mode i.e this is OK and does the job

sudo -E rclonefs mytmp:/tmp /mnt/toto -o  rw,log-level=DEBUG,log-file=tmp/rclone.log,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid

but adding the --rc mode still fails.
I double checked that no other rclone process was there ?

22:12:19-docker@$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o  rw,rc,rc-no-auth,log-level=DEBUG,log-file=tmp/rclone.log,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
22:13:30-docker@7HA-PF0WPK8G:/home/mlamarin/workspace/pipelines/us/projects_data_syncer$ sudo cat tmp/rclone.log
2023/09/05 22:12:30 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclonefs" "mount" "mytmp:/tmp" "/mnt/toto" "--rc" "--rc-no-auth" "--log-level=DEBUG" "--log-file=tmp/rclone.log" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--daemon"]
2023/09/05 22:12:30 NOTICE: Serving remote control on http://127.0.0.1:5572/
2023/09/05 22:12:30 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 22:12:30 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 22:12:30 DEBUG : rclone: Version "v1.63.1" starting with parameters ["/usr/bin/rclone" "mount" "mytmp:/tmp" "/mnt/toto" "--rc" "--rc-no-auth" "--log-level=DEBUG" "--log-file=tmp/rclone.log" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--daemon"]
2023/09/05 22:12:30 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5572: bind: address already in use
2023/09/05 22:13:30 DEBUG : Daemon timed out. Terminating daemon pid 1620
2023/09/05 22:13:30 Fatal error: mount not ready

22:16:35-docker@sudo netstat -lnp | grep 5572
22:16:37-docker@$ ps -eaf 
UID          PID    PPID  C STIME TTY          TIME CMD
docker         1       0  0 19:05 pts/0    00:00:02 bash
docker      1145       1  0 21:48 pts/0    00:00:00 nano /etc/fuse.conf
docker      1445       0  0 22:01 pts/1    00:00:00 bash
docker      1683       1  0 22:16 pts/0    00:00:00 ps -eaf

asdffdsa · September 5, 2023, 8:19pm

well, according to rclone, that ip:port is in use, and 5572 is the default port for rclone rc.
if you cannot figure it out, might use another port

ps -aux | grep rclone
root        3098  0.0  2.5 829236 50384 pts/1    Sl+  Sep04   0:11 rclone mount ...

Marc_Lamarine · September 5, 2023, 8:31pm

Hello
It does not seem to be that, ps -aux gave nothing I had also checked the port bindings (see the nestast -nlp)
Using another port 5573 produced the same behaviour

$ sudo -E rclonefs mytmp:/tmp /mnt/toto -o  rw,rc,rc-no-auth,rc-addr=localhost:5573,log-level=DEBUG,log-file=tmp/rclone.log,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid
2023/09/05 22:25:19 DEBUG : rclone: Version "v1.63.1" starting with parameters ["rclonefs" "mount" "mytmp:/tmp" "/mnt/toto" "--rc" "--rc-no-auth" "--rc-addr=localhost:5573" "--log-level=DEBUG" "--log-file=tmp/rclone.log" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--daemon"]
2023/09/05 22:25:19 NOTICE: Serving remote control on http://127.0.0.1:5573/
2023/09/05 22:25:19 DEBUG : Creating backend with remote "mytmp:/tmp"
2023/09/05 22:25:19 DEBUG : Using config file from "/home/mlamarin/workspace/pipelines/us/projects_data_syncer/myrclone.conf"
2023/09/05 22:25:20 DEBUG : rclone: Version "v1.63.1" starting with parameters ["/usr/bin/rclone" "mount" "mytmp:/tmp" "/mnt/toto" "--rc" "--rc-no-auth" "--rc-addr=localhost:5573" "--log-level=DEBUG" "--log-file=tmp/rclone.log" "--config=myrclone.conf" "--vfs-cache-mode=off" "--uid=1000" "--gid=1000" "--daemon"]
2023/09/05 22:25:20 Failed to start remote control: failed to init server: listen tcp 127.0.0.1:5573: bind: address already in use

Thanks to all for your help, I will come back on that tomorrow. All the best

asdffdsa · September 5, 2023, 8:35pm

not sure if these are relevant?

Option --rc does not work together with --daemon or in mount.rclone mode
rc-server - if enabled, it will bind to port in parent and cause "cannot bind" in daemonized child

Marc_Lamarine · September 6, 2023, 8:33am

Dear @kapitainsky and @asdffdsa

From the last pointers, I ended up on

github.com/rclone/rclone

mount: improve implementation of --daemon mode

opened 12:01PM - 05 Oct 21 UTC

ivandeex

VFS / mount thinking OS: Unix terminal refactoring

### References This ticket: - supersedes (?) #2618 - draws ideas from https…://github.com/rclone/rclone/issues/2968#issuecomment-716286668 - and discussion at https://github.com/rclone/rclone/issues/2968#issuecomment-716679470 till https://github.com/rclone/rclone/issues/2968#issuecomment-728309447 - is related to #4764 Also related to a number of [console password input](https://github.com/rclone/rclone/labels/terminal) tickets - see `Password prompts` in a comment below. ### What problem are you are trying to solve? Looking at the rclone initialization code from the --daemon point of view I can see stuff that should run in the child only and not in parent: - [rc-server](https://github.com/rclone/rclone/blob/v1.52.3/cmd/cmd.go#L379) - if enabled, it will bind to port in parent and cause "cannot bind" in daemonized child - [NewFsDir](https://github.com/rclone/rclone/blob/v1.52.3/cmd/mountlib/mount.go#L305) - it chains to backend-specific, unconstrained, generally fork-unsafe [Fs.NewFs](https://github.com/rclone/rclone/blob/v1.52.3/fs/cache/cache.go#L90) This calls for refactoring, esp. [initConfig](https://github.com/rclone/rclone/blob/v1.52.3/cmd/cmd.go#L362) can be split into a function common to all commands to be used in `cobra.OnInitialize` and into another function to contain child-only steps. Most commands would inherit it from `rootCmd.PersistentPreRun`. Daemonizing commands would disable it by `PersistentPreRun: func(){return nil}` and then call from child... just an idea ### How do you think rclone should be changed to solve that? We might refactor backgrounding to have rughly 3 stages : - _parent pre-init_ will run a hand-picked subset of fork-safe operations from current `cmd.initConfig`, before callsing`os.StartProcess` - _child setup_ will take place in the child performing other unsafe operations from `initConfig` e.g. start rc-server, calls `fs.NewFs`, set up _vfs_, start fuse mount and return its status to parent. This stage closely mimics current non-daemonizing flow. At this stage child has fully working stdin/stdout/stderr (either parent redirects child streams through iself or perhaps the child just sits attached to the parent's terminal session) and can log debug/error messages to stderr or request user input from stdin. Parent just pipes child's streams and waits for status or timeout. - _child detach_ - parent receives daemonization/mount status from child (or detects timeout), logs it and exits, while child: - finishes sending status to parent as a specially formatted line on stdout/stderr or via dedicated _pipe_, - replaces stdin/stdout/stderr by _/dev/null_, - stops output from stats and pacers, - closes extra files and pipes, - calls _setsid()_, - probably does _setpgrp_ (detaches from process group) to ensure it will be later reaped by _pid 1_ when it exits or aborts, - reopens log file descriptor and syslog connection if needed (because if any file descriptors remain from parent, the child later ends up as a zombie), - probably detaches itself from original _cgroup_ (if it helps with systemd), - proceeds with business as usual! As a consequence, we have to: - triage `initConfig` contents (it's rather small) between stages 1 and 2; - think of switching child logger between file and redirected stdout(err); - rethink config key; - deactivate systemd log format in child; - parent should issue warnings in advance if child's log file or stdin helper isn't configured. - refactor all ReadLine, ReadPassword, fmt.Scanf and alikes in the core including _fs/config_ and in backends (where they usually read _2fa_ secrets) into a common wrapper that can: - read terminal when available (and optionally hide user input if it's secret), or - read stdin (prompt on stdout) piped from parent at stage 2, or - use universally configured helper executables(files) when detached, or - log errors when not configured; Another idea is to keep the `StartDaemon` where it was already butuse a sort of sync mechanism between the parent process and the child process (pipe?). ### How to use GitHub * Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue. * Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue. * Subscribe to receive notifications on status change and new comments.

Where @ncw suggested to add

I had a try with --daemon-wait=0

sudo -E rclonefs mytmp:/tmp /mnt/toto -o  log-level=DEBUG,rw,rc,rc-no-auth,daemon-wait=0,config=myrclone.conf,vfs-cache-mode=off,uid=1000,gid=1000,dev,suid

and it works

10:25:08-docker@7HA-PF0WPK8G:$ ps -eaf 
UID          PID    PPID  C STIME TTY          TIME CMD
docker         1       0  0 Sep05 pts/0    00:00:02 bash
docker      1145       1  0 Sep05 pts/0    00:00:00 nano /etc/fuse.conf
docker      1445       0  0 Sep05 pts/1    00:00:00 bash
root        1939       1  0 10:24 pts/1    00:00:00 /usr/bin/rclone mount mytmp:/tmp /mnt/toto --log-level=DEBUG --rc --rc-no-auth --daemon-wait=0 --config=myrclone.conf --vfs-cache-mode=off --uid=1000 --gid=1000 --daemon
docker      1966       1  0 10:26 pts/0    00:00:00 ps -eaf
10:26:57-docker@7HA-PF0WPK8G:$ mount | grep mytmp
mytmp:/tmp on /mnt/toto type fuse.rclone (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other)
10:27:03-docker@7HA-PF0WPK8G:$ ll /mnt/toto/ | head
total 678
drwxr-xr-x 1 docker host_user_group      0 Sep  6 10:24 ./
drwxr-xr-x 1 root   root              4096 Sep  5 19:16 ../
drwxr-xr-x 1 docker host_user_group      0 Sep  1 09:03 .X11-unix/
drwxr-xr-x 1 docker host_user_group      0 Oct 31  2022 RtmpIP04VL/
drwxr-xr-x 1 docker host_user_group      0 Aug 29 17:41 downloaded_packages/
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:39 file6213c097f/
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:40 file678bd0f91/
drwxr-xr-x 1 docker host_user_group      0 Mar  9 11:40 file722fbd91/
drwxr-xr-x 1 docker host_user_group      0 Mar  9 10:49 file72b5cacd7/

10:29:19-docker@7HA-PF0WPK8G:$ rclone rc vfs/list
{
	"vfses": [
		"mytmp:/tmp"
	]
}

So think it's fine and that we can close the issue.
Thanks to everyone for the help on this and thanks for bringing rclone to the world.

ncw · September 6, 2023, 11:48am

Do you have lots of files in your VFS cache so it takes the mount a long time to start up? I think that is probably the root cause of this issue. Needless to say there is an issue about that! Reduce startup time with vfs cache (writes/full) · Issue #4595 · rclone/rclone · GitHub

Marc_Lamarine · September 6, 2023, 2:39pm

Hello @ncw

Thanks for the tip,
Your are right I have a lot of files in the real case scenario I am working for.
But it happens also in the example with a mount on a /tmp/ file with les than 20 megas and 82 files.
I will any way investigate this metadata key=value. I was not aware of it.
Thanks

system · October 6, 2023, 2:40pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.