Issue setting up rclone in systemd

rtstarliper · November 29, 2024, 12:24am

What is the problem you are having with rclone?

I am attempting to have rclone automatically map an Azure blob storage location at boot using a systemd service (config below). It fails to run, and give an error about not finding the cache file:



#### Run the command 'rclone version' and share the full output of the command.
root@wmssfs00:~# rclone version
rclone v1.68.2
- os/version: debian 11.11 (64 bit)
- os/kernel: 5.10.0-33-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.3
- go/linking: static
- go/tags: none
root@wmssfs00:~#

Current version is showing as 1.68.2 at https://rclone.org/download

####  Which cloud storage system are you using? (eg Google Drive)

Microsoft Azure Blob Storage

#### The command you were trying to run (eg `rclone copy /tmp remote:tmp`)  
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

systemctl start rclone

The rclone.service file (in /etc/systemd/system/ contains the following:

/etc/systemd/system/rclone.service

[Unit]
Description=Auto Mount MASD Azure Blob
Wants=network-online.target
After=network-online.target

[Service]
Type=idle
ExecStart=/usr/bin/rclone mount
--config /etc/default/rclone/rclone.conf
--log-file /var/log/rclone.log
--log-level DEBUG
--allow-non-empty
--allow-other
--cache-dir /var/cache/rclone
MASD_Azure:/ /var/backups/rclone
--daemon

ExecStop=/usr/bin/fusermount -u /var/backups/rclone

[Install]
WantedBy=multi-user.target

#### Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.
<!--  You should use 3 backticks to begin and end your paste to make it readable.   -->

[MASD_Azure]
type = azureblob
account = XXX
key = XXX

Double check the config for sensitive info before posting publicly


#### A log from the command that you were trying to run with the `-vv` flag  
<!-- You should use 3 backticks to begin and end your paste to make it readable.  Or use a service such as https://pastebin.com or https://gist.github.com/   -->

From the rclone.log when attempting to start rclone.service:

2024/11/28 16:41:13 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone" "--daemon"]
2024/11/28 16:41:13 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/28 16:41:13 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/28 16:41:13 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/28 16:41:13 DEBUG : rclone: Version "v1.68.2" finishing with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone" "--daemon"]

The rclone.service shows an error when checked, and the location is not mounted:

root@wmssfs00:/var/log# systemctl status rclone
● rclone.service - Auto Mount MASD Azure Blob
     Loaded: loaded (/etc/systemd/system/rclone.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

Nov 28 13:28:21 wmssfs00 systemd[1]: rclone.service: Control process exited, code=exited, status=1/FAILURE
Nov 28 13:28:21 wmssfs00 systemd[1]: rclone.service: Failed with result 'exit-code'.
Nov 28 13:37:05 wmssfs00 systemd[1]: Started Auto Mount MASD Azure Blob.
Nov 28 13:37:05 wmssfs00 rclone[1566132]: 2024/11/28 13:37:05 DEBUG : Failed to find user cache dir, using temporary directory: neither $XDG_CACHE_>
Nov 28 13:37:05 wmssfs00 fusermount[1566153]: /usr/bin/fusermount: failed to unmount /var/backups/rclone: Device or resource busy
Nov 28 13:37:05 wmssfs00 systemd[1]: rclone.service: Control process exited, code=exited, status=1/FAILURE
Nov 28 13:37:05 wmssfs00 systemd[1]: rclone.service: Failed with result 'exit-code'.
Nov 28 16:41:13 wmssfs00 systemd[1]: Started Auto Mount MASD Azure Blob.
Nov 28 16:41:13 wmssfs00 rclone[1624009]: 2024/11/28 16:41:13 DEBUG : Failed to find user cache dir, using temporary directory: neither $XDG_CACHE_>
Nov 28 16:41:13 wmssfs00 systemd[1]: rclone.service: Succeeded.
root@wmssfs00:/var/log#

Note that I am specifying /var/cache/rclone as the cache location. This location does exist:

root@wmssfs00:/var/cache# ls | grep rclone
drwxr-xr-x 2 root root 4096 Nov 28 12:55 rclone
root@wmssfs00:/var/cache# cd rclone
root@wmssfs00:/var/cache/rclone# pwd
/var/cache/rclone
root@wmssfs00:/var/cache/rclone#

If I run the command manually:

root@wmssfs00:/etc/systemd/system# /usr/bin/rclone mount --config /etc/default/rclone/rclone.conf --log-file /var/log/rclone.log --log-level DEBUG --allow-non-empty --allow-other --cache-dir /var/cache/rclone MASD_Azure:/ /var/backups/rclone --daemon -vv
root@wmssfs00:/etc/systemd/system#

This is the result in the log file (it errored with the -vv so I dropped it)

2024/11/28 19:15:43 CRITICAL: Can't set -v and --log-level
2024/11/28 19:15:59 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone" "--daemon"]
2024/11/28 19:15:59 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/28 19:15:59 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/28 19:15:59 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/28 19:15:59 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone" "--daemon"]
2024/11/28 19:15:59 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/28 19:15:59 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/28 19:15:59 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/28 19:15:59 INFO  : Azure root: poll-interval is not supported by this remote
2024/11/28 19:15:59 DEBUG : Azure root: Mounting on "/var/backups/rclone"
2024/11/28 19:15:59 DEBUG : : Root:
2024/11/28 19:15:59 DEBUG : : >Root: node=/, err=<nil>
2024/11/28 19:15:59 DEBUG : rclone: Version "v1.68.2" finishing with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone" "--daemon"]

It is running and the blob is mounted:
root@wmssfs00:/var/log# ps aux | grep rclone
root 1672297 0.0 0.0 1289392 52268 pts/1 Sl 19:15 0:00 /usr/bin/rclone mount --config /etc/default/rclone/rclone.conf --log-file /var/log/rclone.log --log-level DEBUG --allow-non-empty --allow-other --cache-dir /var/cache/rclone MASD_Azure:/ /var/backups/rclone --daemon
root 1672943 0.0 0.0 6240 648 pts/1 S+ 19:17 0:00 grep -i --color=auto rclone
root@wmssfs00:/var/log# rclone ls /var/backups/rclone
0 data/testing.txt
73 data/wmssfs00/README
964 data/wmssfs00/config
52 data/wmssfs00/hints.1
41258 data/wmssfs00/index.1
190 data/wmssfs00/integrity.1
16 data/wmssfs00/nonce
530 data/wmssfs00/data/0/0
17 data/wmssfs00/data/0/1
root@wmssfs00:/var/log#

I have searched multiple times, this matches some of the systemd configurations I found to run rclone as a service (other configurations are to configure it as a mount for automount to grab, not running as a service).

I have tried multiple iterations to get this to function, including putting the entire command on a single line in the rclone.service file, with no change.  It keeps complaining about the cache file and temp.  Looking for that particular error gives me nothing related, I found some hits relating to Google drive, but the requirements appear to be different than the Azure blob we have.

If I can provide any additional information, let me know what.

Thanks!

kapitainsky · November 29, 2024, 4:20am

when using systemd you do not need this flag. Remove it and try again.

rtstarliper · November 29, 2024, 4:33pm

I could have sworn I tried that in one of my iterations. Apparently not. I still am getting a 'Failed to find user cache dir' message, but the Azure blob is mounted.

Any ideas on that particular message?

asdffdsa · November 29, 2024, 4:56pm

hi, please restart the systemd service, post a full debug log that shows the exact error.

rtstarliper · November 29, 2024, 9:42pm

Here is the rclone.service file:

# /etc/systemd/system/rclone.service
[Unit]
Description=Auto Mount MASD Azure Blob
Wants=network-online.target
After=network-online.target

[Service]
Type=idle
ExecStart=/usr/bin/rclone mount \
          --config /etc/default/rclone/rclone.conf \
          --log-file /var/log/rclone.log \
          --log-level DEBUG \
          --allow-non-empty \
          --allow-other \
          --cache-dir /var/cache/rclone \
          MASD_Azure:/ /var/backups/rclone \

ExecStop=/usr/bin/fusermount -u /var/backups/rclone

[Install]
WantedBy=multi-user.target

The error does not appear in the logs, but it does appear in the status (emphasis added):

● rclone.service - Auto Mount MASD Azure Blob
     Loaded: loaded (/etc/systemd/system/rclone.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-11-29 16:35:15 EST; 35s ago
   Main PID: 2071172 (rclone)
      Tasks: 15 (limit: 115929)
     Memory: 13.6M
        CPU: 159ms
     CGroup: /system.slice/rclone.service
             └─2071172 /usr/bin/rclone mount --config /etc/default/rclone/rclone.conf --log-file /var/log/rclone.log --log-level D>

Nov 29 16:35:15 wmssfs00 systemd[1]: Started Auto Mount MASD Azure Blob.
Nov 29 16:35:15 wmssfs00 rclone[2071172]: 2024/11/29 16:35:15 DEBUG : **Failed to find user cache dir, using temporary directory**: ne>

I am specifying a cache directory, yet the service has the message about failing to find the user cache directory. Is the cache directory I supplied not the cache directory it is looking for? Note that I will be changing the logging level. I don't really need DEBUG logs filling up my log server

ETA: Forgot to include the log from the startup:

2024/11/29 16:44:44 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone"]
2024/11/29 16:44:44 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/29 16:44:44 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/29 16:44:44 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/29 16:44:44 INFO  : Azure root: poll-interval is not supported by this remote
2024/11/29 16:44:44 DEBUG : Azure root: Mounting on "/var/backups/rclone"
2024/11/29 16:44:44 DEBUG : : Root: 
2024/11/29 16:44:44 DEBUG : : >Root: node=/, err=<nil>

asdffdsa · November 29, 2024, 9:48pm

does the cache dir exist?
please, re-start the service, post the top thirty lines of the debug log.

should not be used, unless 1000% sure you need it

rtstarliper · November 29, 2024, 10:11pm

I restarted the service again. Here are the last 30 lines of /var/log/rclone.log. If there is a separate debug log somewhere, please give me a hint, as I do not see it in the log directory. I have nothing for rclone in any home directory.

2024/11/29 16:58:06 DEBUG : borg/wmssfs00/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/11/29 16:58:13 DEBUG : /: Attr: 
2024/11/29 16:58:13 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/11/29 16:58:13 DEBUG : /: Lookup: name="borg"
2024/11/29 16:58:13 DEBUG : /: >Lookup: node=borg/, err=<nil>
2024/11/29 16:58:13 DEBUG : borg/: Attr: 
2024/11/29 16:58:13 DEBUG : borg/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/11/29 16:58:13 DEBUG : borg/: Lookup: name="wmssfs00"
2024/11/29 16:58:13 DEBUG : borg/: >Lookup: node=borg/wmssfs00/, err=<nil>
2024/11/29 16:58:13 DEBUG : borg/wmssfs00/: Attr: 
2024/11/29 16:58:13 DEBUG : borg/wmssfs00/: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/11/29 17:06:25 DEBUG : : forgetting directory cache
2024/11/29 17:06:25 DEBUG : borg: forgetting directory cache
2024/11/29 17:06:25 DEBUG : borg/wmssfs00: forgetting directory cache
2024/11/29 17:06:25 DEBUG : borg/wmssfs00/data: forgetting directory cache
2024/11/29 17:06:25 DEBUG : borg/wmssfs00/data/0: forgetting directory cache
2024/11/29 17:06:25 DEBUG : conf: forgetting directory cache
2024/11/29 17:06:25 DEBUG : conf/firewall: forgetting directory cache
2024/11/29 17:06:25 DEBUG : data: forgetting directory cache
2024/11/29 17:06:25 DEBUG : win: forgetting directory cache
2024/11/29 17:08:03 DEBUG : /var/backups/rclone: Unmounted externally. Just exit now.
2024/11/29 17:08:03 DEBUG : rclone: Version "v1.68.2" finishing with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone"]
2024/11/29 17:08:03 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-non-empty" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone"]
2024/11/29 17:08:03 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/29 17:08:03 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/29 17:08:03 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/29 17:08:03 INFO  : Azure root: poll-interval is not supported by this remote
2024/11/29 17:08:03 DEBUG : Azure root: Mounting on "/var/backups/rclone"
2024/11/29 17:08:03 DEBUG : : Root: 
2024/11/29 17:08:03 DEBUG : : >Root: node=/, err=<nil>

The cache directory does exist:

root@wmssfs00:/var/cache/rclone# pwd
/var/cache/rclone

asdffdsa · November 29, 2024, 10:15pm

did not see that in the debug log snippet you posted.

fwiw, to keep the log small, each time you start the service, delete the rclone debug log.

rtstarliper · November 29, 2024, 10:19pm

It does not appear in the log. It appears in the status:

● rclone.service - Auto Mount MASD Azure Blob
     Loaded: loaded (/etc/systemd/system/rclone.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-11-29 17:08:03 EST; 11min ago
   Main PID: 2081245 (rclone)
      Tasks: 15 (limit: 115929)
     Memory: 19.4M
        CPU: 176ms
     CGroup: /system.slice/rclone.service
             └─2081245 /usr/bin/rclone mount --config /etc/default/rclone/rclone.conf --log-file /var/log/rclone.log --log-level D>

Nov 29 17:08:03 wmssfs00 systemd[1]: Started Auto Mount MASD Azure Blob.
Nov 29 17:08:03 wmssfs00 rclone[2081245]: 2024/11/29 17:08:03 DEBUG : Failed to find user cache dir, using temporary directory: ne>
root@wmssfs00:/var/log#

Forgot to add - the log is getting nuked periodically. Once I have everything set, I will be changing the log level and setting up rotation with the rest of the system logs.

asdffdsa · November 29, 2024, 10:29pm

for testing, run the command direct on the command line.
once that is working 100%, then use systemd.

rtstarliper · November 29, 2024, 10:44pm

I only see that message when launching via systemd. If I launch on the command line:

root@wmssfs00:/var/log# /usr/bin/rclone mount --config /etc/default/rclone/rclone.conf --log-file /var/log/rclone.log --log-level DEBUG --allow-other --cache-dir /var/cache/rclone MASD_Azure:/ /var/backups/rclone

it just sits there (which I would expect, as I did not specify it to run as a daemon).
I see no error in the log, either:

2024/11/29 17:33:06 DEBUG : rclone: Version "v1.68.2" starting with parameters ["/usr/bin/rclone" "mount" "--config" "/etc/default/rclone/rclone.conf" "--log-file" "/var/log/rclone.log" "--log-level" "DEBUG" "--allow-other" "--cache-dir" "/var/cache/rclone" "MASD_Azure:/" "/var/backups/rclone"]
2024/11/29 17:33:06 DEBUG : Creating backend with remote "MASD_Azure:/"
2024/11/29 17:33:06 DEBUG : Using config file from "/etc/default/rclone/rclone.conf"
2024/11/29 17:33:06 DEBUG : fs cache: renaming cache item "MASD_Azure:/" to be canonical "MASD_Azure:"
2024/11/29 17:33:06 INFO  : Azure root: poll-interval is not supported by this remote
2024/11/29 17:33:06 DEBUG : Azure root: Mounting on "/var/backups/rclone"
2024/11/29 17:33:06 DEBUG : : Root: 
2024/11/29 17:33:06 DEBUG : : >Root: node=/, err=<nil>
2024/11/29 17:33:47 DEBUG : /: Attr: 
2024/11/29 17:33:47 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2024/11/29 17:33:47 DEBUG : : Statfs: 
2024/11/29 17:33:47 DEBUG : : >Statfs: stat={Blocks:274877906944 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>

I get some different output in the log, though.

I am beginning to thing this is something odd to do with systemd. I look in /var/cache/rclone/ and see nothing. I look in the system temp directory (/tmp/) and see nothing. I am executing this as root, which is what I believe systemd does. There is no temp directory in /root/.

As far as I can tell, it is working - I have been able to access the blob both for reading and writing - and I verified I was accessing it there by shutting down rclone (and the connection) and am unable to get to the /var/backups/rclone mount point. Which is what I expected to happen.

By the way, I did get rid of the --allow-non-empty. Not sure exactly why that was there in the first place. I did a LOT of googling when I was trying to get this to work, that must have come from an example somewhere.

asdffdsa · November 29, 2024, 11:04pm

i would expect to see something like this in your output

DEBUG : vfs cache: data root is     "\\\\?\\d:\\rclone\\cache\\zork\\vfs\\wasabi01\\zork"
DEBUG : vfs cache: metadata root is "\\\\?\\d:\\rclone\\cache\\zork\\vfsMeta\\wasabi01\\zork"

rclone config paths

rtstarliper · November 29, 2024, 11:41pm

So even though I specify the paths on the command line, they are not used?

I have --config /etc/default/rclone.conf; --log-file /var/log/rclone.log; and --cache-dir /var/cache/rclone in the command when I launch. I know it is writing the logs to /var/log/rclone.log like I specified. I would have expected it to use the other directories as specified. Is there a way to specify what to use for each of those?

If I run the rclone config paths, it shows:

root@wmssfs00:/var/log# rclone config paths
Config file: /root/.config/rclone/rclone.conf
Cache dir:   /root/.cache/rclone
Temp dir:    /tmp
root@wmssfs00:/var/log#

When I looked at how to change the paths, I saw the aforementioned switches. If there is something else I should be doing to make it work with the paths I choose, please let me know where to look.

Thanks for your help, by the way. I am configuring backups across a number of Linux and Windows servers/workstations, and am using Borg backup (at least for Linux, haven't gotten to Windows yet). It was in the Borg documentation I came across rclone. I am not exactly a Linux novice, but for some reason am getting hung up with rclone.

system · December 2, 2024, 11:42pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.