Rclone works but not from cron (raspberry pi)

Howdy, I know there are years of posts related to this and I've read many of 'em. I still haven't got this working, so, I come hat in hand to ask for help.

What is the problem you are having with rclone?

rclone is the answer I've been looking for. I can use rclone from command line. It works when I put it in a bash script (*.sh) and I can call it through python scripts too, (*.py). But I can't get it to work from cron. Not if I put the command in the crontab, not if I call the script (*.sh or *.py) from cron. I can cron plenty other jobs, but somehow it's like cron is ignoring rclone. Maybe that's a clue.

What is your rclone version (output from rclone version)

pi@zero-tester:~ $ rclone version
rclone v1.53.2
- os/arch: linux/arm
- go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Raspberry Pi OS (32-bit) Lite
Minimal image based on Debian Buster
Version: August 2020
Release date: 2020-08-20
Kernel version: 5.4

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

These work

pi@zero-tester:~ $ rclone move /home/pi/Google/ Google:/rpi/
pi@zero-tester:~ $ /usr/bin/rclone move /home/pi/Google/ Google:/rpi/
pi@zero-tester:~ $ /home/pi/google.sh

The google.sh script looks like this:

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# move files from local/source to cloud:destination
/usr/bin/rclone move /home/pi/Google/ Google:/rpi/
exit

And yes, :slight_smile:

pi@zero-tester:~ $ sudo chmod +x google.sh

However, when I open crontab

pi@zero-tester:~ $ sudo crontab -e

and insert any of these jobs I get no action.

*/1 * * * * /usr/bin/rclone move /home/pi/Google/ Google:/rpi/
*/1 * * * * /bin/bash /home/pi/google.sh >/dev/null 2>&1
*/1 * * * * /home/pi/google.sh

NOTE: I tried those ^ by editing

~ $ sudo crontab -e

Should I be entering rclone as a job in this file below?

~ $ sudo nano /etc/crontab

Also, I have read many comments on cron SHELL and PATH. Here is what my /etc/crontab file shows ...

SHELL=/bin/sh:/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

So that's where I'm coming from. I'm using a Raspberry Pi. My use case is this: when a sensor triggers I make a note in a log. We copy that log to a backup directory called

/home/pi/Google/

and then in the script (python) I can run

import subprocess
subprocess.call(['sh','/home/pi/google.sh'])

and it works.

But I want to schedule cron jobs to upload the logs at intervals rather than every time a sensor triggers. I can imagine workarounds, but I want this to work in cron. I use cron for other jobs. Love it. What am I missing? I bet it's something fundamental and easy but I don't really understand cron or bash, SHELL or PATH. So I just haven't figured it out yet. Really appreciate any help.

You have not posted any logs, but you could try adding the path to your config file in your google.sh shell script.

Change this command from:

/usr/bin/rclone move /home/pi/Google/ Google:/rpi/

to:

/usr/bin/rclone --config="/path/to/rclone.conf" move /home/pi/Google/ Google:/rpi/

of course change this /path/to/rclone.conf to match yours

you can get this path using this command

rclone config file

Or try crontab -e without sudo, as using sudo will be editing the root crontab rather than your own users crontab (at least that is the case on ubuntu, I am not familiar with a pi)

Well I am impressed! Thank you blim5001. I did as you suggested and now it works.

To be specific, here's what I did:
I stop cron, then sudo open crontab to edit as suggested, reload cron, start cron, breathe easy.

pi@zero-tester:~ $ sudo service cron stop
pi@zero-tester:~ $ sudo crontab -e

In crontab I have confirmed both the following lines work. (NOTE: Only use one or the other, not both, as that'd be redundant.) To be specific, the first line is edited as blim5001 suggested. The second line calls a script that has been edited to match the first line.

*/1 * * * * /usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/
*/1 * * * * /home/pi/google.sh >/dev/nll 2>&1

Here is the updated google.sh script with the OLD rclone command commented out and the NEW rclone command visible:

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# move files from local/source to cloud:destination
#/usr/bin/rclone move /home/pi/Google/ Google:/rpi/
/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/
exit

Lastly, I reload and start cron like this...

pi@zero-tester:~ $ sudo service cron reload
pi@zero-tester:~ $ sudo service cron start

And that is it. Now it's working. If something fails I'll report back here, otherwise, I hope this helps the next person. And thank you blim5001, you helped me out!

Follow up. Regarding logfiles, crontabs, and sudo.

The cron solution still works.
Thanks @blim5001! But entering rclone via command line now thows permissions errors (see logs below.) I can work with it, so it's not a show-stopper, but I don't understand.

Bottom-line
How to eliminate the error: Failed to load config file?

What follows is a post-mortem of sorts.

Crontabs
Up front, I don't understand the difference between

~$ sudo crontab -e

and

~$ crontab -e

... so I'll look into that. I thought "sudo crontab -e" is the right place for all the cron jobs I want to perform, and it works, so I left it at that. But maybe it's something I should better understand. I'm open to any pointers.

Log files
From the command line, when try without "sudo" I get 'permission denied' errors. First, here is the log:

2020/11/06 09:10:04 Failed to load config file "/home/pi/.config/rclone/rclone.conf": open /home/pi/.config/rclone/rclone.conf: permission denied

That message is generated by each of the following commands:

~$ rclone move /home/pi/Google/ Google:/rpi/ --log-file rclone.txt
~$ rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/ --log-file rclone.txt

So those do not work, but I still have my bash script (google.sh see above) and that works with sudo.

~$ sudo ./google.sh

After testing that, I added 'sudo' to the front end of the rclone command in google.sh and now I can run it as this:

~$ ./google.sh

So that's nice.
I can run rclone in cron (sudo crontab -e) and I can run rclone in my google.sh script. My project can proceed. But I want deeper knowledge.
Thanks to any who help.

Will edit the crontab file for the root user

Will edit the crontab file for the user you are currently logged in as. Which I am guessing is pi

if you do
cd /home/pi/.config/rclone
then
ls -la

Who owns the config file and what are the permissions?

@blim5001, Yes, "pi" is the user.
When I do as you ask I get this:

pi@zero-tester:~ $ cd /home/pi/.config/rclone
pi@zero-tester:~/.config/rclone $ ls -la
total 12
drwxr-xr-x 2 pi   pi 4096 Nov  6 10:26 .
drwxr-xr-x 3 pi   pi 4096 Nov  2 13:37 ..
-rw------- 1 root pi  607 Nov  6 10:26 rclone.conf

What do you see?

so the rclone config file is owner by root and -rw------- means only root can read and write it

which is why you need to use sudo

I would probably change the owner to pi

chown pi:pi rclone.conf

and set the permissions to rw for user pi and r for eveyone else, at least thats what the perms are on that file on my mac.

chmod 644 rclone.conf

reply

As directed, I did this...

pi@zero-tester:~ $ cd /home/pi/.config/rclone
pi@zero-tester:~/.config/rclone $ chown pi:pi rclone.conf
chown: changing ownership of 'rclone.conf': Operation not permitted

At first try it didn't work. So I ran chown as 'sudo chown' and it worked.

pi@zero-tester:~/.config/rclone $ sudo chown pi:pi rclone.conf
pi@zero-tester:~/.config/rclone $ sudo chmod 644 rclone.conf

Seems good. Reviewing permissions...

pi@zero-tester:~ $ cd /home/pi/.config/rclone
pi@zero-tester:~/.config/rclone $ ls -la
total 12
drwxr-xr-x 2 pi   pi 4096 Nov  6 12:26 .
drwxr-xr-x 3 pi   pi 4096 Nov  2 13:37 ..
-rw-r--r-- 1 root pi  607 Nov  6 12:26 rclone.conf

Don't really know what that means but I could find out.
But first, testing. Command line rclone works.

pi@zero-tester:~ $ rclone move /home/pi/Google/ Google:/rpi/
pi@zero-tester:~ $ google.sh

Calling my rclone bash script (google.sh) from command line or from within a python script works with each of the three iterations shown.

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# move files from local/source to cloud:destination
#rclone move /home/pi/Google/ Google:/rpi/
#/usr/bin/rclone move /home/pi/Google/ Google:/rpi/
/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/
exit

Now testing in

~$ sudo crontab -e 

Hmmmm. Mixed results.

# this does not work (google.sh formatted as above)
*/1 * * * * /home/pi/google.sh >/dev/null 2>&1
# this works
*/1 * * * * /usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/

I'm confused. I suppose my desired outcome is to call rclone bash script from cron and enter at command line as desired. But I seem to be experiencing either/or situations. I can still perform the functions I want: specifically I can make a cron job if I enter the rclone command in crontab but not if I call an rclone bash script from cron. I don't understand these inconsistencies. Thoughts? Also, maybe I need to have a fresh look at this later tonight.

I really appreciate your time and efforts @blim5001. This is all very helpful. Thank you.

Not quite sure why this does not work.

What happens if you put in your own users crontab with just

crontab -e

Right. Yes.

*/1 * * * * /home/pi/google.sh >/dev/null 2>&1

Works in

crontab -e

Does not (currently) work in

sudo crontab -e

So, is this a permissions thing? Is there a chmod solution that'll allow both 'sudo' and user 'pi' equal access or am I thinking about this all wrong? Also, I am very ignorant of these environments so I beg your pardon.

So I've currently got a set of solutions. That's good. But I also feel like I'm missing some permissions setting or some other thing that'd make this all more cohesive. But it works, so, there's that.

This is a process I'm going to implement on several raspberry pi machines so I don't have to scp files over my network and then up to the cloud. And if I have to go through a number of steps on each machine I just want to make sure I'm getting 'em right.

Happy Friday, btw.

It is generally best practice to avoid using sudo (root) where possible.

So if it all works as your user pi without using sudo then that's good

To debug why it does not work in root crontab, you would need to specify a log file in your shell script and look in there for errors.

e.g. in your shell script

/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" --log-level DEBUG --log-file /home/pi/logs/rclone.log move /home/pi/Google/ Google:/rpi/

Okay. So this is a proper log. I think maybe I don't understand logs yet :slight_smile:
Added the following line

/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" --log-level DEBUG --log-file /home/pi/logs/rclone.log move /home/pi/Google/ Google:/rpi/

first tried it in 'sudo crontab -e' and see success...
NOTE: I don't know if the MD5 string is a password so I edited that bit but otherwise this is the entire output...

2020/11/06 14:41:03 DEBUG : rclone: Version "v1.53.2" starting with parameters ["/usr/bin/rclone" "--config=/home/pi/.config/rclone/rclone.conf" "--log-level" "DEBUG" "--log-file" "/home/pi/logs/rclone.log" "move" "/home/pi/Google/" "Google:/rpi/"]
2020/11/06 14:41:03 DEBUG : Creating backend with remote "/home/pi/Google/"
2020/11/06 14:41:03 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2020/11/06 14:41:03 DEBUG : Creating backend with remote "Google:/rpi/"
2020/11/06 14:41:03 DEBUG : Google: Loaded invalid token from config file - ignoring
2020/11/06 14:41:06 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2020/11/06 14:41:06 DEBUG : Google: Saved new token in config file
2020/11/06 14:41:07 DEBUG : Google drive root 'rpi': root_folder_id = "0AOcy-IZFyBV6Uk9PVA" - save this in the config to speed up startup
2020/11/06 14:41:07 DEBUG : fs cache: renaming cache item "Google:/rpi/" to be canonical "Google:rpi"
2020/11/06 14:41:07 DEBUG : Google drive root 'rpi': Waiting for checks to finish
2020/11/06 14:41:07 DEBUG : Google drive root 'rpi': Waiting for transfers to finish
2020/11/06 14:41:09 DEBUG : test-20201106-133200.txt: MD5 = d4****(is this confidential?)***27e OK
2020/11/06 14:41:09 INFO  : test-20201106-133200.txt: Copied (new)
2020/11/06 14:41:09 INFO  : test-20201106-133200.txt: Deleted
2020/11/06 14:41:09 DEBUG : test-20201106-133225.txt: MD5 = d4****(is this confidential?)***27e OK
2020/11/06 14:41:09 INFO  : test-20201106-133225.txt: Copied (new)
2020/11/06 14:41:09 INFO  : test-20201106-133225.txt: Deleted
2020/11/06 14:41:09 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 4 / 4, 100%
Deleted:                2
Renamed:                2
Transferred:            2 / 2, 100%
Elapsed time:         7.7s

And then I pasted the command into the bash script and get the following...

2020/11/06 15:01:05 DEBUG : 5 go routines active
2020/11/06 15:02:02 DEBUG : rclone: Version "v1.53.2" starting with parameters ["/usr/bin/rclone" "--config=/home/pi/.config/rclone/rclone.conf" "--log-level" "DEBUG" "--log-file" "/home/pi/logs/rclone.log" "move" "/home/pi/Google/" "Google:/rpi/"]
2020/11/06 15:02:03 DEBUG : Creating backend with remote "/home/pi/Google/"
2020/11/06 15:02:03 DEBUG : Using config file from "/home/pi/.config/rclone/rclone.conf"
2020/11/06 15:02:03 DEBUG : Creating backend with remote "Google:/rpi/"
2020/11/06 15:02:05 DEBUG : Google drive root 'rpi': root_folder_id = "0AOcy-IZFyBV6Uk9PVA" - save this in the config to speed up startup
2020/11/06 15:02:05 DEBUG : fs cache: renaming cache item "Google:/rpi/" to be canonical "Google:rpi"
2020/11/06 15:02:06 DEBUG : Google drive root 'rpi': Waiting for checks to finish
2020/11/06 15:02:06 DEBUG : Google drive root 'rpi': Waiting for transfers to finish
2020/11/06 15:02:07 DEBUG : test-20201106-150142.txt: MD5 = d4****(is this confidential?)***27e OK
2020/11/06 15:02:07 INFO  : test-20201106-150142.txt: Copied (new)
2020/11/06 15:02:07 INFO  : test-20201106-150142.txt: Deleted
2020/11/06 15:02:07 DEBUG : test-20201106-150148.txt: MD5 = d4****(is this confidential?)***27e OK
2020/11/06 15:02:07 INFO  : test-20201106-150148.txt: Copied (new)
2020/11/06 15:02:07 INFO  : test-20201106-150148.txt: Deleted
2020/11/06 15:02:07 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 4 / 4, 100%
Deleted:                2
Renamed:                2
Transferred:            2 / 2, 100%
Elapsed time:         5.2s

2020/11/06 15:02:07 DEBUG : 10 go routines active

I'm ignorant of logs so please let me know if I'm supposed to share some other info. I don't know how to evaluate this.

Hold on. It's working. It worked throughout that whole log process and I was so focused on figuring out how to get the logs I didn't realize it was working.

To be clear.

sudo crontab -e

is calling

*/1 * * * * /home/pi/google.sh >/dev/null 2>&1

and the shell script has the extended log command @blim5001 suggested

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# move files from local/source to cloud:destination
#rclone move /home/pi/Google/ Google:/rpi/
#/usr/bin/rclone move /home/pi/Google/ Google:/rpi/
#/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/G$
# Now with Logs!
/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" --log-level DEBUG --log-file /home/pi/logs/rclone.log move /home/pi/Google/ Google:/rpi/
exit

I know I should now test

crontab -e

and other things but I'm needed elsewhere and my brain is a bit swirly from all this. I need a break.

Tested calling script from command line and it works...

~$ ./google.sh

So, note to self: it's working in 'sudo crontab -e' (calling script) and from cmd line (calling script.)

I'm going to look at this again in a few hours. I need to regroup so I can parse what I do / not know.
Thank you!

All is well

OK. I've had some time to marinate. I can tell you it works in every way I want it to and in every way I can think to test. So, it works from sudo crontab and user crontab. It works entering rclone at command line, from python or bash scripts. It works calling scripts from cron or from other scripts. It all works (for now!)

For posterity, and because I will probably come back here and refer to this thread as I set up subsequent pi's for IOT projects, what follows is a sloppy summary. --Verbose

Goal: Use rclone on Raspberry Pi

  • (1) from command line
  • (2) from within bash scripts and python scripts
  • (3) from cron

The Solutions applied from the above thread involve a combination of /complete/path/names/, explicitly calling the --config file, setting permissions and ownership of rclone.conf, and not mentioned above but learned elsewhere, proper formatting of bash scripts and cron commands. I honestly don't know if everything was required or if I made errors in my testing, but everything in this thread helped me. Thank you @blim5001. So if anyone or future me is stuck, try it from the top. Now, here's where I ended up.

crontab
Works in either 'sudo crontab' or 'user crontab':

~$ sudo crontab -e

or what I'm calling 'user crontab'...

~$ crontab -e

Best practice says use the secone one, it's the crontab specific to the user, so I'll call the first, 'sudo crontab' and the second, 'user crontab'. At the advice of my betters I'll be using 'user crontab'. To modify 'user crontab' on the raspberry pi I first had to set the root password, which I did like this:

~$ sudo su -
~# sudo passwd
(enter your password)
~# exit 

Great. Based on where I ended up, I want to emphasize that I think the combination of chown/chmod permissions and sudo/user crontab placement is central, but I don't yet clearly have it sorted out. So I'll brain-dump. Big thanks again to the generosity of @blim5001.

Set permissions on rclone.conf:

~$ cd /home/pi/.config/rclone
~/.config/rclone $ sudo chown pi:pi rclone.conf
~/.config/rclone $ sudo chmod 644 rclone.conf
:~/.config/rclone $ ls -la
total 12
drwxr-xr-x 2 pi   pi 4096 Nov  6 12:26 .
drwxr-xr-x 3 pi   pi 4096 Nov  2 13:37 ..
-rw-r--r-- 1 root pi  607 Nov  6 12:26 rclone.conf

NOTE: Permissions of 644 mean that the owner of the file has read and write access, while the group members and other users on the system only have read access. So, making sure 'user' has r/w access.

Then enter the 'user crontab' like this:

~$ crontab -e

I've verified all three of these methods work. I prefer (v.1):

#########################################################
#	rclone in crontab 
#	all 3 of these work. pick one.
#
# (v.1) call the bash script 
*/1 * * * * /home/pi/google.sh >/dev/null 2>&1
#
# (v.2) execute the command
#*/1 * * * * /usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" mo$
#
# (v.3) execute the command with logs
#/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" --log-level DEBUG --log-file /home/pi/logs/rclone.log move /home/pi/Google/ Google:/rpi/
#
#########################################################

This command line entry works...

~$ rclone move /home/pi/Google/ Google:/rpi/

This bash script format works...

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# move files from local/source to cloud:destination
rclone move /home/pi/Google/ Google:/rpi/
exit

This python script format works. It calls the google.sh bash script above...

#!/usr/bin/python
import subprocess
subprocess.call(['sh','/home/pi/google.sh'])

That's really it. Cron. Bash. Python. Command line. Permissions set to allow user read/write access. User crontab preferred, though it does work in sudo crontab, too. In the bash script you can use any version of the rclone command, I just prefer the minimalist.

While testing, here are all versions of rclone command I used...

# simple
rclone move /home/pi/Google/ Google:/rpi/
# explicit rclone path
/usr/bin/rclone move /home/pi/Google/ Google:/rpi/
# explicit path to rclone and to rclone.conf
/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" move /home/pi/Google/ Google:/rpi/
# explicit paths and now with logs
/usr/bin/rclone --config="/home/pi/.config/rclone/rclone.conf" --log-level DEBUG --log-file /home/pi/logs/rclone.log move /home/pi/Google/ Google:/rpi/

Wow, is that everything? I hope this helps someone else. If it does, big thanks to @blim5001 for bouncing ideas.

Enjoy what you're doing!

Final word

I just set up a new headless Raspberry Pi zero with rclone and it works 100%.

pi@zero:~ $ sudo apt install rclone
pi@zero:~ $ rclone config

I ran through the setup for connecting to Google Drive.
Then I set the rclone.conf permissions like this...

pi@zero:~ $ cd /home/pi/.config/rclone
pi@zero:~/.config/rclone $ sudo chown pi:pi rclone.conf
pi@zero:~/.config/rclone $ sudo chmod 644 rclone.conf
pi@zero:~/.config/rclone $ ls -la
total 12
drwxr-xr-x 2 pi   pi 4096 Nov  6 12:26 .
drwxr-xr-x 3 pi   pi 4096 Nov  2 13:37 ..
-rw-r--r-- 1 pi   pi  434 Nov  6 23:04 rclone.conf

Then I made a bash script called google.sh stored at /home/pi/* but it could be anywhere...

#!/bin/bash
if pidof -o %PPID -x "google.sh"; then
exit 1
fi
# cron calls this to move files from local/source to cloud:destination
rclone move /home/pi/files/ Google:/rpi/
exit

I made it executable

pi@zero:~ $ sudo chmod a+x /home/pi/google.sh

I put test files into /home/pi/files and ran the google.sh script

pi@zero:~ $ /home/pi/google.sh

It worked.
Then I made a quick python script named 'call.py'

#!/usr/bin/python
import subprocess
subprocess.call(['sh','/home/pi/google.sh'])

It calls the 'google.sh' script. I ran it...

pi@zero:~ $ python call.py

It worked.
Then I entered crontab like this...

pi@zero:~ $ crontab -e

And I pasted the following command in there.

# auto backup /home/pi/files/ to Google:/rpi/
*/1 * * * * /home/pi/google.sh >/dev/null 2>&1

I put some more test files in /home/pi/files/ and after a minute they moved up to the Google Drive.

That's it. It works. It's replicatable. I can cloud connect my IoT projects. I hope this helps someone. Thanks.

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.