Rclone script problem when run as a cron job

#1

hi guys
i am using a script to create daily backups of a folder + sql file on a server running centos using rclone that uploads to azure.

this script works fine when I run it via ssh console, but if I add it as cron job in /etc/crontab, the script runs, creates the dump.sql, puts the archive in tmp folder, but does not upload it to azure. i am guessing it fails at the rclone part.

here is the script,what is the problem here, can someone tell me?

#!/usr/bin/env bash
set -e
set -u

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
export PATH

today=$(date +%d-%m-%Y-%H:%M)

logs="/rclone.log"
# name of the directory where the archive_file will be created
backupdir="/tmp"
# Container name on AZURE
container="apache-backup"
# name and location of sql dump file
sql_dump_file="/dump.sql"
# AZURE location
backend="azure:${container}"
# name of the archive file, change backup_ to backup_servername if using script on multiple different servers to avoid confusion and possibly overwriting of backups.
archive_file="$backupdir/backup_$(date '+%Y-%m-%d_%H%M%S').tar.gz"
# Rclone exports for connecting to AZURE
export RCLONE_CONFIG_AZURE_TYPE=azureblob
export RCLONE_CONFIG_AZURE_ACCOUNT=xxxxx
export RCLONE_CONFIG_AZURE_KEY=xxxxxxxx

# dumps the sql database into sql_dump_file
echo $today "Creating sql dump" >> $logs
mysqldump --quick --single-transaction --all-databases > "${sql_dump_file}"
if [ -e "${sql_dump_file}" ] ; then
        echo $today "sql dump file created" >> $logs
else
        echo $today "creation of sql dump failed, exiting" >> $logs
        exit 1
fi
# archives the dirs var/www, root and the sql_dump_file into a tar.gz file (excluding .gnupg dir, 
echo $today "Creating archive file" >> $logs
/usr/bin/tar --exclude='.gnupg' -czf "${archive_file}" /var/www "${sql_dump_file}"
if [ -e "${archive_file}" ] ; then
        echo $today "Archive file created" >> $logs
else
        echo $today "Creation of Archive file failed, cleaning and exiting" >> $logs
        rm -rf "${sql_dump_file}"
        exit 1
fi
# copies the archive file to the AZURE backup location
echo $today "Copying archive to AZURE" >> $logs
/usr/bin/rclone copy -v "${archive_file}" "${backend}" >> $logs 2>&1

        echo $today "Cleanup" >> $logs
        echo "----------------------------------------------------" >> $logs
        rm "${archive_file}" "${sql_dump_file}"
exit 0
#2

What user is the crontab running as - the same as you tested rclone with?

#3

Yes, as root.

This is what my /etc/crontab file looks like

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
@hourly root sh /rclone.sh > /output-crontab.log
#4

What do the logs say?

The usual problem people have with rclone in crontabs is it not being able to find its config file, so you can try adding the --config /path/to/config directive. Use rclone config file to show where it is.

#5

Ran the ‘rclone config’ command

Here is the output

rclone config
2018/12/05 17:26:54 NOTICE: Config file “/root/.config/rclone/rclone.conf” not found - using defaults
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config

#6

The script runs without a config file set because we mention variables in the script itself. Are you saying we should add a config for the remote destination?

#7

Sorry, missed that bit in your script. You don’t need a config file in that case.

What do the logs say from the cronjob? They will have a clue in I would have thought.

#8

the logs are empty.
If you see the script above, its actually doing so:

  1. making sql dump
  2. adding a folder to backup in archive + the sql dump made above
  3. uploading file to rclone
  4. deleting it locally

the error comes on part number 3 (uploading to rclone), if i run this script as cron.
this is because the .tar files keep accumulating in /tmp folder but are not uploaded to azure when I put on the cron.

if i run the same script via ssh terminal as ‘sh scriptfile.sh’, then there are no problems.

please help me.

#9

What’s the full command you are running and the actual error message?

running from cron usually means you are missing a parameter or something you have set that isn’t being set in cron as it doesn’t read in your profile.

#10

how do I add bash code in my reply on this forum? i can only find quote preformatting

#11

You can put the 's before and after it.

So like

here
#12 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed
@hourly root sh /rclone.sh > /output-crontab.log

This is the command I am using in /etc/crontab for cron.

#13

What’s in the rclone.sh and what’s the error?

#14

I just updated my post above with full scripts - please check 1st post for bash script.

#15

You can do a few things.

You can add a “-x” after the bash on the first line and that’ll dump all the output of the commands out when it runs. You can see which command is failing.

I think you are missing passing the config file location to rclone.

You think this line is failing?

/usr/bin/rclone copy -v "${archive_file}" "${backend}" >> $logs 2>&1

As an example this fails for me:

* * * * * /usr/bin/rclone lsd gcrypt: >> /tmp/test.log

but this works

* * * * * /usr/bin/rclone --config /data/rclone/rclone.conf lsd gcrypt: >> /tmp/test.log

You can also set it in the script with this:

# RClone Config file
RCLONE_CONFIG=/data/rclone/rclone.conf
#16

You mean like ‘set -x’ like mentioned on the top of the script here?

#!/usr/bin/env bash
set -e
set -u

Or you mean in the /etc/crontab file?

#17

Like this:

felix@gemini:~/scripts$ cat test.sh
#!/bin/bash -x
ls /etc/hosts
felix@gemini:~/scripts$ ./test.sh
+ ls /etc/hosts
/etc/hosts

Each command will be printed out when you run the script.

#18

Ah, but thats not the problem.
The script runs fine when I run like this. It only fails in cron.

#19

I mean add the -x to the script so when you run it from cron it outputs all the command and you can see which one is failing rather than guessing.

My other post had the way to add in the rclone config which I think is your issue.

#20

Ok I did like you said

This is the top of my script file now

#!/usr/bin/env bash -x
set -e
set -u

But when I ran it, there was no output created of the commands like you said.