Ubuntu 18.04 Crontab Gdrive Upload Not Working

Hey All. I'll admit, I'm a bit of an Ubuntu noob.

I feel as though I followed all of Animosities steps on his Github page. I have everything working but the cron job to auto upload from my server to my Google Drive each evening.

In /etc/crontab I have:

#Cloud Upload
00 23   * * *   root    /opt/rclone/scripts/upload_cloud

In /opt/rclone/scripts/upload_cloud I have:

#!/bin/bash
# RClone Config file
RCLONE_CONFIG=/data/rclone/rclone.conf
export RCLONE_CONFIG
LOCKFILE="/var/lock/`basename $0`"

(
  # Wait for lock for 5 seconds
  flock -x -w 5 200 || exit 1

# Move older local files to the cloud
/usr/bin/rclone move "/home/russell/data/iTunes/" "gcrypt:iTunes/" --log-file "/opt/rclone/logs/upload.log" --progress -v --drive-chunk-size 64M --delete-empty-src-dirs --fast-list

) 200> ${LOCKFILE}

Nothing turns up in the upload.log.

Syslog has:
Aug 14 23:00:01 [SVR] CRON[25271]: (root) CMD ( /opt/rclone/scripts/upload_cloud)
Aug 14 23:00:01 [SVR] postfix/pickup[13643]: BE32E1A0C24: uid=0 from=
Aug 14 23:00:01 [SVR] postfix/cleanup[25275]: BE32E1A0C24: message-id=<20190814130001.BE32E1A0C24@[SVR]>
Aug 14 23:00:01 [SVR] postfix/qmgr[13644]: BE32E1A0C24: from=<root@[SVR]>, size=663, nrcpt=1 (queue active)
Aug 14 23:00:07 [SVR] postfix/smtp[25277]: BE32E1A0C24: to=<root@[SVR]>, orig_to=, relay=smtp.gmail.com[IP ADDRESS]:587, delay=5.5, delays=0.19/0.11/3.5/1.7, dsn=$
Aug 14 23:00:07 [SVR] postfix/qmgr[13644]: BE32E1A0C24: removed

So it looks like the crontab is working, so I'm guessing the issue is with my upload script?

When I run the command manually, it works fine.
/usr/bin/rclone move "/home/russell/data/iTunes/" "gcrypt:iTunes/" --log-file "/opt/rclone/logs/upload.log" --progress -v --drive-chunk-size 64M --delete-empty-src-dirs --fast-list

As a side note, if anyone also knows how to change the email address postfix is sending to that'd be great too :laughing:

Actually, disregard the postfix comment - I've worked that out :crazy_face:

But there is an additional question to the upload script not working. Is there a way to also have it clean up the directories so I'm not left with a heap of [TVShow]/[Season 01] etc directories?

It should clean them up with the --delete-empty-src-dirs argument. Is it not cleaning them up?

Sorry, you're right. The manual upload finally completed and I can see on completion it's purged the empty directories so that's great.

Only remaining issue now is from my original post and why my upload script won't run as a scheduled cron task if anyone has any ideas.

Try adding -vv to the rclone command then you should definitely see something in the log.

Ok, so I changed upload_script to be:

#!/bin/bash
# RClone Config file
RCLONE_CONFIG=/data/rclone/rclone.conf
export RCLONE_CONFIG
LOCKFILE="/var/lock/`basename $0`"

(
  # Wait for lock for 5 seconds
  flock -x -w 5 200 || exit 1

# Move older local files to the cloud
/usr/bin/rclone move "/home/russell/data/iTunes/" "gcrypt:iTunes/" --log-file "/opt/rclone/logs/upload.log" --progress -vv --drive-chunk-size 64M --delete-empty-src-dirs --fast-list

) 200> ${LOCKFILE}

and there was no entry made into upload.log at all.

Syslog has:

Aug 16 10:00:01 [SVR] CRON[7134]: (root) CMD (   /opt/rclone/scripts/upload_cloud)
Aug 16 10:00:01 [SVR] postfix/pickup[26802]: 4832A1A0BA4: uid=0 from=<root>
Aug 16 10:00:01 [SVR] postfix/cleanup[7138]: 4832A1A0BA4: message-id=<20190816000001.4832A1A0BA4@RBCMVPS.com>
Aug 16 10:00:01 [SVR] postfix/qmgr[28874]: 4832A1A0BA4: from=<root@[SVR].com>, size=659, nrcpt=1 (queue active)
Aug 16 10:00:06 [SVR] postfix/smtp[7140]: 4832A1A0BA4: to=<root@[SVR].com>, orig_to=<root>, relay=smtp.gmail.com[IP]:587, delay=5.8, delays=$
Aug 16 10:00:07 [SVR] postfix/qmgr[28874]: 4832A1A0BA4: removed

Crontab is currently setup as:

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
18 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
23 02   * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
38 02   * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
48 02   1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#
#Cloud Upload
00 10   * * *   root    /opt/rclone/scripts/upload_cloud

Is that line right? There is just a $ after the log file name and no log file specified.

Sorry, I've fixed up the post so it has the full script. I don't know what happened when I was copying and pasting to here.

I'd guess it has something to do with the lockfile being there if you are never seeing a log file update.

I do this instead:

#exit if running
if [[ "`pidof -x $(basename $0) -o %PPID`" ]]; then exit; fi

It might be worth adding >> /tmp/rclone.log 2>&1 on the end of your crontab line - maybe there is a typo in the command line I can't see!

Note that you don't want --progress in a crontab - it doesn't play nice with log files!

Do you have your config at /data/rclone/rclone.conf? When I first set it up I had to change it to "RCLONE_CONFIG=/home/joe/.config/rclone/rclone.conf".

Sorry! Don't know what I was doing, thought it was a copy of yours from github but must have gotten that from somewhere else. I've updated it and it's now:
#!/usr/bin/bash
# RClone Config file
RCLONE_CONFIG=/data/rclone/rclone.conf
export RCLONE_CONFIG

#exit if running
if [[ "`pidof -x $(basename $0) -o %PPID`" ]]; then exit; fi

# Move older local files to the cloud
/usr/bin/rclone move "/home/russell/data/iTunes/" "gcrypt:iTunes/" --log-file "/opt/rclone/logs/upload.log" -v --drive-chunk-size 64M --delete-empty-src-dirs --fast-list

Thanks. Double checked and the one I'm using is definitely /data/rclone/rclone.conf.

This has gotten me a bit further! The rclone.log from crontab was showing permission denied. So I chown the directory and contents to be root:root now and chmod 777 the upload_cloud file. That's changed it to now display:
/bin/sh: 1: /opt/rclone/scripts/upload_cloud: not found

/etc/crontab is:
# /etc/crontab: system-wide crontab
# Unlike any other crontab you don't have to run the `crontab'
# command to install the new version when you edit this file
# and files in /etc/cron.d. These files also have username fields,
# that none of the other crontabs do.

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

# m h dom mon dow user  command
18 *    * * *   root    cd / && run-parts --report /etc/cron.hourly
23 02   * * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily )
38 02   * * 7   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.weekly )
48 02   1 * *   root    test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.monthly )
#
#Cloud Upload
10 18  * * *   root /opt/rclone/scripts/upload_cloud >> /tmp/rclone.log 2>&1

That is saying it can't find the shell script.

What does

ls -l /opt/rclone/scripts/upload_cloud

show?

Hey ncw. It shows:
-rwxrwxrwx 1 root root 386 Aug 17 18:15 /opt/rclone/scripts/upload_cloud

That should be

#Cloud Upload
00 23   * * * /opt/rclone/scripts/upload_cloud

Hopefully that will fix it!

Any entry in /etc/crontab does need the user. Only per-user crontabs don't have the user field.

What output does the below command give:

which bash

Sorry didn't spot that, I thought it was a user crontab

Hey, it shows:
root@[SVR]:/# which bash
/bin/bash