Prepping Script for Cron Job

smercer · July 16, 2020, 6:38pm

What is the problem you are having with rclone?

Still very much a newbie with rclone, bash, and Cron so I appreciate the patience and help. I've been putting a test script together based on forum research, but there's a lot I don't understand and the script itself isn't working. I've added notes under each line with questions I hope I can get some help answering. Any help at all is greatly appreciated.

What is your rclone version (output from `rclone version`)

v1.52.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac Mojave 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

#!/bin/bash
if pidof -0 %PPID -x “master_sync”; then exit; fi  		
## I understand this line very little, and after I run chmod +x and try to execute, terminal doesn't like this line
LOGFILE=“/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log”
## ideally, I'd prefer this script generate a dated log file (log_DATE) as the number of sync operations will greatly increase. I just have no idea how to make that. This is my placeholder until I figure that out.
FLAGS=“--bwlimit 8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt --log-file=$LOGFILE -v -P -rc”
## very important here, I was told the -rc flag would make sure each sync operation finished before the next one started, I've yet to be able to confirm it works
start=$(date +’%d-%H-%M-%s’)

echo “$(date “+%Y-%m-%d-%H%M RCLONE SYNC STARTED” | tee -a $LOGFILE

/usr/sbin/rclone sync /volumes/VoiceOvers VoiceOvers: $FLAGS	

/usr/sbin/rclone sync /volumes/07_MEDIA 07_MEDIA: $FLAGS
## I was also told I had to use the "absolute path" for rclone to play nice with Cron, not sure if this is correct. I tried looking with invisible files turned on and wasn't able to confirm this is the right directory

echo “$(date “+%Y-%m-%d-%H%M RCLONE SYNC FINISHED IN $(($(date +’%d-%H-%M-%s’)- $start))”$ | tee -a $LOGFILE

exit

The rclone config contents with secrets removed.

type= drive
scope=drive

A log from the command with the `-vv` flag

(from terminal) 
/Users/nahb/Desktop/Rclone/master_sync: line 2: syntax error near unexpected token `then'
/Users/nahb/Desktop/Rclone/master_sync: line 2: `If pidof -0 %PPID -x “master_sync”; then exit; fi

asdffdsa · July 16, 2020, 6:44pm

yeah, scripting can be confusing.

try if instead of If

smercer · July 16, 2020, 7:01pm

Corrected that, now I'm getting this from terminal. Additionally, the log file was not edited.

line 19: syntax error: unexpected end of file
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.
Deleting expired sessions...179 completed.

[Process completed]

smercer · July 16, 2020, 7:10pm

The source and destination check out. Should I not be double clicking the executable?

I've run the same command without the variable and "/usr/sbin/" in the past without issue...

EDIT: there is a mistake in the destination name, correcting now

smercer · July 16, 2020, 7:17pm

Looks like I didn't report the whole story terminal. The full info is as follows

archivemanager:~ nahb$ /Users/nahb/Desktop/Rclone/master_sync ; exit;
/Users/nahb/Desktop/Rclone/master_sync: line 2: pidof: command not found
/Users/nahb/Desktop/Rclone/master_sync: line 6: FLAGS: command not found
/Users/nahb/Desktop/Rclone/master_sync: line 10: unexpected EOF while looking for matching `)'
/Users/nahb/Desktop/Rclone/master_sync: line 19: syntax error: unexpected end of file
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

I corrected the spelling mistake I made previously both here and in the code. The log file still has not been updated however, and it doesn't appear to have processed the sync.

asdffdsa · July 16, 2020, 7:18pm

some flags need two dashes
--bwlimit
--exclude-from
--log-file

try
export LOGFILE= instead of LOGFILE=
and
export FLAGS= instead of FLAGS=

asdffdsa · July 16, 2020, 7:27pm

try
echo $(date +%Y-%m-%d-%H%M) RCLONE SYNC STARTED

smercer · July 16, 2020, 7:33pm

hmm, looks like adding "export" has it throwing errors on all the flags now

archivemanager:~ nahb$ /Users/nahb/Desktop/Rclone/master_sync ; exit;
/Users/nahb/Desktop/Rclone/master_sync: line 2: pidof: command not found
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `8.65M': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `—-exclude-from': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `/Users/nahb/Desktop/Rclone/rclone_exclude_files.txt': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `—-log-file=“/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log”': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `-v': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `-P': not a valid identifier
/Users/nahb/Desktop/Rclone/master_sync: line 6: export: `-rc”': not a valid identifier
tee: “/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log”: No such file or directory
2020-07-16-1532 RCLONE SYNC STARTED
/Users/nahb/Desktop/Rclone/master_sync: line 12: /usr/sbin/rclone: No such file or directory
/Users/nahb/Desktop/Rclone/master_sync: line 14: /usr/sbin/rclone: No such file or directory
/Users/nahb/Desktop/Rclone/master_sync: line 16: unexpected EOF while looking for matching `)'
/Users/nahb/Desktop/Rclone/master_sync: line 19: syntax error: unexpected end of file
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

asdffdsa · July 16, 2020, 7:37pm

if you change the script, then you need to re-post it.

export is not the problem.
you need to fix the errors in order and the first error is line 2.

smercer · July 16, 2020, 7:50pm

Good suggestion, I've made a few more tweaks as well. Looks like I was right to worry about not seeing the rclone code in the directories I mentioned above. How do I find out where this stuff lives?

#!/bin/bash
if pidof -0 %PPID -x “master_sync.sh”; then exit; fi  		

RCLONE_CONFIG=/Users/nahb/.config/rclone/rclone.conf.    	
export RCLONE_CONFIG

LOGFILE=“/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log”
export LOGFILE

FLAGS=“--bwlimit 8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt --log-file=$LOGFILE -v -P -rc”
export FLAGS

start=$(date +’%d-%H-%M-%s’)

echo $(date +%Y-%m-%d-%H%M) RCLONE SYNC STARTED | tee -a $LOGFILE

/usr/sbin/rclone sync /volumes/VoiceOvers VoiceOvers: $FLAGS	

/usr/sbin/rclone sync /volumes/07_MEDIA 07_MEDIA: $FLAGS

echo "$(date +%Y-%m-%d-%H%M RCLONE SYNC FINISHED IN $($(date +’%d-%H-%M-%s’)- $start))”$ | tee -a $LOGFILE

exit

New terminal errors

/Users/nahb/Desktop/Rclone/master_sync ; exit;
archivemanager:~ nahb$ /Users/nahb/Desktop/Rclone/master_sync ; exit;
/Users/nahb/Desktop/Rclone/master_sync: line 2: pidof: command not found
/Users/nahb/Desktop/Rclone/master_sync: line 10: 8.65M: command not found
tee: “/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log”: No such file or directory
2020-07-16-1547 RCLONE SYNC STARTED
/Users/nahb/Desktop/Rclone/master_sync: line 17: /usr/sbin/rclone: No such file or directory
/Users/nahb/Desktop/Rclone/master_sync: line 19: /usr/sbin/rclone: No such file or directory
/Users/nahb/Desktop/Rclone/master_sync: line 21: unexpected EOF while looking for matching `"'
/Users/nahb/Desktop/Rclone/master_sync: line 24: syntax error: unexpected end of file
logout
Saving session...
...copying shared history...
...saving history...truncating history files...
...completed.

[Process completed]

smercer · July 16, 2020, 9:28pm

Try as I might, I just can't seem to get this working. I ditched the pidof command since this is only running weekly.

#!/bin/bash

RCLONE_CONFIG=/Users/nahb/.config/rclone/rclone.conf.    	
export RCLONE_CONFIG

LOGFILE=/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log
export LOGFILE

FLAGS=“—-bwlimit 8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt —-log-file=$LOGFILE -v -P -rc”
export FLAGS

start=$(date +’%d-%H-%M-%s’)

echo $(date +%Y-%m-%d-%H%M) RCLONE SYNC STARTED | tee -a $LOGFILE

/usr/local/bin/rclone sync /volumes/VoiceOvers VoiceOvers: $FLAGS	

/usr/local/bin/rclone sync /volumes/07_MEDIA 07_MEDIA: $FLAGS

echo "$(date +%Y-%m-%d-%H-%M RCLONE SYNC FINISHED IN $($(date +’%d-%H-%M-%s’)- $start))"$ | tee -a $LOGFILE

exit

What terminal spits back at me

archivemanager:~ nahb$ /Users/nahb/Desktop/Rclone/master_sync.sh 
/Users/nahb/Desktop/Rclone/master_sync.sh: line 9: 8.65M: command not found
2020-07-16-1726 RCLONE SYNC STARTED
2020/07/16 17:26:22 NOTICE: Config file "/Users/nahb/.config/rclone/rclone.conf." not found - using defaults
2020/07/16 17:26:22 Failed to create file system for "VoiceOvers:": didn't find section in config file
2020/07/16 17:26:22 NOTICE: Config file "/Users/nahb/.config/rclone/rclone.conf." not found - using defaults
2020/07/16 17:26:22 Failed to create file system for "07_MEDIA:": didn't find section in config file
/Users/nahb/Desktop/Rclone/master_sync.sh: line 20: ’16-17-26-1594934782’-: command not found
date: illegal time format
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ... 
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
$

asdffdsa · July 16, 2020, 9:42pm

you need to use the correct amount of - hyphens for each flag
https://rclone.org/docs/#remote-control
--rc, not -rc

export FLAGS="--bwlimit=8.65M --exclude-from=/Users/nahb/Desktop/Rclone/rclone_exclude_files.txt --log-file=$LOGFILE -v -P --rc"

asdffdsa · July 16, 2020, 10:19pm

rclone cannot find your config file

asdffdsa · July 16, 2020, 10:38pm

by your own adminssion, "Still very much a newbie with rclone, bash, and Cron "
so i suggest that you create the simplest script to get rclone working.
and then add fluff like all those dates and tees

so, for now, remove

start=$(date +’%d-%H-%M-%s’)
echo $(date +%Y-%m-%d-%H%M) RCLONE SYNC STARTED | tee -a $LOGFILE
echo "$(date +%Y-%m-%d-%H-%M RCLONE SYNC FINISHED IN $($(date +’%d-%H-%M-%s’)- $start))"$ | tee -a $LOGFILE

Lex · July 17, 2020, 3:25am

^^ that

If you simply want to get it running and are okay to learn your bash scripting later/in parallel you can try this:

crontab -e to open your crontab editor

paste the following

@weekly rclone sync /volumes/VoiceOvers VoiceOvers: —-bwlimit 8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt —-log-file=/Users/nahb/Desktop/Rclone/Reports/Master_Sync.log -v -P -rc

Change @weekly to whatever frequency / start time you like. Check https://crontab.guru/

If you want to limit how long the cron runs then add timeout e.g. @weekly timeout 12h rclone blah blah ..

Copy paste that line with your second source-dest pair.

Then play with bash scripts without pressure. There are great tutorials on google/youtube

smercer · July 17, 2020, 6:36pm

So I tore the code apart and added each element back in one at a time. Currently this basic form is working, but the log file doesn't include the command or remote info, making it hard to keep track of where I get errors.

The code

#!/bin/bash

RCLONE_CONFIG=/Users/nahb/.config/rclone/rclone.conf

FLAGS="-P --bwlimit=8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt -v --log-file=/Users/nahb/Desktop/Rclone/Reports/Master_Sync_2.log"

/usr/local/bin/rclone sync /volumes/VoiceOvers VoiceOvers: $FLAGS

/usr/local/bin/rclone sync /volumes/07_MEDIA 07_MEDIA: $FLAGS

exit

The log file

2020/07/17 14:22:33 INFO  : Starting bandwidth limiter at 8.650MBytes/s
2020/07/17 14:23:25 INFO  : There was nothing to transfer
2020/07/17 14:23:25 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              2259 / 2259, 100%
Elapsed time:         0.0s

2020/07/17 14:23:25 INFO  : Starting bandwidth limiter at 8.650MBytes/s
2020/07/17 14:24:12 INFO  : There was nothing to transfer
2020/07/17 14:24:12 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              1563 / 1563, 100%
Elapsed time:         0.0s

Is it possible to get something like this (without resorting to the -vv flag)?

**rclone sync /volumes/VoiceOvers VoiceOvers:**
2020/07/17 14:22:33 INFO  : Starting bandwidth limiter at 8.650MBytes/s
2020/07/17 14:23:25 INFO  : There was nothing to transfer
2020/07/17 14:23:25 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              2259 / 2259, 100%
Elapsed time:         0.0s

**rclone sync /volumes/07_MEDIA 07_MEDIA:**
2020/07/17 14:23:25 INFO  : Starting bandwidth limiter at 8.650MBytes/s
2020/07/17 14:24:12 INFO  : There was nothing to transfer
2020/07/17 14:24:12 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:              1563 / 1563, 100%
Elapsed time:         0.0s

smercer · July 18, 2020, 3:40pm

There's only one thing I'd still like to figure out: how do you tell the script to verify its not already running before it executes the command? If more than ~5.5TB is added to the storage, this script will still be running by the time it starts running again with my weekly cron schedule. Not exactly sure what'll happen, but I think it might be better to prevent that scenario.

The working code (without "already running?" protection) (and with basic descriptors to help other novices like me in the future)

#!/bin/bash

SECONDS=0
duration=$SECONDS
# used rclone config file to find this location
RCLONE_CONFIG=/Users/nahb/.config/rclone/rclone.conf
# creates a new log file Master_Sync_CURRENT DATE AND TIME on each run
LOGFILE="/Users/nahb/Desktop/Rclone/Reports/Master_Sync_$(date +%Y-%m-%d_%H-%M).log"
# Prevents upload limit exceeded errors, excludes hidden files from a custom list, gives verbose info to the logfile
FLAGS="-P --bwlimit=8.65M --exclude-from /Users/nahb/Desktop/Rclone/rclone_exclude_files.txt -v --log-file=$LOGFILE"
# Establish sync commands as variables to reduce clutter and make code expansion easier, used application called "easy find" to search for the rclone hidden file and discover the "absolute path" for use with cron
cmd1="/usr/local/bin/rclone sync /volumes/VoiceOvers VoiceOvers:"
cmd2="/usr/local/bin/rclone sync /volumes/07_MEDIA 07_MEDIA:"
# displays message in terminal that the first command has started and details that command (without the flags) in the log file
echo $(date +%Y-%m-%d_%H:%M) $cmd1 STARTED | tee -a $LOGFILE
# Runs the first command with flags
$cmd1 $FLAGS
echo $(date +%Y-%m-%d_%H:%M) $cmd2 STARTED | tee -a $LOGFILE
$cmd2 $FLAGS
# displays in terminal the operation has completed, displays how long it took and makes a note in the log file
echo "$(date +%Y-%m-%d_%H:%M) RCLONE SYNC FINISHED IN $(($duration / 3600)) HOURS AND $(($duration / 60)) MINUTES" | tee -a $LOGFILE

exit

The cron job code

# run the above code at midnight on Saturday, every Saturday, and don't send an automatic email
0 0 * * 6 /Users/nahb/Desktop/Rclone/master_sync.sh >/dev/null 2>&1

Special thanks to both @Lex and @asdffdsa for getting me to slow down, start simple and work my way up to a working code. I really appreciate it!

asdffdsa · July 18, 2020, 3:44pm

glad to help with the rclone issues.

as for cron,
https://www.google.com/search?client=firefox-b-1-d&q=cron+prevent+multiple+jobs

system · September 17, 2020, 11:44am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.