Dropbox Upload Using Bash

Help and Support
1

Hi
Im moving my files using rclone move, im using a bash script to run rclone move cmd, mean when source folder have any folder for upload bash run rclone move cmd,
My bash script is running since few days and it was working perfect until today evening rclone cmd was hang and there is no error log, My first question is about bash script, is it reliable to run rclone cmd or should have to move on python to run it automatically?
My second question is can i run multiple rclone cmd parallel?

Thanks

2

hello,
please answer the questions in the help and support template.
please post the bash script.

i use python to run rclone, i can share some code.

tho, if you have the correct rclone command, should not matter - bash or python.

3

please, no private messages if possible.

best to share in the public forum so other rcloners can comment.
and i am not a bash expert.

4 
#!/usr/bin/bash
#
# - config -
PATH_TO_INSPECT="/www/wwwroot/videos/rclone/tmp"
MINIMUM_SECONDS_SINCE_LAST_MODIFICATION=10
TEMP_LOCATION_BASE="/www/wwwroot/videos/rclone/move"
SLEEP_IN_SECONDS=20
# - end of config -
#############################################################################

move_and_rclone() {
    # get the last bit of the path
    BASE_NAME=$(basename $1)
    # calculate the date
    DATE=$(date +"%Y/%m/%d")
    # construct the output dir
    OUTPUT_DIR="$TEMP_LOCATION_BASE"
    # verify output dir existance
    mkdir -p $OUTPUT_DIR
    # move
    echo "Moving $1 to $OUTPUT_DIR/$BASE_NAME"
    mv $1 $OUTPUT_DIR/$BASE_NAME
    # rclone
    echo "rcloning $OUTPUT_DIR/$BASE_NAME"
    rclone move $OUTPUT_DIR/$BASE_NAME/ dropbox:web/box/videos/$DATE/$BASE_NAME/ --delete-empty-src-dirs --tpslimit 12 --tpslimit-burst 12 --transfers 64 --dropbox-batch-size 1000 --dropbox-batch-timeout 10s --checkers 32 --checksum --log-file /www/wwwroot/videos/rclone/rclone.log
};

export -f move_and_rclone
export TEMP_LOCATION_BASE

while /bin/true;
do
    # this command will find the directories old enough and calls themove_and_rclone() function
    find $PATH_TO_INSPECT -mindepth 1 -maxdepth 1 -type d -not -newermt "-$MINIMUM_SECONDS_SINCE_LAST_MODIFICATION seconds" -exec bash -c 'move_and_rclone "$0"' {} \;
    
    find /www/wwwroot/videos/rclone/move/* -type d -empty -delete
    
    # run rclone command
    echo "Sleeping for $SLEEP_IN_SECONDS seconds"
    sleep $SLEEP_IN_SECONDS
done

Here is my bash script

5

need to post the rclone debug log that shows the problem/error/issue

sure, possible, but maybe not practical.

6

running this bash script rclone was hang moving a folder and as i mention above there is no any error log, just it was hanged, after rerun it start working normally.
can you share python code for rclone move?

Thanks

7

well, need to use a debug log, add --log-level=DEBUG
else, no idea why sometimes the command works and sometimes it does not work

really, just a fancy function for subprocess.run
i use this function, whenever i need to run a command, such as 7z.exe, fastcopy.exe, rclone.exe, vshadow.exe and so on.
not sure how useful it will be to you.

    def DoCmd(self, Text, Cmd, Input='', Env=''):
        try:
            if Common.DebugMode:
                Common.Log(self, f'DebugMode: DoCmd: {Text}={Cmd}')
                return
            Common.Log(self, f'DoCmd: Start:   {Text}={Cmd}')
            the_env = {};   the_env = dict(os.environ)
            if Env != '':   the_env.update(Env)
            x=subprocess.run(Cmd, cwd=Common.ScriptsDir, capture_output=True, text=True, input=Input, env=the_env)
            if len(x.stdout) != 0:  xfile = open(f'{Common.BackupLogDir}\\{Text}.stdout.log', "w");   xfile.write(x.stdout); xfile.close()
            if len(x.stderr) != 0:  xfile = open(f'{Common.BackupLogDir}\\{Text}.stderr.log', "w");   xfile.write(x.stderr); xfile.close()
            if x.returncode == 0:
                Common.Log(self, f'DoCmd: End:   {Text}={Cmd}')
                return x
            else:
                Common.DoEnd(self, f'ERROR: DoCmd: {Text} - {Cmd} with returncode={x.returncode}')
        except Exception as exception:
            Common.DoEnd(self, f'ERROR: DoCmd: {Text} - {Cmd} - {traceback.format_exc()}')
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.