Script works but i have some stalling/ major issue

aniel3000 · May 8, 2022, 3:42pm

What is the problem you are having with rclone?

am not sure if this is related directly to rclone, the script, linux etc but i have to start somewhere. the issue is that am using a custom bash script to automatically/ periodically upload/ move files from a given folder to gdrive using 4 SA. the problem is that after a while the script will stop uploading despite having stuff to be uploaded still and as result the local storage run out of space.

Run the command 'rclone version' and share the full output of the command.

\rclone v1.59.0-beta.6078.bab91e440

os/version: ubuntu 18.04 (64 bit)
os/kernel: 4.15.0-175-generic (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.1
go/linking: static
go/tags: none

Are you on the latest version of rclone?
no

Which cloud storage system are you using? (eg Google Drive)

Google Driv

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

(current script revision that a fried wrote for me from another friend custom script as idk bash scripting at the moment. i believe they all base from animosity022 script, all credits goes to the pp involved)

#!/bin/bash
# WARNING THIS IS CURRENTLY BEING TESTED
# ===================================
#          CONFIGURATION
# ===================================
# Custom config
# Leave empty to use default
RCLONE_CONFIG=""

# SOURCE PATH, UPLOAD FROM
# local path only, Must not be fuse mount path
PATH_FROM="/media/mergerfs/local/media/"

# DESTINATION PATH, UPLOAD TO
# rclone_mount:folder
PATH_TO="gcrypt:/media/"

# FULL LOG PATH
RCLONE_LOG="/media/logs/uploadmedia.log"

# SERVICE ACCOUNTS
SERVICE_ACC=(
    "/docker/scripts/sa/bypass1.json"
    "/docker/scripts/sa/bypass2.json"
    "/docker/scripts/sa/bypass3.json"
    "/docker/scripts/sa/bypass4.json"
)

# RCLONE TEST RUN
# Enable/Disable rclone dry run
# true = Enabled,  false = Disabled
RCLONE_TEST=false

# IGNORE BASH VERSION
# If your version is not the same as the
# tested version, then the script will exit
#  true = Don't exit, false = Exit if versions dont match
IGNORE_VERSION=true

# ===================================
#            FUNCTIONS
# ===================================
# Minimum required age of files in order to upload
min_file_age() {
    echo "$(date "+%Y/%m/%d %T") SCRIPT: Looking for files older than 1 minute." | tee -a $RCLONE_LOG
    if find "$PATH_FROM" -type f -mmin +5 | read; then
        echo "$(date "+%Y/%m/%d %T") SCRIPT: Found files that match criteria" | tee -a $RCLONE_LOG
        return 0
    fi
    echo "$(date "+%Y/%m/%d %T") SCRIPT: No files matching criteria" | tee -a $RCLONE_LOG
    return 1
}
# Rotates logs based on rize
rotate_log() {
    MAX_SIZE=1000000 # 1MB

    if [[ $(stat -c%s "$RCLONE_LOG") -ge $MAX_SIZE ]]; then

        GET_NAME="$(basename -- "$RCLONE_LOG")"
        PATH_NOEXT="$(dirname -- "$RCLONE_LOG")/${GET_NAME%.*}"
        FILE_EXT="${GET_NAME##*.}"

        REMOVE_OLDEST=true
        
        for i in {2..1}; do
            if [[ -f "$PATH_NOEXT-$i.$FILE_EXT" ]]; then
                if [[ "$REMOVE_OLDEST" == true ]]; then
                    rm -f "$PATH_NOEXT-$i.$FILE_EXT"
                else
                    mv "$PATH_NOEXT-$i.$FILE_EXT" "$PATH_NOEXT-$((i+1)).$FILE_EXT"
                fi
            fi
            REMOVE_OLDEST=false
        done

        mv "$RCLONE_LOG" "$PATH_NOEXT-1.$FILE_EXT"

    fi
}
# ===================================
#           MAIN CODE
# ===================================
# flock ensures 1 instance runs at a time.
FLOCK_KEY="/var/lock/$(basename $0 .sh)"
(
    # if already running then exit
    flock -x -w 5 200 || { echo "script is already running"; exit 1; }
    
    # Rotate logs
    rotate_log

    (
        echo "YYYY/MM/DD HH:MM:SS"
        echo "-------------------------------------------------------"
        echo "$(date "+%Y/%m/%d %T") SCRIPT: RCLONE UPLOAD SCRIPT"
        echo "            Script Version: 2.1"
        echo "       Tested Linux Version: 21.10"
        echo "               Bash Version: 5.1.8(1)-release"
        echo "          Your Bash Version: ${BASH_VERSION}"
        echo "-------------------------------------------------------"
    ) | tee -a $RCLONE_LOG

    # Safey checks
    IS_EXIT=false
    if [[ "${BASH_VERSINFO:-0}" -lt 5 ]] && ! $IGNORE_VERSION; then
        (
            echo "$(date "+%Y/%m/%d %T") SCRIPT: WARNING - Your bash version hasn't been tested"
            echo "                            Set IGNORE_VERSION=true if you wish to continue"
            echo ""
        )  | tee -a $RCLONE_LOG
        IS_EXIT=true
    fi

    if ! which flock > /dev/null 2>&1; then
        echo "$(date "+%Y/%m/%d %T") SCRIPT: ERROR - unable to find 'flock' command."
        IS_EXIT=true
    fi

    # If Source path is fuse mount, log and exit
    if /bin/findmnt $PATH_FROM -o FSTYPE -n | grep fuse; then
        (
            echo "$(date "+%Y/%m/%d %T") SCRIPT: ERROR - Path not local '$PATH_FROM'"
        ) | tee -a $RCLONE_LOG
        IS_EXIT=true
    fi

    if [[ "$IS_EXIT" == true ]]; then
        echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.."
        exit 1
    fi

    # Find files with future date and fix them.
    echo "$(date "+%Y/%m/%d %T") SCRIPT: Checking files for future modified dates" | tee -a $RCLONE_LOG
    find $PATH_FROM -type f -mtime -0  | while IFS= read p; do
        i=$(($i+1))
        if [[ -z $FOUND_FILES ]]; then
            echo "$(date "+%Y/%m/%d %T") SCRIPT: Found" | tee -a $RCLONE_LOG
            FOUND_FILES="onetime"
        fi
        RUN_TOUCH="$(touch "$p" 2>&1)"
        if [[ $? -ne 0 ]]; then
            echo "$(date "+%Y/%m/%d %T") SCRIPT: $(echo "$RUN_TOUCH" | grep -io "cannot touch '.*': Permission denied")" | tee -a $RCLONE_LOG
        fi
    done

    # Only run rclone if file age is greater than 1 minute
    if min_file_age; then
        
        echo "$(date "+%Y/%m/%d %T") SCRIPT: Checking rclone config" | tee -a $RCLONE_LOG
        # If we're using custom rclone config path
        if [[ -n $RCLONE_CONFIG ]]; then
            # Check config file exists
            if [[ -f $RCLONE_CONFIG ]]; then
                export RCLONE_CONFIG
                echo "$(date "+%Y/%m/%d %T") SCRIPT: Rclone using config file from custom path '$RCLONE_CONFIG'" | tee -a $RCLONE_LOG
            else
                echo "$(date "+%Y/%m/%d %T") SCRIPT: Rclone config not found in custom path '$RCLONE_CONFIG'" | tee -a $RCLONE_LOG
            fi
        fi

        # Verify rclone can use config
        RCLONE_GREP="$(echo "$(rclone config show 2>&1)" | grep -o 'Config file ".*" not found')"
        if [[ -n "$RCLONE_GREP" ]]; then
            (
                echo "$(date "+%Y/%m/%d %T") RCLONE: $RCLONE_GREP"
                echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.."
            )| tee -a $RCLONE_LOG
            exit 1
        fi

        (
            echo "$(date "+%Y/%m/%d %T") SCRIPT: Calling Rclone move"
            echo ""
            if [[ "$RCLONE_TEST" == true ]]; then
                echo "$(date "+%Y/%m/%d %T") RCLONE: Dry run enabled, no action will be taken"
            fi
        )  | tee -a $RCLONE_LOG

        # Loop through service accounts
        for ACCOUNT in ${SERVICE_ACC[@]}; do
            # Log what Service Account we're using
            echo "$(date "+%Y/%m/%d %T") RCLONE: Using SA '$ACCOUNT'" | tee -a $RCLONE_LOG

            # Rclone arguments
            RCLONE_ARG=(
                "move" "-vP"
                "$PATH_FROM" "$PATH_TO"
                "--drive-service-account-file" "$ACCOUNT"

                "--exclude" "*UNPACK*/**"

                "--delete-empty-src-dirs"
                "--fast-list"
                "--max-transfer" "720G"
                
                "--min-age" "1m"
                "--drive-chunk-size" "128M"
                "--tpslimit" "12"
                "--tpslimit-burst" "12"
                "--transfers" "6"
                "--log-file" "$RCLONE_LOG"
            )

            # Include dry run if enabled
            if [[ "$RCLONE_TEST" == true ]]; then
                RCLONE_ARG+=("--dry-run")
            fi
            
            # Timer
            TIMESTAMP=$(date +'%s')

            # Run rclone with arguments
            rclone ${RCLONE_ARG[@]}
            
            echo "$(date "+%Y/%m/%d %T") RCLONE: Upload finnished in $(($(date +'%s')-$TIMESTAMP)) SECONDS" | tee -a $RCLONE_LOG
            echo "" | tee -a $RCLONE_LOG
            # Exit loop if no more files to upload
            if ! min_file_age; then
                break
            fi
        done
    fi
    
    echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.." | tee -a $RCLONE_LOG
) 200>$FLOCK_KEY

# REF
# https://www.tothenew.com/blog/foolproof-your-bash-script-some-best-practices/
# https://stackoverflow.com/questions/1715137/what-is-the-best-way-to-ensure-only-one-instance-of-a-bash-script-is-running

The rclone config contents with secrets removed.

[gcrypt]
type = crypt
remote = gdrive:/media
password = dCiqhRASCQuDVZ2GHIPY83fmflF1-gI-Tpuso_NQ
password2 = qy-wf63ERxqGsn2Mf3YalxM5o0UQ6xs9qyrxFYcG

[gdrive]
type = drive
client_id = 983333297007-aj3kjafal5428l8q0pb6p2pdul9f9jiadljf.apps.googleusercontent.com
client_secret = 7bYg_B1AaaMLRFY2X3WwHafp2yG
scope = drive
token = {"access_token":"ya29.A0ARafrdaM-KPogdrr7q-mBkadXsirItbgUU9JTja1-IDY9BI_AGeFblQ6adNevN6eA1fR3bl_7RvH3P00aWgzvzUXRAJSQqq63tu2N88uQGufx59yookKgYpX-0MbE11vUP19wadsWJ3z7LKafrOeHxVjZfoEjXXxgWpy_WMIA","token_type":"Bearer","refresh_token":"1//09VuG439V9YwuCgYIARAAadGAkSNwF-L9Ir4-lkosAX0C0N03MxYoIEChxrChW2ndNCEhK23_RPsvvFTZ-VopFrFZCWX8qf4o7ok6A","expiry":"2022-05-08T18:05:38.973187035+02:00"}
team_drive = 0AErGafVdSouAwCUadk9PVA

A log from the command with the `-vv` flag

not sure how to do this but here are some logs:

asdffdsa · May 8, 2022, 4:50pm

hi,

best to use https://pastebin.com, as it enable to view the text in raw format.

need to post a rclone debug log, not a log from your script.

post the output of $RCLONE_LOG

# FULL LOG PATH
RCLONE_LOG="/media/logs/uploadmedia.log"

aniel3000 · May 8, 2022, 9:20pm

pastebin didn't support this many characters that is why i had to use https://pastelink.net

that is literally what i posted however because i have everything on a stop there really no file processing logs

aniel3000 · May 18, 2022, 1:58pm

edit: with the help of a friend i was able to do some more debugging. I also forgot to mention in the main post that eventually stuff would get uploaded on its own (presumably once the daily quote get reseted) assuming i were able to catch the whole thing before running out of space, however i can also replicate that behavior by running script manually/ using a different/ non affected SA. I believe the script is getting stuck on affected SAs. my question is, is there a way to make sure rclone skips SAs that have quote limits in them so it doesn't get stuck or should i try using more SAs (am only using 4 at the moment)? i will be attaching some more logs in case it is of any help. I also tried changing this setting from 12 to 5 as recommended here ([User Rate Limit Exceeded] Is this error normal? Would it have any affect? - #16 by Animosity022) but same issue still happening.

"--tpslimit" "5"
"--tpslimit-burst" "5"
"--transfers" "5"

logs for when issue happens and when rclone recovers from it on its own: rclone logs – Google Drive

thank you in advance.

system · June 17, 2022, 1:58pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.