What is the problem you are having with rclone?
am not sure if this is related directly to rclone, the script, linux etc but i have to start somewhere. the issue is that am using a custom bash script to automatically/ periodically upload/ move files from a given folder to gdrive using 4 SA. the problem is that after a while the script will stop uploading despite having stuff to be uploaded still and as result the local storage run out of space.
Run the command 'rclone version' and share the full output of the command.
\rclone v1.59.0-beta.6078.bab91e440
- os/version: ubuntu 18.04 (64 bit)
- os/kernel: 4.15.0-175-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.18.1
- go/linking: static
- go/tags: none
Are you on the latest version of rclone?
no
Which cloud storage system are you using? (eg Google Drive)
Google Driv
The command you were trying to run (eg rclone copy /tmp remote:tmp)
(current script revision that a fried wrote for me from another friend custom script as idk bash scripting at the moment. i believe they all base from animosity022 script, all credits goes to the pp involved)
#!/bin/bash
# WARNING THIS IS CURRENTLY BEING TESTED
# ===================================
# CONFIGURATION
# ===================================
# Custom config
# Leave empty to use default
RCLONE_CONFIG=""
# SOURCE PATH, UPLOAD FROM
# local path only, Must not be fuse mount path
PATH_FROM="/media/mergerfs/local/media/"
# DESTINATION PATH, UPLOAD TO
# rclone_mount:folder
PATH_TO="gcrypt:/media/"
# FULL LOG PATH
RCLONE_LOG="/media/logs/uploadmedia.log"
# SERVICE ACCOUNTS
SERVICE_ACC=(
"/docker/scripts/sa/bypass1.json"
"/docker/scripts/sa/bypass2.json"
"/docker/scripts/sa/bypass3.json"
"/docker/scripts/sa/bypass4.json"
)
# RCLONE TEST RUN
# Enable/Disable rclone dry run
# true = Enabled, false = Disabled
RCLONE_TEST=false
# IGNORE BASH VERSION
# If your version is not the same as the
# tested version, then the script will exit
# true = Don't exit, false = Exit if versions dont match
IGNORE_VERSION=true
# ===================================
# FUNCTIONS
# ===================================
# Minimum required age of files in order to upload
min_file_age() {
echo "$(date "+%Y/%m/%d %T") SCRIPT: Looking for files older than 1 minute." | tee -a $RCLONE_LOG
if find "$PATH_FROM" -type f -mmin +5 | read; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: Found files that match criteria" | tee -a $RCLONE_LOG
return 0
fi
echo "$(date "+%Y/%m/%d %T") SCRIPT: No files matching criteria" | tee -a $RCLONE_LOG
return 1
}
# Rotates logs based on rize
rotate_log() {
MAX_SIZE=1000000 # 1MB
if [[ $(stat -c%s "$RCLONE_LOG") -ge $MAX_SIZE ]]; then
GET_NAME="$(basename -- "$RCLONE_LOG")"
PATH_NOEXT="$(dirname -- "$RCLONE_LOG")/${GET_NAME%.*}"
FILE_EXT="${GET_NAME##*.}"
REMOVE_OLDEST=true
for i in {2..1}; do
if [[ -f "$PATH_NOEXT-$i.$FILE_EXT" ]]; then
if [[ "$REMOVE_OLDEST" == true ]]; then
rm -f "$PATH_NOEXT-$i.$FILE_EXT"
else
mv "$PATH_NOEXT-$i.$FILE_EXT" "$PATH_NOEXT-$((i+1)).$FILE_EXT"
fi
fi
REMOVE_OLDEST=false
done
mv "$RCLONE_LOG" "$PATH_NOEXT-1.$FILE_EXT"
fi
}
# ===================================
# MAIN CODE
# ===================================
# flock ensures 1 instance runs at a time.
FLOCK_KEY="/var/lock/$(basename $0 .sh)"
(
# if already running then exit
flock -x -w 5 200 || { echo "script is already running"; exit 1; }
# Rotate logs
rotate_log
(
echo "YYYY/MM/DD HH:MM:SS"
echo "-------------------------------------------------------"
echo "$(date "+%Y/%m/%d %T") SCRIPT: RCLONE UPLOAD SCRIPT"
echo " Script Version: 2.1"
echo " Tested Linux Version: 21.10"
echo " Bash Version: 5.1.8(1)-release"
echo " Your Bash Version: ${BASH_VERSION}"
echo "-------------------------------------------------------"
) | tee -a $RCLONE_LOG
# Safey checks
IS_EXIT=false
if [[ "${BASH_VERSINFO:-0}" -lt 5 ]] && ! $IGNORE_VERSION; then
(
echo "$(date "+%Y/%m/%d %T") SCRIPT: WARNING - Your bash version hasn't been tested"
echo " Set IGNORE_VERSION=true if you wish to continue"
echo ""
) | tee -a $RCLONE_LOG
IS_EXIT=true
fi
if ! which flock > /dev/null 2>&1; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: ERROR - unable to find 'flock' command."
IS_EXIT=true
fi
# If Source path is fuse mount, log and exit
if /bin/findmnt $PATH_FROM -o FSTYPE -n | grep fuse; then
(
echo "$(date "+%Y/%m/%d %T") SCRIPT: ERROR - Path not local '$PATH_FROM'"
) | tee -a $RCLONE_LOG
IS_EXIT=true
fi
if [[ "$IS_EXIT" == true ]]; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.."
exit 1
fi
# Find files with future date and fix them.
echo "$(date "+%Y/%m/%d %T") SCRIPT: Checking files for future modified dates" | tee -a $RCLONE_LOG
find $PATH_FROM -type f -mtime -0 | while IFS= read p; do
i=$(($i+1))
if [[ -z $FOUND_FILES ]]; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: Found" | tee -a $RCLONE_LOG
FOUND_FILES="onetime"
fi
RUN_TOUCH="$(touch "$p" 2>&1)"
if [[ $? -ne 0 ]]; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: $(echo "$RUN_TOUCH" | grep -io "cannot touch '.*': Permission denied")" | tee -a $RCLONE_LOG
fi
done
# Only run rclone if file age is greater than 1 minute
if min_file_age; then
echo "$(date "+%Y/%m/%d %T") SCRIPT: Checking rclone config" | tee -a $RCLONE_LOG
# If we're using custom rclone config path
if [[ -n $RCLONE_CONFIG ]]; then
# Check config file exists
if [[ -f $RCLONE_CONFIG ]]; then
export RCLONE_CONFIG
echo "$(date "+%Y/%m/%d %T") SCRIPT: Rclone using config file from custom path '$RCLONE_CONFIG'" | tee -a $RCLONE_LOG
else
echo "$(date "+%Y/%m/%d %T") SCRIPT: Rclone config not found in custom path '$RCLONE_CONFIG'" | tee -a $RCLONE_LOG
fi
fi
# Verify rclone can use config
RCLONE_GREP="$(echo "$(rclone config show 2>&1)" | grep -o 'Config file ".*" not found')"
if [[ -n "$RCLONE_GREP" ]]; then
(
echo "$(date "+%Y/%m/%d %T") RCLONE: $RCLONE_GREP"
echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.."
)| tee -a $RCLONE_LOG
exit 1
fi
(
echo "$(date "+%Y/%m/%d %T") SCRIPT: Calling Rclone move"
echo ""
if [[ "$RCLONE_TEST" == true ]]; then
echo "$(date "+%Y/%m/%d %T") RCLONE: Dry run enabled, no action will be taken"
fi
) | tee -a $RCLONE_LOG
# Loop through service accounts
for ACCOUNT in ${SERVICE_ACC[@]}; do
# Log what Service Account we're using
echo "$(date "+%Y/%m/%d %T") RCLONE: Using SA '$ACCOUNT'" | tee -a $RCLONE_LOG
# Rclone arguments
RCLONE_ARG=(
"move" "-vP"
"$PATH_FROM" "$PATH_TO"
"--drive-service-account-file" "$ACCOUNT"
"--exclude" "*UNPACK*/**"
"--delete-empty-src-dirs"
"--fast-list"
"--max-transfer" "720G"
"--min-age" "1m"
"--drive-chunk-size" "128M"
"--tpslimit" "12"
"--tpslimit-burst" "12"
"--transfers" "6"
"--log-file" "$RCLONE_LOG"
)
# Include dry run if enabled
if [[ "$RCLONE_TEST" == true ]]; then
RCLONE_ARG+=("--dry-run")
fi
# Timer
TIMESTAMP=$(date +'%s')
# Run rclone with arguments
rclone ${RCLONE_ARG[@]}
echo "$(date "+%Y/%m/%d %T") RCLONE: Upload finnished in $(($(date +'%s')-$TIMESTAMP)) SECONDS" | tee -a $RCLONE_LOG
echo "" | tee -a $RCLONE_LOG
# Exit loop if no more files to upload
if ! min_file_age; then
break
fi
done
fi
echo "$(date "+%Y/%m/%d %T") SCRIPT: Exiting.." | tee -a $RCLONE_LOG
) 200>$FLOCK_KEY
# REF
# https://www.tothenew.com/blog/foolproof-your-bash-script-some-best-practices/
# https://stackoverflow.com/questions/1715137/what-is-the-best-way-to-ensure-only-one-instance-of-a-bash-script-is-running
The rclone config contents with secrets removed.
[gcrypt]
type = crypt
remote = gdrive:/media
password = dCiqhRASCQuDVZ2GHIPY83fmflF1-gI-Tpuso_NQ
password2 = qy-wf63ERxqGsn2Mf3YalxM5o0UQ6xs9qyrxFYcG
[gdrive]
type = drive
client_id = 983333297007-aj3kjafal5428l8q0pb6p2pdul9f9jiadljf.apps.googleusercontent.com
client_secret = 7bYg_B1AaaMLRFY2X3WwHafp2yG
scope = drive
token = {"access_token":"ya29.A0ARafrdaM-KPogdrr7q-mBkadXsirItbgUU9JTja1-IDY9BI_AGeFblQ6adNevN6eA1fR3bl_7RvH3P00aWgzvzUXRAJSQqq63tu2N88uQGufx59yookKgYpX-0MbE11vUP19wadsWJ3z7LKafrOeHxVjZfoEjXXxgWpy_WMIA","token_type":"Bearer","refresh_token":"1//09VuG439V9YwuCgYIARAAadGAkSNwF-L9Ir4-lkosAX0C0N03MxYoIEChxrChW2ndNCEhK23_RPsvvFTZ-VopFrFZCWX8qf4o7ok6A","expiry":"2022-05-08T18:05:38.973187035+02:00"}
team_drive = 0AErGafVdSouAwCUadk9PVA
A log from the command with the -vv flag
not sure how to do this but here are some logs: