I'm trying to use rclone copy with --drive-stop-on-upload-limit but I see errors that it's an unknown flag with rclone v1.55.1 and Google Drive. Is that flag not supported with rclone copy? Fatal error: unknown flag: --drive-stop-on-upload-limit
The upload command is part of a script that works fine without the flag
find "${LOCAL_ROOT}.encrypt/${filename}" -exec rclone copy -c --drive-stop-on-upload-limit '{}' "${RCLONE_LABEL}:${RCLONE_SUBDIR}/${destpath}" \; 2>&1 || echo "$(date) ERROR! Failed to transfer file ${n}."
2021/05/21 13:16:53 DEBUG : 4 go routines active
Fri May 21 13:16:56 EDT 2021 - Uploading /data/local.decrypt/Software/GGL AS/Office16/Logs.txt to Google Drive.
2021/05/21 13:16:56 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2021/05/21 13:16:56 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "-c" "/data/local.encrypt/YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq/y7MiyHfhsRdl5nobe5,UbLmg" "gsuite://YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq" "--drive-stop-on-upload-limit" "-vv"]
2021/05/21 13:16:56 DEBUG : Creating backend with remote "/data/local.encrypt/YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq/y7MiyHfhsRdl5nobe5,UbLmg"
2021/05/21 13:16:56 DEBUG : fs cache: adding new entry for parent of "/data/local.encrypt/YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq/y7MiyHfhsRdl5nobe5,UbLmg", "/data/local.encrypt/YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq"
2021/05/21 13:16:56 DEBUG : Creating backend with remote "gsuite://YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq"
2021/05/21 13:16:56 DEBUG : gsuite: detected overridden config - adding "{+LSY5}" suffix to name
2021/05/21 13:16:57 DEBUG : fs cache: renaming cache item "gsuite://YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq" to be canonical "gsuite{+LSY5}:YOH3Ivv7sQSIn6EKGPRMhA,6/0qIVDSPum1ZWIf9ulBZsG-Og/7zWBQJvfO5wqvFMTfB9gdMdq"
2021/05/21 13:16:58 DEBUG : y7MiyHfhsRdl5nobe5,UbLmg: MD5 = aded441487280fb5e2f598551aea0994 OK
2021/05/21 13:16:58 DEBUG : y7MiyHfhsRdl5nobe5,UbLmg: Size and MD5 of src and dst objects identical
2021/05/21 13:16:58 DEBUG : y7MiyHfhsRdl5nobe5,UbLmg: Unchanged skipping
2021/05/21 13:16:58 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks: 1 / 1, 100%
Elapsed time: 1.5s
But when it gets to a file that isn't already uploaded, I still get the error 2021/05/21 14:10:12 Fatal error: unknown flag: --drive-stop-on-upload-limit
thanks. I'm not sure it's the file name escaping but a simple rclone copy command with the flag appears to work so it's not an issue with the binary. I'll try to get it sorted in the script.
rclone copy /etc/hosts gsuite: --drive-stop-on-upload-limit -vv
2021/05/21 14:54:32 DEBUG : Using config file from "/home/admin/.config/rclone/rclone.conf"
2021/05/21 14:54:32 DEBUG : rclone: Version "v1.55.1" starting with parameters ["rclone" "copy" "/etc/hosts" "gsuite:" "--drive-stop-on-upload-limit" "-vv"]
2021/05/21 14:54:32 DEBUG : Creating backend with remote "/etc/hosts"
2021/05/21 14:54:32 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2021/05/21 14:54:32 DEBUG : Creating backend with remote "gsuite:"
2021/05/21 14:54:32 DEBUG : gsuite: detected overridden config - adding "{+LSY5}" suffix to name
2021/05/21 14:54:32 DEBUG : fs cache: renaming cache item "gsuite:" to be canonical "gsuite{+LSY5}:"
2021/05/21 14:54:33 DEBUG : hosts: Need to transfer - File not found at Destination
2021/05/21 14:54:34 DEBUG : hosts: MD5 = 5a23d837f4d2993789ad43b867ba23c9 OK
2021/05/21 14:54:34 INFO : hosts: Copied (new)
2021/05/21 14:54:34 INFO :
Transferred: 222 / 222 Bytes, 100%, 138 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.9s
If you can get the debug log when it fails, you'll see the problem here. This shows the parsed command line and I'm guessing that the problem will show up here.
If there a command issue, it never makes it to a log file though as that only spits out to stderr I'd imagine:
felix@gemini:~$ rclone copy /etc/hosts GD: --log-file /tmp/blah.log --notaflag -vv --log-file test.log
Error: unknown flag: --notaflag
Usage:
rclone copy source:path dest:path [flags]
Flags:
--create-empty-src-dirs Create empty source dirs on destination after copy
-h, --help help for copy
Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
2021/05/21 17:22:28 Fatal error: unknown flag: --notaflagy
felix@gemini:~$ cat test.log
cat: test.log: No such file or directory
felix@gemini:~$
If the first line captured to a log, you can probably see what is messing it up.
It's interesting because the script works flawlessly with --bwlimit in place of --drive-stop-on-upload-limit.
The upload script I'm using is here:
$ cat bin/cloudupload
#!/bin/sh
##############################################################################
# cloudupload - mirror locally hosted media to cloud drive
##############################################################################
# If we're already running, just abort.
if pidof -o %PPID -x "$(basename "$0")" >/dev/null 2>&1 ; then
echo "$(basename "$0") already running. Aborted."
exit 3
fi
# Load configuration
. /etc/mediaserver.conf
export ENCFS6_CONFIG
# Find locally stored media
find "${LOCAL_ROOT}.decrypt" -type f |
while read -r n; do
# Ignore hidden or partial files.
case "$n" in
(*~) continue ;;
(*.QTFS) continue ;;
(*.fuse*) continue ;;
esac
# Encode file name and destination.
filename=$(encfsctl encode --extpass "echo $PASSWORD" "${LOCAL_ROOT}.encrypt" "$(echo "$n" | sed -e s@"${LOCAL_ROOT}.decrypt/"@@)")
destpath=$(dirname "${LOCAL_ROOT}.encrypt/${filename}" | sed -e s@"${LOCAL_ROOT}.encrypt"@@)
# Catch NULL file and path names and skip them.
if [ "$filename" = "" ] || [ "$destpath" = "" ]; then
echo "$(date) - ERROR: undefined source or destination."
continue
fi
# If file is in-use, wait 10 seconds and retry.
while [ "$(lsof 2>&1 | grep -c "$(echo "$filename" | sed s/[][]/./g)")" -gt "0" ] || \
[ "$(lsof 2>&1 | grep -c "$(echo "$n" | sed s/[][]/./g)")" -gt "0" ]; do
echo "$(date) - ${n} in use. Waiting 10 seconds before retry..."
sleep 10
done
# Upload to "the cloud."
echo "$(date) - Uploading $n to Google Drive."
find "${LOCAL_ROOT}.encrypt/${filename}" -exec rclone copy -c --bwlimit 10M '{}' "${RCLONE_LABEL}:${RCLONE_SUBDIR}/${destpath}" \; 2>&1 || echo "$(date) ERROR! Failed to transfer file ${n}."
done
When I replace --bwlimit 10M with --drive-stop-on-upload-limit is when I run into that error. I'll have to do some further debugging to figure out what's going on.
After deleting /usr/bin/rclone the script functions as intended with the new --drive-stop-on-upload-limit flag. Can't believe I didn't think of this earlier, but regardless, thank you very much for the help.