What is the problem you are having with rclone?
TLDR: Logs are unreadable, am I doing something wrong?
First time using rclone. I configured Google workspace Enterprise with 2 users using the same Teams share. In the rclone.conf, both users have their own access to the google drive, but everything else is configured identically. I have two scripts for cron configured identically to the below script, except each script is excluding different folders (to avoid confusion) and the destination is the users's access to the google teams share; e.g.
User1account>Google Drive>Teams Share
User2account>Google Drive>Teams Share
In addition to the logging junk, please review my config for best practices, tips, or tricks. I run fiber gigabit internet and I am replicating my plex library as an offsite replication. Someday I may use mergerfs but its not needed right now.
Thank you in advance, I appreciate any help.
Run the command 'rclone version' and share the full output of the command.
Docker deployment on Ubuntu host
rclone v1.57.0
- os/version: alpine 3.14.2 (64 bit)
- os/kernel: 5.11.0-46-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
Google Drive - Team Share
The command you were trying to run (eg rclone copy /tmp remote:tmp)
log output
2022/01/18 17:21:37 DEBUG : vmn59c95nlt9hq8dfn9u3f2tu8/ugquhe791bqr3seu9bg0ubbevadjatr0ilcr490vr75nd6npn8nit7a422g661hhs2qqkde16mgmi/1kr9spj1ce975u8o7oonc7rv67lbjl79ig0hig6k81ta0q3fnih94obqd16vtuq8haq2phaku838rdguj1uo0lkedcc8lk2ra4a3n4b3iovq5uohb08iq2cjqftjutghc8bqmu9lphln3h6plckhd162la7etr1cfmptmh01jkl871soaicg: Sending chunk 3489660928 length 33554432
2022/01/18 17:21:39 DEBUG : vmn59c95nlt9hq8dfn9u3f2tu8/hsabmqqre9g7bcbsa8tr2v7l55me8le4ahl19m3ifmrnq6904s0g/452ijk2c59qjds8t5cttho9i9aaa7r4siler2epdre062lj6nf395l5uvrm25cmstq20nd5ahl0sotkmfnl4od7kms49b36j102ta5ikn5s3ilctibmuone5cc028b07plahm29ptuehj3l9ucmkrb3bd8: Sending chunk 3154116608 length 33554432
2022/01/18 17:21:41 DEBUG : vmn59c95nlt9hq8dfn9u3f2tu8/ugquhe791bqr3seu9bg0ubbevadjatr0ilcr490vr75nd6npn8nit7a422g661hhs2qqkde16mgmi/1kr9spj1ce975u8o7oonc7rv67lbjl79ig0hig6k81ta0q3fnih94obqd16vtuq8haq2phaku838rdguj1uo0lkedcc8lk2ra4a3n4b3iovq5uohb08iq2cjqftjutghc8bqmu9lphln3h6plckhd162la7etr1cfmptmh01jkl871soaicg: Sending chunk 3523215360 length 33554432
2022/01/18 17:21:43 DEBUG : vmn59c95nlt9hq8dfn9u3f2tu8/hsabmqqre9g7bcbsa8tr2v7l55me8le4ahl19m3ifmrnq6904s0g/452ijk2c59qjds8t5cttho9i9aaa7r4siler2epdre062lj6nf395l5uvrm25cmstq20nd5ahl0sotkmfnl4od7kms49b36j102ta5ikn5s3ilctibmuone5cc028b07plahm29ptuehj3l9ucmkrb3bd8: Sending chunk 3187671040 length 33554432
The rclone config contents with secrets removed.
#!/bin/sh
# RClone Config file
RCLONE_CONFIG=/config/rclone/rclone.conf
export RCLONE_CONFIG
# ------------------------------------------------------------------
# Rclone Sync script
# ------------------------------------------------------------------
# REQUIRED SETTINGS
ReplicationType="copy" # choose your replication type e.g. move, copy, sync
# copy - Copy files from source to dest, skipping already copied.
# sync - Make source and dest identical, modifying destination only.
# move - Move files from source to dest.
# check - Check if the files in the source and destination match.
Source="/media" # Source is local media folder
Destination="gteam-media-crypt:gmedia" # Destination is Google teams share
MinimumAge="30m" # sync files suffix ms|s|m|h|d|w|M|y
ModSort="ascending" # "ascending" oldest files first, "descending" newest files first
MaxTransfer="700G" # Maximum size of data to transfer (default off) - Google is 750
# LOGGING SETTINGS
CURRENT_DATE=$(date +%Y%m%d)
LOG_FILE="/logs/rclone-$ReplicationType-$CURRENT_DATE.log"
echo "Logging to $LOG_FILE"
# Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight.
BWLimit1Time="01:00" # can be listed as off or a number like 15M
BWLimit1="off"
BWLimit2Time="08:00"
BWLimit2="20M"
BWLimit3Time="16:00"
BWLimit3="off"
# OPTIONAL SETTINGS
# Add extra commands or filters
Command1="--exclude downloads/**"
Command2="--exclude homevideos/**"
Command3="--exclude music/**"
Command4="--exclude music-Old/**"
Command5="--exclude hallmark/**"
Command6="--exclude tvshowslocalonly/**"
Command7="--exclude tvshows4k/**"
Command8="--exclude tvshows/**"
Command9="--exclude movies/**"
Command10=""
# process files
rclone $ReplicationType $Source $Destination \
-vv \
--log-file $LOG_FILE \
--fast-list \
--progress \
--drive-chunk-size 32M \
--transfers 2 \
--order-by modtime,$ModSort \
--min-age $MinimumAge \
$Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 $Command9 $Command10 \
--exclude *fuse_hidden* \
--exclude *_HIDDEN \
--exclude .recycle** \
--exclude .Recycle.Bin/** \
--exclude *.backup~* \
--exclude *.partial~* \
--exclude *@eaDir \
--max-transfer $MaxTransfer \
--drive-stop-on-download-limit \
--drive-stop-on-upload-limit \
--bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \
# Give us the ability to see the error in our ways
chmod a+r $LOG_FILE
echo "Log file can be found at $LOG_FILE"
# Prune old logs
echo "Pruning old logs"
find /logs/ -name "*.log" -type f -mtime +30 -delete
The second user account's script is almost identical except for these deltas and the log location.
Destination="derinne-crypt:gmedia" # Destination is Google teams share
# Add extra commands or filters
Command1="--exclude downloads/**"
Command2="" #--exclude homevideos/**"
Command3="--exclude music/**"
Command4="--exclude music-Old/**"
Command5="" #--exclude hallmark/**"
Command6="" #--exclude tvshowslocalonly/**"
Command7="--exclude tvshows4k/**"
Command8="--exclude tvshows/**"
Command9="--exclude movies/**"
Command10="--exclude movies4k/**"
RCLONE.CONF
[gteam-media]
type = drive
client_id = [REMOVED]
client_secret = [REMOVED]
scope = drive
root_folder_id =
token = [REMOVED]
team_drive = [Random alphanumeric that represents the team share of gmedia]
[gteam-media-crypt]
type = crypt
remote = gteam-media:crypt
password = [REMOVED]
[derinne]
type = drive
client_id = [REMOVED]
client_secret = [REMOVED]
scope = drive
root_folder_id =
token = [REMOVED]
team_drive = [Random alphanumeric that represents the team share of gmedia]
[derinne-crypt]
type = crypt
remote = derinne:crypt
password = [REMOVED]
A log from the command with the -vv flag
I already had -vv set. Should I not have?