rClone stopping upload for a couple hours every 1.xTB

Aerya · December 6, 2020, 11:57am

Hi,

What is the problem you are having with rclone?

It happens every 1.3-1.5TB moved using rClone script.
It uploads about 1.3Tb, 1 hour later it checks for duplicates and ignores them and then doesn't do anything for about 2 more hours. However I have files to send.
There's no error, it just seems to stop and wait.
This script is launched using cron:

- - - - /home/aerya/outils/scripts/move.sh

What is your rclone version (output from `rclone version`)

rclone v1.54.0-beta.4833.5164c3d2d

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.x

Which cloud storage system are you using? (eg Google Drive)

Google Drive, not Teamdrive as it's limited to 400k files.

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

#!/bin/bash
LOG="/home/aerya/outils/logs/rclone.log"

if pidof -o %PPID -x "$0"; then
   exit 1
fi

sleep 30
while true
do
  echo "$(date "+%d.%m.%Y %T") RCLONE UPLOAD STARTED" | tee -a "$LOG"

rclone move -c -v -P --min-age 5m --fast-list --bwlimit 8.3M --transfers=3 --checkers=3 --stats 5s --log-file="$LOG" /home/aerya/mnt/move/ gdrivecrypt:/
sleep 480

find "/home/aerya/mnt/move/" -mindepth 2 -type d -empty -delete
  echo "$(date "+%d.%m.%Y %T") RCLONE UPLOAD ENDED" | tee -a "$LOG"
done

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = a
client_secret = a
scope = drive
token = a

[gdrivecrypt]
type = crypt
remote = gdrive:
filename_encryption = standard
directory_name_encryption = true
password = a
password2 = a

A log from the command with the `-vv` flag

2020/12/06 09:25:34 INFO  : 
Transferred:   	    1.308T / 1.308 TBytes, 100%, 8.049 MBytes/s, ETA 0s
Checks:             56244 / 56244, 100%
Deleted:            28122
Renamed:            28122
Transferred:        28122 / 28122, 100%
Elapsed time:   48h2m33.4s

06.12.2020 09:33:35 RCLONE UPLOAD ENDED
06.12.2020 09:33:35 RCLONE UPLOAD STARTED
2020/12/06 09:33:36 INFO  : Starting bandwidth limiter at 8.300MBytes/s
2020/12/06 10:22:22 NOTICE: a: Duplicate directory found in destination - ignoring
2020/12/06 10:22:22 NOTICE: a: Duplicate directory found in destination - ignoring
2020/12/06 10:22:22 NOTICE: a: Duplicate directory found in destination - ignoring
2020/12/06 12:55:16 INFO  : a: Copied (new)
2020/12/06 12:55:19 INFO  : a: Deleted

Morphy · December 6, 2020, 6:42pm

Upload quota exceeded maybe since Google max upload pr. day is 750G - using scripts to bypass the upload?

Aerya · December 7, 2020, 7:52am

Nope, as said and pasted, there's no error and my bandwidth is limited to 8.3MBps.

ncw · December 7, 2020, 11:08am

Can you add a log with -vv please?

Also you should fix the duplicate directories - that has the potential to confuse rclone - see rclone dedup.

Aerya · December 7, 2020, 1:30pm

Thanks, i'll get back to you with more logs in a couple of days then.

Aerya · December 8, 2020, 8:23am

Here we go:


07.12.2020 14:54:31 RCLONE UPLOAD STARTED
2020/12/07 14:54:31 DEBUG : --min-age 5m0s to 2020-12-07 14:49:31.387402531 +0100 CET m=-299.910638195
2020/12/07 14:54:31 DEBUG : rclone: Version "v1.54.0-beta.4833.5164c3d2d" starting with parameters ["rclone" "move" "-c" "-vv" "-P" "--min-age" "5m" "--fast-list" "--bwlimit" "8.3M" "--transfers=3" "--checkers=3" "--stats" "5s" "--log-file=/home/aerya/outils/logs/rclone.log" "/home/aerya/mnt/move/" "gdrivecrypt:/"]
2020/12/07 14:54:31 DEBUG : Creating backend with remote "/home/aerya/mnt/move/"
2020/12/07 14:54:31 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/12/07 14:54:31 INFO  : Starting bandwidth limiter at 8.300MBytes/s
2020/12/07 14:54:31 DEBUG : Creating backend with remote "gdrivecrypt:/"
2020/12/07 14:54:31 DEBUG : Creating backend with remote "gdrive:"
2020/12/07 14:54:32 DEBUG : Google drive root '': root_folder_id = "a" - save this in the config to speed up startup
2020/12/07 14:54:37 DEBUG : Enc: Skipping undecryptable dir name: illegal base32 data at input byte 3
2020/12/07 14:59:00 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (16) returned no entries
2020/12/07 14:59:00 DEBUG : Google drive root '': Recycled 16 entries
2020/12/07 14:59:04 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
2020/12/07 15:12:09 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (3) returned no entries
2020/12/07 15:12:09 DEBUG : Google drive root '': Recycled 3 entries
2020/12/07 15:12:10 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
2020/12/07 15:17:15 DEBUG : gdrive: Loaded fresh token from config file
2020/12/07 15:17:15 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2020/12/07 15:17:15 DEBUG : gdrive: Saved new token in config file
2020/12/07 16:07:08 DEBUG : Google drive root '': Disabling ListR to work around bug in drive as multi listing (21) returned no entries
2020/12/07 16:07:08 DEBUG : Google drive root '': Recycled 21 entries
2020/12/07 16:07:13 DEBUG : Google drive root '': Re-enabling ListR as previous detection was in error
2020/12/07 16:14:06 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 0 length 8388608
2020/12/07 16:14:06 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 0 length 8388608
2020/12/07 16:14:06 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 0 length 8388608
2020/12/07 16:14:10 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 8388608 length 8388608
2020/12/07 16:14:10 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 8388608 length 8388608
2020/12/07 16:14:10 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 8388608 length 8388608
2020/12/07 16:14:12 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 16777216 length 8388608
2020/12/07 16:14:13 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 16777216 length 8388608
2020/12/07 16:14:13 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 16777216 length 8388608
2020/12/07 16:14:15 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25165824 length 8388608
2020/12/07 16:14:16 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 25165824 length 8388608
2020/12/07 16:14:16 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25165824 length 8388608
2020/12/07 16:14:18 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 33554432 length 8388608
2020/12/07 16:14:19 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 33554432 length 8388608
2020/12/07 16:14:19 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 33554432 length 8388608
2020/12/07 16:14:20 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 41943040 length 8388608
2020/12/07 16:14:22 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 41943040 length 8388608

...

2020/12/07 21:10:42 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 22087204864 length 8388608
2020/12/07 21:10:43 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 23681040384 length 8388608
2020/12/07 21:10:43 INFO  : a: Copied (new)
2020/12/07 21:10:43 INFO  : a: Deleted
2020/12/07 21:10:43 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 23706206208 length 8388608
2020/12/07 21:10:44 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 21952987136 length 8388608
2020/12/07 21:10:44 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 0 length 8388608
2020/12/07 21:10:44 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 22196256768 length 8388608

...

2020/12/07 21:16:15 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 22917677056 length 8388608
2020/12/07 21:16:16 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 24654118912 length 8388608
2020/12/07 21:16:17 DEBUG : jbgqc12ulddnv0bml891o8c38qv3o97tg5bh2rnihuqj0q0g94rbeg15fu87r91tged7q4oqfspooe454ik3g1vqe2dftpqm071idq8: Sending chunk 23160946688 length 8388608
2020/12/07 21:16:17 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020/12/07 21:16:17 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 964689920 length 8388608
2020/12/07 21:16:17 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020/12/07 21:16:18 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 23060283392 length 8388608
2020/12/07 21:16:18 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2020/12/07 21:16:18 DEBUG : gdrive: Saved new token in config file
2020/12/07 21:16:18 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 24679284736 length 8388608
2020/12/07 21:16:18 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 22926065664 length 8388608
2020/12/07 21:16:18 DEBUG : Keeping previous permissions for config file: -rw-r--r--
2020/12/07 21:16:18 DEBUG : gdrive: Saved new token in config file
2020/12/07 21:16:19 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 24662507520 length 8388608
2020/12/07 21:16:21 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 23068672000 length 8388608

...

2020/12/07 21:20:15 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1652555776 length 8388608
2020/12/07 21:20:15 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 23563599872 length 8388608
2020/12/07 21:20:16 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 23790092288 length 8388608
2020/12/07 21:20:16 INFO  : a: Copied (new)
2020/12/07 21:20:16 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25350373376 length 8388608
2020/12/07 21:20:17 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 23798480896 length 8388608

...

2020/12/07 21:28:53 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3154116608 length 8388608
2020/12/07 21:28:53 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 26851934208 length 8388608
2020/12/07 21:28:54 INFO  : a: Copied (new)
2020/12/07 21:28:54 INFO  : a: Deleted
2020/12/07 21:28:54 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25023217664 length 8388608
2020/12/07 21:28:55 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3162505216 length 8388608

...

2020/12/07 21:30:43 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3481272320 length 8388608
2020/12/07 21:30:43 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25635586048 length 8388608
2020/12/07 21:30:43 INFO  : a: Copied (new)
2020/12/07 21:30:43 INFO  : a: Deleted
2020/12/07 21:30:44 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25341984768 length 8388608
2020/12/07 21:30:44 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 318767104 length 8388608
2020/12/07 21:30:45 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3489660928 length 8388608
2020/12/07 21:30:45 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1845493760 length 8388608
2020/12/07 21:30:46 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25643974656 length 8388608
2020/12/07 21:30:46 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 327155712 length 8388608
2020/12/07 21:30:46 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3498049536 length 8388608
2020/12/07 21:30:47 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25350373376 length 8388608
2020/12/07 21:30:48 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1853882368 length 8388608
2020/12/07 21:30:48 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 335544320 length 8388608
2020/12/07 21:30:48 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3506438144 length 8388608
2020/12/07 21:30:48 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25652363264 length 8388608
2020/12/07 21:30:50 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25358761984 length 8388608
2020/12/07 21:30:50 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 343932928 length 8388608
2020/12/07 21:30:50 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3514826752 length 8388608
2020/12/07 21:30:50 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1862270976 length 8388608
2020/12/07 21:30:51 INFO  : a: Copied (new)
2020/12/07 21:30:51 INFO  : a: Deleted
2020/12/07 21:30:51 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25660751872 length 8388608
2020/12/07 21:30:52 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 352321536 length 8388608
2020/12/07 21:30:52 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3523215360 length 8388608
2020/12/07 21:30:53 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25367150592 length 8388608
2020/12/07 21:30:53 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1870659584 length 8388608
2020/12/07 21:30:54 INFO  : a: Copied (new)
2020/12/07 21:30:54 INFO  : a: Deleted
2020/12/07 21:30:54 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 360710144 length 8388608
2020/12/07 21:30:54 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25669140480 length 8388608
2020/12/07 21:30:54 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3531603968 length 8388608
2020/12/07 21:30:56 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 369098752 length 8388608
2020/12/07 21:30:56 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3539992576 length 8388608
2020/12/07 21:30:56 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25375539200 length 8388608
2020/12/07 21:30:56 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1879048192 length 8388608
2020/12/07 21:30:57 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25677529088 length 8388608
2020/12/07 21:30:58 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 377487360 length 8388608
2020/12/07 21:30:58 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3548381184 length 8388608
2020/12/07 21:30:59 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25383927808 length 8388608
2020/12/07 21:30:59 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1887436800 length 8388608
2020/12/07 21:30:59 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25685917696 length 8388608
2020/12/07 21:31:00 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 385875968 length 8388608
2020/12/07 21:31:00 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3556769792 length 8388608
2020/12/07 21:31:01 INFO  : a: Copied (new)
2020/12/07 21:31:01 INFO  : a: Deleted
2020/12/07 21:31:02 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 394264576 length 8388608
2020/12/07 21:31:02 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25392316416 length 8388608
2020/12/07 21:31:02 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3565158400 length 8388608
2020/12/07 21:31:02 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1895825408 length 8388608
2020/12/07 21:31:02 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25694306304 length 8388608
2020/12/07 21:31:03 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 402653184 length 8388608
2020/12/07 21:31:04 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3573547008 length 8388608
2020/12/07 21:31:04 INFO  : a: Copied (new)
2020/12/07 21:31:04 INFO  : a: Deleted
2020/12/07 21:31:05 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25400705024 length 8388608
2020/12/07 21:31:05 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1904214016 length 8388608
2020/12/07 21:31:05 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 411041792 length 8388608
2020/12/07 21:31:05 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25702694912 length 8388608
2020/12/07 21:31:06 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3581935616 length 8388608
2020/12/07 21:31:07 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25409093632 length 8388608
2020/12/07 21:31:07 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 419430400 length 8388608
2020/12/07 21:31:08 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3590324224 length 8388608
2020/12/07 21:31:08 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1912602624 length 8388608
2020/12/07 21:31:08 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25711083520 length 8388608
2020/12/07 21:31:09 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 427819008 length 8388608
2020/12/07 21:31:10 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3598712832 length 8388608
2020/12/07 21:31:10 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25417482240 length 8388608
2020/12/07 21:31:11 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 436207616 length 8388608
2020/12/07 21:31:11 INFO  : a: Copied (new)
2020/12/07 21:31:11 INFO  : a: Deleted
2020/12/07 21:31:11 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 1920991232 length 8388608
2020/12/07 21:31:11 DEBUG : atphpj0ig0iv2uafbtj8bj3h3igrbtbjsidpia2grins4g82p0j5fav6dd642s91u8fg6n18pqlgjq7qppg2qe3if13rds2ut16j5f4s09ri6uv4d4ginv07q26g3a0j: Sending chunk 25719472128 length 8388608
2020/12/07 21:31:12 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3607101440 length 8388608
2020/12/07 21:31:13 DEBUG : o0ek18se0nf0lgpgdd7s1ntaluosmahbmiss3hhtkkfma84cu4pc971j7jt085jqdoagc23n3ha17jskfulkiluqnojq2u53gngtt5o: Sending chunk 25425870848 length 8388608
2020/12/07 21:31:13 DEBUG : 3qnd8qup35pg8430bt07afseoej348nn63u2r6qfokcc5pvrsq9ur1ombst5d8m2da6obndpfvnr0q0fsucg0egp50dm7dl8rcuhc8g: Sending chunk 444596224 length 8388608
2020/12/07 21:31:14 DEBUG : kdposg28feomfssbb7p469slvs5j8236spcblccovberg2c88ah5u6ki7u78ef326cm4vda7v9oejm45k407ikedbqu0i5rqsgvfn3b11c6f8oj6gkrnjkqd467k0fmr: Sending chunk 3615490048 length 8388608
2020/12/07 21:31:14 INFO  : a: Copied (new)
2020/12/07 21:31:14 INFO  : a: Deleted

...

ncw · December 8, 2020, 4:30pm

I've checked that log quite carefully and there doesn't seem to be any problems I can see in it.

Files are being transferred.

The only thing is the file a being transferred and deleted a lot, but I'm guessing you changed the file names here?

What problems did you encounter in this upload?

Aerya · December 8, 2020, 4:45pm

Yeah I changed all keys/names with "a".
I guess this log might shows my problem: the script started at 3PM and the 1rst file was copied at 9PM.

ncw · December 8, 2020, 5:07pm

The log shows you are sending 3 large files simultaneously.

The first file is transferred at 21:16 after starting at 16:14 so about 5 hours transfer. It is about 23160946688 bytes.

However you have set the bandwidth limit to 8.300MBytes/s

So I'd expect a your file to take 23160946688/8.3E6 = 2790 seconds if it was the only file being transferred. You are transferring 3 files though so it will take 3 times as long, so about 2.3 Hours.

To speed this up you want to increase --drive-chunk-size

  --drive-chunk-size SizeSuffix       Upload chunk size. Must a power of 2 >= 256k. (default 8M)

Try using 64M or 256M - that should help a lot.

Aerya · December 8, 2020, 10:21pm

Ok, i'll give it a look then. Thanks for rClone and support!

system · February 7, 2021, 6:21pm

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.