Rclone sync gdrive pacer: low level retry 10/10


#1

CMD line:
2018/12/31 09:00:09 DEBUG : rclone: Version "v1.45-050-g1c01d0b8-beta" starting with parameters ["/usr/bin/rclone" "sync" "dropbox:" "gdrive:" "--retries=5" "--transfers=6" "--checkers=6" "--stats" "60s" "--tpslimit" "3" "--drive-chunk-size" "128M" "--log-level" "DEBUG" "--log-file=/home/plex/logs/syncgdrive.log"]

I still get a lot of pacer: low level retry 10/10

2018/12/31 10:20:09 INFO  : 
Transferred:   	   12.898G / 12.898 GBytes, 100%, 2.751 MBytes/s, ETA 0s
Errors:                10 (retrying may help)
Checks:             11362 / 11362, 100%
Transferred:            4 / 4, 100%
Elapsed time:   1h20m0.5s

2018/12/31 09:06:05 ERROR : crypt/rijbp68ecdsepkm033j05hjoos/2ne792n2ein8fc50nh9dghf2eg/6ujr39a7dl6bgvegc8r6sdh6g4: error reading destination directory: couldn't list directory: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded

@ncw any way to get the actual file names during copy eg

  • crypt/hm2ka8or5qp1pe…4fbfjkjl0t6obos134lo: 64% /8.712G, 7.195M/s, 7m19s
    

I thought could do it with --crypt-show-mapping but its not working.


#2

I think your transfers/checkers are causing the issue. I’d move them down to more like 3/3 and if that works, move up to 4/4 or 5/5 to get no more than 10 total.

I’m not quite sure how the tpslimit plays with those two.


#3

Check out the issue that was opened up and if you’d like to test and provide feedback, that would be great!


#4

You can decode them with rclone cryptdecode


#5

Will test it, well with tpslimit 1 there never should be 403 pacer error, the problem is then its really slow.

I have 150TB+ of data eg 9K+ movies and around 50K TV shows and each have like 2 subtitles that almost 200K.

But I cant use it with sync eg to see decoded names in log file.


#6

This is interesting, much over the 750GB upload limit gdrive enforces
Command line:
/usr/bin/rclone sync dropbox: gdrive: --retries=5 --transfers=10 --checkers=3 --tpslimit=2 --drive-pacer-min-sleep=100ms --bwlimit=35M --drive-chunk-size=256M --log-level=INFO --log-file=$LogFile

2019/01/07 17:04:30 INFO  : 
Transferred:   	    3.745T / 4.292 TBytes, 87%, 33.292 MBytes/s, ETA 4h46m54s
Errors:                11 (retrying may help)
Checks:            140395 / 140395, 100%
Transferred:          165 / 216, 76%
Elapsed time:  32h46m0.4s
Transferring:
 *     crypt/0ibcad98vheuku…1niqjke6df4of3ap9oh0: 22% /50.926G, 3.316M/s, 3h22m42s
 *     crypt/0ibcad98vheuku…1v9bdn1loi4mfdt77ou2: 20% /48.405G, 3.238M/s, 3h21m35s
 *     crypt/0ibcad98vheuku…2dpfmbm5m77f5fmm7psk: 21% /56.085G, 3.497M/s, 3h36m8s
 *     crypt/0ibcad98vheuku…7lqjtei3kqgavdt2hkkg: 23% /69.469G, 3.653M/s, 4h9m23s
 *     crypt/0ibcad98vheuku…bku4sdjjnk61lubuelo8: 31% /47.144G, 3.590M/s, 2h32m26s
 *     crypt/0ibcad98vheuku…fdpc49eorf4ccmm80da8: 19% /52.072G, 3.404M/s, 3h29m27s
 *     crypt/0ibcad98vheuku…hsbem9en3oastv8gfcvo: 20% /49.175G, 3.195M/s, 3h27m44s
 *     crypt/0ibcad98vheuku…op0aap34r59ehtli4sgg: 25% /41.127G, 3.772M/s, 2h19m23s
 *     crypt/0ibcad98vheuku…rmh25clh54bq54316uqg: 19% /53.052G, 3.524M/s, 3h27m1s
 *     crypt/0ibcad98vheuku…tgfh873a192i7hboog08: 32% /48.363G, 3.750M/s, 2h29m11s

#7

Interesting… What do you think is causing that?


#8

Just for grins I tried with the exact parameters you have and was banned at ~750G for the day.


#9

It seems it was copying those big 50GB files over and over without the ban
posted findings and logs here https://github.com/ncw/rclone/issues/2880#issuecomment-452226385


#10

This works great now, good speed and no 403 errors & upload limit’s

CMD.
/usr/bin/rclone sync dropbox: gdrive: --retries=5 --transfers=4 --checkers=4 --tpslimit=2 --drive-pacer-min-sleep=100ms --bwlimit=9M --dropbox-chunk-size 150M --drive-chunk-size=512M --log-level=INFO --log-file=$LogFile

last log entry ( still going )

2019/01/22 15:31:12 INFO  :
Transferred:        8.211T / 8.840 TBytes, 93%, 8.968 MBytes/s, ETA 20h26m8s
Errors:                32 (retrying may help)
Checks:            227174 / 227174, 100%
Transferred:         1889 / 2170, 87%
Elapsed time:  266h40m0.6s
Transferring:
 *     crypt/qbu2chuhnvlm4g…a7e27sjkg2gnoqh6rcci: 61% /4.041G, 2.251M/s, 11m40s
 *     crypt/qbu2chuhnvlm4g…ce3u1jfauok15sj6m80e: 98% /3.683G, 2.257M/s, 17s
 *     crypt/qbu2chuhnvlm4g…iecaat1saaik3r32kmdu: 23% /4.094G, 2.257M/s, 23m45s
 *     crypt/qbu2chuhnvlm4g…sq05ko7c53b49uin31fu: 50% /4.127G, 2.233M/s, 15m40s

@ncw will flag --drive-pacer-min-sleep be added beta releases