Ajki
(Ajki)
December 31, 2018, 9:17am
1
CMD line:
2018/12/31 09:00:09 DEBUG : rclone: Version "v1.45-050-g1c01d0b8-beta" starting with parameters ["/usr/bin/rclone" "sync" "dropbox:" "gdrive:" "--retries=5" "--transfers=6" "--checkers=6" "--stats" "60s" "--tpslimit" "3" "--drive-chunk-size" "128M" "--log-level" "DEBUG" "--log-file=/home/plex/logs/syncgdrive.log"]
I still get a lot of pacer: low level retry 10/10
2018/12/31 10:20:09 INFO :
Transferred: 12.898G / 12.898 GBytes, 100%, 2.751 MBytes/s, ETA 0s
Errors: 10 (retrying may help)
Checks: 11362 / 11362, 100%
Transferred: 4 / 4, 100%
Elapsed time: 1h20m0.5s
2018/12/31 09:06:05 ERROR : crypt/rijbp68ecdsepkm033j05hjoos/2ne792n2ein8fc50nh9dghf2eg/6ujr39a7dl6bgvegc8r6sdh6g4: error reading destination directory: couldn't list directory: googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded
@ncw any way to get the actual file names during copy eg
I thought could do it with --crypt-show-mapping but its not working.
I think your transfers/checkers are causing the issue. I’d move them down to more like 3/3 and if that works, move up to 4/4 or 5/5 to get no more than 10 total.
I’m not quite sure how the tpslimit plays with those two.
Check out the issue that was opened up and if you’d like to test and provide feedback, that would be great!
ncw
(Nick Craig-Wood)
December 31, 2018, 7:08pm
4
You can decode them with rclone cryptdecode
Ajki
(Ajki)
December 31, 2018, 7:45pm
5
Animosity022:
I think your transfers/checkers are causing the issue. I’d move them down to more like 3/3 and if that works, move up to 4/4 or 5/5 to get no more than 10 total.
I’m not quite sure how the tpslimit plays with those two.
Will test it, well with tpslimit 1 there never should be 403 pacer error, the problem is then its really slow.
I have 150TB+ of data eg 9K+ movies and around 50K TV shows and each have like 2 subtitles that almost 200K.
But I cant use it with sync eg to see decoded names in log file.
Ajki
(Ajki)
January 7, 2019, 4:06pm
6
This is interesting, much over the 750GB upload limit gdrive enforces
Command line:
/usr/bin/rclone sync dropbox: gdrive: --retries=5 --transfers=10 --checkers=3 --tpslimit=2 --drive-pacer-min-sleep=100ms --bwlimit=35M --drive-chunk-size=256M --log-level=INFO --log-file=$LogFile
2019/01/07 17:04:30 INFO :
Transferred: 3.745T / 4.292 TBytes, 87%, 33.292 MBytes/s, ETA 4h46m54s
Errors: 11 (retrying may help)
Checks: 140395 / 140395, 100%
Transferred: 165 / 216, 76%
Elapsed time: 32h46m0.4s
Transferring:
* crypt/0ibcad98vheuku…1niqjke6df4of3ap9oh0: 22% /50.926G, 3.316M/s, 3h22m42s
* crypt/0ibcad98vheuku…1v9bdn1loi4mfdt77ou2: 20% /48.405G, 3.238M/s, 3h21m35s
* crypt/0ibcad98vheuku…2dpfmbm5m77f5fmm7psk: 21% /56.085G, 3.497M/s, 3h36m8s
* crypt/0ibcad98vheuku…7lqjtei3kqgavdt2hkkg: 23% /69.469G, 3.653M/s, 4h9m23s
* crypt/0ibcad98vheuku…bku4sdjjnk61lubuelo8: 31% /47.144G, 3.590M/s, 2h32m26s
* crypt/0ibcad98vheuku…fdpc49eorf4ccmm80da8: 19% /52.072G, 3.404M/s, 3h29m27s
* crypt/0ibcad98vheuku…hsbem9en3oastv8gfcvo: 20% /49.175G, 3.195M/s, 3h27m44s
* crypt/0ibcad98vheuku…op0aap34r59ehtli4sgg: 25% /41.127G, 3.772M/s, 2h19m23s
* crypt/0ibcad98vheuku…rmh25clh54bq54316uqg: 19% /53.052G, 3.524M/s, 3h27m1s
* crypt/0ibcad98vheuku…tgfh873a192i7hboog08: 32% /48.363G, 3.750M/s, 2h29m11s
ncw
(Nick Craig-Wood)
January 8, 2019, 11:05am
7
Interesting… What do you think is causing that?
calisro
(Rob)
January 8, 2019, 6:58pm
8
Just for grins I tried with the exact parameters you have and was banned at ~750G for the day.
Ajki
(Ajki)
January 9, 2019, 4:07am
9
It seems it was copying those big 50GB files over and over without the ban
posted findings and logs here https://github.com/ncw/rclone/issues/2880#issuecomment-452226385
1 Like
Ajki
(Ajki)
January 22, 2019, 2:35pm
10
This works great now, good speed and no 403 errors & upload limit’s
CMD.
/usr/bin/rclone sync dropbox: gdrive: --retries=5 --transfers=4 --checkers=4 --tpslimit=2 --drive-pacer-min-sleep=100ms --bwlimit=9M --dropbox-chunk-size 150M --drive-chunk-size=512M --log-level=INFO --log-file=$LogFile
last log entry ( still going )
2019/01/22 15:31:12 INFO :
Transferred: 8.211T / 8.840 TBytes, 93%, 8.968 MBytes/s, ETA 20h26m8s
Errors: 32 (retrying may help)
Checks: 227174 / 227174, 100%
Transferred: 1889 / 2170, 87%
Elapsed time: 266h40m0.6s
Transferring:
* crypt/qbu2chuhnvlm4g…a7e27sjkg2gnoqh6rcci: 61% /4.041G, 2.251M/s, 11m40s
* crypt/qbu2chuhnvlm4g…ce3u1jfauok15sj6m80e: 98% /3.683G, 2.257M/s, 17s
* crypt/qbu2chuhnvlm4g…iecaat1saaik3r32kmdu: 23% /4.094G, 2.257M/s, 23m45s
* crypt/qbu2chuhnvlm4g…sq05ko7c53b49uin31fu: 50% /4.127G, 2.233M/s, 15m40s
@ncw will flag --drive-pacer-min-sleep be added beta releases
1 Like
ncw
(Nick Craig-Wood)
January 22, 2019, 8:02pm
11
Amazing!
Yes I plan to put this in the release. I have one more tweak here
https://beta.rclone.org/branch/v1.45-059-gf935acb2-fix-2880-drive-pacer-beta/ (uploaded in 15-30 mins)
This lets the pacer burst when it hasn’t used any calls recently - this should be what rclone mount needs.
The binary has a --driver-pacer-burst flag to control this.
2 Likes
Ajki
(Ajki)
January 23, 2019, 6:43am
12
Will wait for linux release
ncw
(Nick Craig-Wood)
January 23, 2019, 10:03am
13