Google Drive pacer issue

nle · September 18, 2020, 3:05pm

What is the problem you are having with rclone?

Trying to remove duplicates, and running the dedupe command is making the pacer nuts.

I've seen this earlier too, on a sync command. I think removing --fast-list solved that sort of, but then I ran into a googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded today, even though I'm on a unlimited account (currently have an issue open with Google, they escalated it and are calling me back).

What is your rclone version (output from `rclone version`)

v1.54.0-beta.4758.c741c02fb.fix-dropbox-batch

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian GNU/Linux 10 (buster)

Which cloud storage system are you using? (eg Google Drive)

Google Drive (GSuite)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone dedupe newest "gdrive-sa:d4g7fbatdhi2sa9f7b59alv9t8" --progress --log-file rclone-dedupe.log -vv --dry-run

The rclone config contents with secrets removed.

[gdrive]
type = drive
client_id = <redacted>
client_secret = <redacted>
scope = drive
root_folder_id = <redacted>
token = {"access_token":"<redacted>"}

A log from the command with the `-vv` flag

2020/09/18 16:48:13 DEBUG : rclone: Version "v1.54.0-beta.4758.c741c02fb.fix-dropbox-batch" starting with parameters ["rclone" "dedupe" "newest" "gdrive-sa:d4g7fbatdhi2k49f7b59alv9t8" "--progress" "--log-file" "rclone-dedupe.log" "-vv" "--dry-run"]
2020/09/18 16:48:13 DEBUG : Using config file from "/home/blaa/.config/rclone/rclone.conf"
2020/09/18 16:48:13 DEBUG : Creating backend with remote "gdrive-sa:d4g7fbatdhi2sa9f7b59alv9t8"
2020/09/18 16:48:13 INFO  : Google drive root 'd4g7fbatdhi2sa9f7b59alv9t8': Looking for duplicates using newest mode.
2020/09/18 16:48:38 DEBUG : Google drive root 'd4g7fbatdhi2sa9f7b59alv9t8': Disabling ListR to work around bug in drive as multi listing (50) returned no entries
2020/09/18 16:48:38 DEBUG : Google drive root 'd4g7fbatdhi2sa9f7b59alv9t8': Recycled 50 entries
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 8.580631ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 53.295667ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 83.560894ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 181.540079ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 255.590963ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 323.364867ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 370.260191ms
2020/09/18 16:48:45 DEBUG : pacer: Reducing sleep to 450.535767ms
(…)
2020/09/18 16:49:12 DEBUG : pacer: Reducing sleep to 334.788352ms
2020/09/18 16:49:13 DEBUG : pacer: Reducing sleep to 239.197301ms
2020/09/18 16:49:13 DEBUG : pacer: Reducing sleep to 240.671016ms
2020/09/18 16:49:13 DEBUG : pacer: Reducing sleep to 101.112241ms
2020/09/18 16:49:13 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 83.860823ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 180.279932ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 266.171862ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 307.531504ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 178.107781ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 235.536694ms
2020/09/18 16:49:14 DEBUG : pacer: Reducing sleep to 13.833128ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 32.509018ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 10.581181ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 109.024831ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 203.884543ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 281.159522ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 351.155642ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 443.874551ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 516.556021ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 555.580022ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 604.157354ms
2020/09/18 16:49:15 DEBUG : pacer: Reducing sleep to 652.648794ms
2020/09/18 16:49:16 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:18 DEBUG : pacer: Reducing sleep to 13.759469ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 5.610476ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 93.151583ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 144.666046ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 231.029063ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 274.901476ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 372.448119ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 472.091481ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 518.866244ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 501.314327ms
2020/09/18 16:49:19 DEBUG : pacer: Reducing sleep to 599.72281ms
2020/09/18 16:49:20 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 14.685803ms
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 45.741773ms
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 55.255554ms
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 81.137632ms
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 118.997978ms
2020/09/18 16:49:21 DEBUG : pacer: Reducing sleep to 107.072952ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 106.340951ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 103.586943ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 122.192429ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 39.51857ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 120.267727ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 139.109813ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 203.452338ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 172.505606ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 229.504ms
2020/09/18 16:49:22 DEBUG : pacer: Reducing sleep to 28.846922ms
2020/09/18 16:49:23 DEBUG : pacer: Reducing sleep to 91.034578ms
2020/09/18 16:49:23 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:30 DEBUG : pacer: Reducing sleep to 76.733404ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 73.514661ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 160.318187ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 231.739008ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 246.877117ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 308.291471ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 388.46855ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 485.210562ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 467.909339ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 486.726778ms
2020/09/18 16:49:31 DEBUG : pacer: Reducing sleep to 61.42263ms
2020/09/18 16:49:32 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:32 DEBUG : pacer: Reducing sleep to 47.0049ms
2020/09/18 16:49:32 DEBUG : pacer: Reducing sleep to 140.769548ms
2020/09/18 16:49:32 DEBUG : pacer: Reducing sleep to 237.205437ms
2020/09/18 16:49:32 DEBUG : pacer: Reducing sleep to 335.003518ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 403.185199ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 304.856708ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 398.772043ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 357.227061ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 448.445805ms
2020/09/18 16:49:33 DEBUG : pacer: Reducing sleep to 280.823911ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 40.954167ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 95.985448ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 181.024066ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 265.364184ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 268.114999ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 363.764042ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 437.642267ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 518.994241ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 581.602026ms
2020/09/18 16:49:34 DEBUG : pacer: Reducing sleep to 467.346318ms
2020/09/18 16:49:35 DEBUG : pacer: Reducing sleep to 338.512716ms
2020/09/18 16:49:35 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 75.800098ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 166.056842ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 259.783017ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 302.804701ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 311.412498ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 403.407829ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 458.171451ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 477.462171ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 538.489078ms
2020/09/18 16:49:36 DEBUG : pacer: Reducing sleep to 384.636938ms
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 84.092122ms
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 125.319001ms
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 209.450162ms
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 302.106442ms
2020/09/18 16:49:37 DEBUG : pacer: Reducing sleep to 394.544786ms
2020/09/18 16:49:38 DEBUG : pacer: Reducing sleep to 349.603863ms
2020/09/18 16:49:38 DEBUG : pacer: Reducing sleep to 443.501124ms
2020/09/18 16:49:38 DEBUG : pacer: Reducing sleep to 510.79367ms
2020/09/18 16:49:38 DEBUG : pacer: Reducing sleep to 591.219312ms
2020/09/18 16:49:38 DEBUG : pacer: Reducing sleep to 379.578107ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 22.96415ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 57.013437ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 37.728189ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 137.490527ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 227.274581ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 312.051361ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 386.634207ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 475.699956ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 568.352598ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 661.820281ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 608.374567ms
2020/09/18 16:49:39 DEBUG : pacer: Reducing sleep to 702.716874ms
2020/09/18 16:49:40 DEBUG : pacer: Reducing sleep to 422.097796ms
2020/09/18 16:49:40 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 29.530573ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 89.283037ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 119.554367ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 164.559404ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 242.631438ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 333.1339ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 430.188206ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 522.264571ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 585.342622ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 643.483659ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 576.041027ms
2020/09/18 16:49:41 DEBUG : pacer: Reducing sleep to 555.096964ms
2020/09/18 16:49:42 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:42 DEBUG : pacer: Reducing sleep to 22.37878ms
2020/09/18 16:49:42 DEBUG : pacer: Reducing sleep to 118.272478ms
2020/09/18 16:49:42 DEBUG : pacer: Reducing sleep to 215.879927ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 297.716132ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 334.943372ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 316.297279ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 407.956351ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 488.665122ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 557.881529ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 642.079027ms
2020/09/18 16:49:43 DEBUG : pacer: Reducing sleep to 606.500884ms
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 102.308234ms
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 46.10155ms
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 108.16587ms
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 173.363943ms
2020/09/18 16:49:44 DEBUG : pacer: Reducing sleep to 258.619343ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 253.567565ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 352.352132ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 424.313391ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 522.264604ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 589.953655ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 619.090856ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 672.335209ms
2020/09/18 16:49:45 DEBUG : pacer: Reducing sleep to 228.436829ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 40.604845ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 85.381707ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 88.603087ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 181.010002ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 263.145646ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 342.61583ms
2020/09/18 16:49:46 DEBUG : pacer: Reducing sleep to 436.513401ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 490.285748ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 539.852725ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 616.523804ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 604.252058ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 634.990565ms
2020/09/18 16:49:47 DEBUG : pacer: Reducing sleep to 303.421402ms
2020/09/18 16:49:48 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:48 DEBUG : pacer: Reducing sleep to 86.378595ms
2020/09/18 16:49:48 DEBUG : pacer: Reducing sleep to 183.535606ms
2020/09/18 16:49:48 DEBUG : pacer: Reducing sleep to 275.850425ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 69.951736ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 149.361547ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 206.138208ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 289.424292ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 356.937199ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 420.111431ms
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:49 DEBUG : pacer: Reducing sleep to 22.921502ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 70.685068ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 150.888868ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 210.422397ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 275.833856ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 307.628461ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 393.513249ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 467.069763ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 562.270492ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 397.178483ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 495.802855ms
2020/09/18 16:49:50 DEBUG : pacer: Reducing sleep to 568.807299ms
2020/09/18 16:49:51 DEBUG : pacer: Reducing sleep to 63.085539ms
2020/09/18 16:49:51 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 75.716261ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 130.942654ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 175.871949ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 214.290114ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 311.360662ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 402.661883ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 471.787969ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 542.984844ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 569.958228ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 643.264196ms
2020/09/18 16:49:52 DEBUG : pacer: Reducing sleep to 659.417958ms
2020/09/18 16:49:53 DEBUG : pacer: Reducing sleep to 243.644465ms
2020/09/18 16:49:53 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 24.259801ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 108.286174ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 204.32066ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 282.899437ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 357.509957ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 403.742372ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 488.291127ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 508.3466ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 589.103923ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 680.336153ms
2020/09/18 16:49:54 DEBUG : pacer: Reducing sleep to 532.301836ms
2020/09/18 16:49:55 DEBUG : pacer: Reducing sleep to 158.272944ms
2020/09/18 16:49:55 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 2.336758ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 57.324316ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 135.792684ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 83.442005ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 181.211832ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 276.881814ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 376.456225ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 415.801414ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 447.126088ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 524.404601ms
2020/09/18 16:49:56 DEBUG : pacer: Reducing sleep to 523.131541ms
2020/09/18 16:49:57 DEBUG : pacer: Reducing sleep to 248.540234ms
2020/09/18 16:49:57 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:57 DEBUG : pacer: Reducing sleep to 53.654104ms
2020/09/18 16:49:57 DEBUG : pacer: Reducing sleep to 119.680253ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 193.872396ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 127.461497ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 212.339819ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 278.648557ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 268.695792ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 367.528603ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 451.907664ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 408.987335ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 469.580056ms
2020/09/18 16:49:58 DEBUG : pacer: Reducing sleep to 301.514733ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 76.792786ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 130.408805ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 223.166227ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 287.003691ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 301.846959ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 371.652187ms
2020/09/18 16:49:59 DEBUG : pacer: Reducing sleep to 399.458856ms
2020/09/18 16:50:00 DEBUG : pacer: Reducing sleep to 266.664161ms
2020/09/18 16:50:00 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:00 DEBUG : pacer: Reducing sleep to 74.752462ms
2020/09/18 16:50:00 DEBUG : pacer: Reducing sleep to 163.193862ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 141.035549ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 25.772969ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 83.980362ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 177.465207ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 266.073837ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 332.304394ms
2020/09/18 16:50:01 DEBUG : pacer: Reducing sleep to 429.294899ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 380.4409ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 422.106645ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 509.716073ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 492.528566ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 568.214132ms
2020/09/18 16:50:02 DEBUG : pacer: Reducing sleep to 280.955181ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 29.256224ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 115.577462ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 187.969671ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 128.808814ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 211.170038ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 277.382702ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 333.756407ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 265.187793ms
2020/09/18 16:50:03 DEBUG : pacer: Reducing sleep to 257.271296ms
2020/09/18 16:50:04 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 59.323066ms
2020/09/18 16:50:05 DEBUG : Google drive root 'd4g7fbatdhi2k49f7b59alv9t8': Re-enabling ListR as previous detection was in error
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 139.40348ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 197.537836ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 274.246927ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 347.51394ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 411.905278ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 364.609695ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 388.477804ms
2020/09/18 16:50:05 DEBUG : pacer: Reducing sleep to 458.583803ms
2020/09/18 16:50:06 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:06 DEBUG : Google drive root 'd4g7fbatdhi2k49f7b59alv9t8': Disabling ListR to work around bug in drive as multi listing (39) returned no entries
2020/09/18 16:50:06 DEBUG : Google drive root 'd4g7fbatdhi2k49f7b59alv9t8': Recycled 39 entries
2020/09/18 16:50:07 DEBUG : pacer: Reducing sleep to 43.726841ms
2020/09/18 16:50:07 DEBUG : pacer: Reducing sleep to 54.174148ms
(…)
2020/09/18 16:50:17 DEBUG : pacer: Reducing sleep to 252.07915ms
2020/09/18 16:50:17 DEBUG : pacer: Reducing sleep to 298.003755ms
2020/09/18 16:50:17 DEBUG : pacer: Reducing sleep to 381.917763ms
2020/09/18 16:50:17 DEBUG : pacer: Reducing sleep to 433.645079ms
2020/09/18 16:50:17 DEBUG : pacer: Reducing sleep to 419.048356ms
2020/09/18 16:50:18 DEBUG : pacer: Reducing sleep to 408.520947ms
2020/09/18 16:50:18 DEBUG : pacer: Reducing sleep to 179.076232ms
2020/09/18 16:50:18 DEBUG : pacer: Reducing sleep to 232.940203ms
2020/09/18 16:50:18 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 4.057061ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 87.753586ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 182.337993ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 257.955026ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 354.961221ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 441.174422ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 397.060421ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 458.152283ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 442.00717ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 385.239083ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 464.7995ms
2020/09/18 16:50:19 DEBUG : pacer: Reducing sleep to 287.183226ms
2020/09/18 16:50:20 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:20 DEBUG : pacer: Reducing sleep to 24.421767ms
2020/09/18 16:50:20 DEBUG : pacer: Reducing sleep to 118.624437ms
2020/09/18 16:50:20 DEBUG : pacer: Reducing sleep to 155.21147ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 136.279203ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 203.304712ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 301.323454ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 379.452597ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 467.848472ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 560.277158ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 566.373044ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 557.675179ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 480.704191ms
2020/09/18 16:50:21 DEBUG : pacer: Reducing sleep to 115.060865ms
2020/09/18 16:50:22 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 34.793412ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 66.231529ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 131.402992ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 224.100399ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 308.760887ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 404.783701ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 487.873961ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 464.833962ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 523.327774ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 613.940375ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 657.697007ms
2020/09/18 16:50:23 DEBUG : pacer: Reducing sleep to 324.370437ms
2020/09/18 16:50:24 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:24 DEBUG : pacer: Reducing sleep to 59.379056ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 98.106561ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 151.038409ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 204.417999ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 234.614375ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 326.084869ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 407.00448ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 450.803103ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 502.129106ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 471.490249ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 443.713978ms
2020/09/18 16:50:25 DEBUG : pacer: Reducing sleep to 359.527002ms
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 149.559472ms
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 19.615253ms
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 85.621483ms
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 185.405101ms
2020/09/18 16:50:26 DEBUG : pacer: Reducing sleep to 244.304051ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 170.973409ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 265.635394ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 209.702698ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 293.98683ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 371.402947ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 468.973185ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 521.575785ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 261.509608ms
2020/09/18 16:50:27 DEBUG : pacer: Reducing sleep to 238.193744ms
2020/09/18 16:50:28 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:28 DEBUG : pacer: Reducing sleep to 18.316999ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 50.647794ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 128.444616ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 204.617056ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 279.553076ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 345.516494ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 420.950329ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 513.235503ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 561.386655ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 581.224205ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 650.060451ms
2020/09/18 16:50:29 DEBUG : pacer: Reducing sleep to 571.085361ms
2020/09/18 16:50:30 DEBUG : pacer: Reducing sleep to 110.421536ms
2020/09/18 16:50:30 DEBUG : pacer: Reducing sleep to 0s
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 41.393637ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 136.800528ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 236.038659ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 314.690317ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 324.891203ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 373.704437ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 440.149321ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 510.637468ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 593.012208ms
2020/09/18 16:50:31 DEBUG : pacer: Reducing sleep to 635.53318ms

Any pointers on this issue? Bug or not?

Animosity022 · September 18, 2020, 3:14pm

It's just Google telling you to slow down and those are normal items from the API depending on how hard you are hitting the API.

nle · September 18, 2020, 4:42pm

I've tested this both on a regular account and on service account with default settings. Don't think I'm hitting their api hard.

And shouldn't the log at least say something more?

Animosity022 · September 18, 2020, 4:46pm

You only get 10 hits per second, You can check your console and you see the errors there as those are just normal API errors when you hit the API more than 10 times per second.

What else would the log say as it reports the error and it's shown in debug?

nle · September 18, 2020, 4:54pm

How am I hitting the API? Is it listing the directories?

nle · September 18, 2020, 5:36pm

API stats from the last 6 hours:

Animosity022 · September 18, 2020, 5:46pm

Does the same happen on the released version?

What is the link you grabbed that beta from as it looks like a specific drop box fix.

nle · September 18, 2020, 5:48pm

Have not tested on the released version. Yes it's a specific fix fixing batch upload on Dropbox.

Animosity022 · September 18, 2020, 5:49pm

Can you post the issue/link/something so I can find it?

nle · September 18, 2020, 6:13pm

https://beta.rclone.org/branch/fix-dropbox-batch/

It's an post by @ncw on the Dropbox forums, contacted @ncw about it and got the link.

nle · September 18, 2020, 6:58pm

Ran it for 1:30 hour, and all I got in the log is pacer. I can try the stable release, since I'm just using gdrive for this.

Animosity022 · September 18, 2020, 7:16pm

That seems strange as I run it and hammer the API and I get these:

2020/09/18 15:15:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=286927808882, userRateLimitExceeded)
2020/09/18 15:15:49 DEBUG : pacer: Rate limited, increasing sleep to 2.669956424s
2020/09/18 15:15:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=286927808882, userRateLimitExceeded)
2020/09/18 15:15:49 DEBUG : pacer: Rate limited, increasing sleep to 4.311950649s
2020/09/18 15:15:49 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=286927808882, userRateLimitExceeded)
2020/09/18 15:15:49 DEBUG : pacer: Rate limited, increasing sleep to 2.35443459s
2020/09/18 15:15:49 DEBUG : pacer: low level retry 4/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=286927808882, userRateLimitExceeded)
2020/09/18 15:15:49 DEBUG : pacer: Rate limited, increasing sleep to 8.360871576s
2020/09/18 15:15:49 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=286927808882, userRateLimitExceeded)
2020/09/18 15:15:49 DEBUG : pacer: Rate limited, increasing sleep to 1.464331808s

Your pacer message seems odd as if you are not hitting Google?

nle · September 18, 2020, 8:24pm

Yes, very strange.
I tested

with the stable version now on, same issue. Running on a VPS.
on a different server (dropbox beta version), same issue.

The goal here is to sync between Dropbox and GSuite, but got two strange issues today when testing.

Uploaded loads of duplicates, don't understand why. Same files, same sizes, same everything. It's going from dropbox normal to a crypt gdrive. I did change to a service account, and that could be a reason (a bit a strange).
Got googleapi: Error 403: The user's Drive storage quota has been exceeded., storageQuotaExceeded. The error is strange at its self, and google confirmed I have unlimited, and 1st teer support did not see any issues. The other thing is that it's not absolute, so I could upload a few files before I got the error again.

Animosity022 · September 18, 2020, 8:26pm

Logs would be the goal here as not sure why you'd get duplicates.

There usually means you have something misconfigured and you are writing to the service account's area which has a low quota of storage.

nle · September 18, 2020, 8:28pm

Yes, how should I go about getting better logs?

storageQuotaExceeded
I created a service account with "edit" rights. Shared a folder in my personal gdrive with that account. In rclone config I lock in a directory.

Animosity022 · September 18, 2020, 8:29pm

You would run the command with -vv and share the copy or sync log.

You'd have to share the full command and config you are running to sync with a log to look at to validate what's going on.

nle · September 18, 2020, 8:45pm

I can create a separate thread about the sync issue, currently have an issue open with google. I guess it's a chance they are related.

The dedupe command is:
rclone dedupe newest "gdrive:d4g7fbatdhi2sa9f7b59alv9t8" --progress --log-file rclone-dedupe.log -vv --dry-run

The logfile I get is literally just the info on top, and then all the pacing. Nothing else.

I get the same issue with both my regular normal account, and with service accounts. The config is the same that I have been using for years.

nle · September 18, 2020, 10:07pm

Ok, I'm running the same command and dumping the headers as well

Cmd:
dedupe rclone dedupe newest "gdrive-sa:d4g7fbatdhi2k49f7b59alv9t8" --progress --log-file rclone-dedupe-dump.log --dump headers,bodies,requests,responses,auth,filters,goroutines,openfiles -vv --dry-run

I get a lot of this, so it seems like it's searching/parsing through, which is good. I just wish for a bit more information in the debug log.

Host: www.googleapis.com
User-Agent: rclone/v1.53.1
Authorization: <redacted>
X-Goog-Api-Client: gl-go/1.15.0 gdcl/20200616
Accept-Encoding: gzip

2020/09/18 23:34:46 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2020/09/18 23:34:47 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2020/09/18 23:34:47 DEBUG : HTTP RESPONSE (req 0xc000178d00)
2020/09/18 23:34:47 DEBUG : HTTP/1.1 200 OK
Transfer-Encoding: chunked
Alt-Svc: h3-Q050=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-27=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-T050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Content-Security-Policy: frame-ancestors 'self'
Content-Type: application/json; charset=UTF-8
Date: Fri, 18 Sep 2020 21:34:46 GMT
Expires: Fri, 18 Sep 2020 21:34:46 GMT
Server: GSE
Vary: Origin
Vary: X-Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

3275
{"incompleteSearch":false,"files":[{"id":"1y407G847x7FpI83L4taQ2DeDp5Ma6Ddo","name":"127jevj519ms2a84n6b0uk6cjc","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["17hbKSiV5ismafvBrRGFgdyZyCpGJ2vUH"],"webViewLink":"https://drive.google.com/drive/folders/1y407G847x7FpI83L4taQ2DeDp5Ma6Ddo","createdTime":"2019-08-22T17:02:19.048Z","modifiedTime":"2019-08-22T17:02:19.048Z"},{"id":"12jWIW7lJD_PredxeidZIU34p3vCSgNUB","name":"9r8a0il33jh2gbu289lbsbrs80","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/drive/folders/12jWIW7lJD_PredxeidZIU34p3vCSgNUB","createdTime":"2019-03-11T18:04:20.993Z","modifiedTime":"2019-03-11T18:04:20.993Z"},{"id":"1tsrQjo-nUfRdTyET2F0fIgJi7cM7Yzkb","name":"1mejtlh0ur9dtdce22q10qujruj04inoa6r87ej95qujanp4k5u0","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1AJfpicDLfBLA7MehXEZuLdBI_emf9VG_"],"webViewLink":"https://drive.google.com/file/d/1tsrQjo-nUfRdTyET2F0fIgJi7cM7Yzkb/view?usp=drivesdk","createdTime":"2019-01-02T18:04:02.193Z","modifiedTime":"2019-01-02T12:58:59.000Z","md5Checksum":"fa0277b13280ab2878831901ca3b6592","size":"1741264"},{"id":"1ocUVIZaLz7KgX198sPdiNhWUiJFoXicg","name":"dihldaf3vrdg7m2f9pvt55it587v5kuqkmfusnmjkpkiqmsb83vg","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1AJfpicDLfBLA7MehXEZuLdBI_emf9VG_"],"webViewLink":"https://drive.google.com/file/d/1ocUVIZaLz7KgX198sPdiNhWUiJFoXicg/view?usp=drivesdk","createdTime":"2019-01-02T18:03:33.301Z","modifiedTime":"2019-01-02T12:58:44.000Z","md5Checksum":"f0ba470a69f1b2b4bdf1c140b9a755ae","size":"87739"},{"id":"1s3rA7owd1P1cdxx6WyFJHCsVEWokTCjG","name":"jvub85irtv1bactjkapcmiimrq234100r4325jf1r2ikb6imog0g","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1AJfpicDLfBLA7MehXEZuLdBI_emf9VG_"],"webViewLink":"https://drive.google.com/file/d/1s3rA7owd1P1cdxx6WyFJHCsVEWokTCjG/view?usp=drivesdk","createdTime":"2018-11-08T18:03:40.024Z","modifiedTime":"2018-11-08T14:58:44.000Z","md5Checksum":"fd84ae4a003e052a8e696a1ef29af111","size":"1939521"},{"id":"1YrYlxhZvdGDpSDxvb0Xki6jDMPFNpyF9","name":"tp313ts9s4u98a4cq4d83v9s6qf8779b2n246rjlqlbnn86hn78dqskti9pdhjpt2j75lq5r3uhlu","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1o8GuXNZe4dFWAbzjXItXeQQUmOuv-F1y"],"webViewLink":"https://drive.google.com/file/d/1YrYlxhZvdGDpSDxvb0Xki6jDMPFNpyF9/view?usp=drivesdk","createdTime":"2018-10-25T17:03:18.476Z","modifiedTime":"2018-10-25T08:06:37.000Z","md5Checksum":"68f0333319dca3dff3fe443bfccd03e1","size":"67267"},{"id":"1TWbFzNhAaJmbIaD1I3QIdzRescDa9eeN","name":"7g63sp24t5vje03eet29o6iet36snko5o3of3vbk7l80310f1mt5nfthv0aeob4uundpei8f68j18","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/drive/folders/1TWbFzNhAaJmbIaD1I3QIdzRescDa9eeN","createdTime":"2018-10-24T17:03:19.326Z","modifiedTime":"2018-10-24T17:03:19.326Z"},{"id":"1Q5lzP3by8RQcw243g6grW0RaUB9-rfwK","name":"fpfr2anhji0ihe5ooqfc65cvpo","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/drive/folders/1Q5lzP3by8RQcw243g6grW0RaUB9-rfwK","createdTime":"2018-10-11T17:03:17.523Z","modifiedTime":"2018-10-11T17:03:17.523Z"},{"id":"1xGr__Zw9vq_FGy_zZPnKRyJ8VBc0sf-s","name":"5qgh27o9mede3jbu4orubffm3o","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["17hbKSiV5ismafvBrRGFgdyZyCpGJ2vUH"],"webViewLink":"https://drive.google.com/drive/folders/1xGr__Zw9vq_FGy_zZPnKRyJ8VBc0sf-s","createdTime":"2018-06-20T19:39:12.430Z","modifiedTime":"2018-06-20T19:39:12.430Z"},{"id":"19PoIEiRuyH0p4X-X4XKWl2vvTPAFlKJO","name":"1bb1nhnd1e8dtfkcvccedhq4d4","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/drive/folders/19PoIEiRuyH0p4X-X4XKWl2vvTPAFlKJO","createdTime":"2018-05-04T17:02:37.126Z","modifiedTime":"2018-05-04T17:02:37.126Z"},{"id":"1qblvZ6s9f3jqDX9fDFIsI8RDPAeJV5O7","name":"gb6oenv07pl4ciccq4p3qmtuvkcviq78ceq2fq437ftmc3m0u9tg","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/drive/folders/1qblvZ6s9f3jqDX9fDFIsI8RDPAeJV5O7","createdTime":"2018-03-25T13:05:40.533Z","modifiedTime":"2018-03-25T13:05:40.533Z"},{"id":"1Ut5T57fFCA_sUSTn_DzKM_ZlJ9i4fZkf","name":"ue5jnmjph6gdllhblubin90476jmi030b4pcuca7r03hjjbl558g","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/drive/folders/1Ut5T57fFCA_sUSTn_DzKM_ZlJ9i4fZkf","createdTime":"2018-03-25T12:57:36.412Z","modifiedTime":"2018-03-25T12:57:36.412Z"},{"id":"12knLFJlUexA7yYmrcc9--t4lD16RXboC","name":"l6pdf5vnu6qghniaojnfl3eh7o","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["17hbKSiV5ismafvBrRGFgdyZyCpGJ2vUH"],"webViewLink":"https://drive.google.com/drive/folders/12knLFJlUexA7yYmrcc9--t4lD16RXboC","createdTime":"2018-03-25T12:57:29.884Z","modifiedTime":"2018-03-25T12:57:29.884Z"},{"id":"1waTy5fqhVeBWXeToU4mY0Rb02TpyM-3y","name":"1h1lalijum8o3n0h93951rpdog","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["17hbKSiV5ismafvBrRGFgdyZyCpGJ2vUH"],"webViewLink":"https://drive.google.com/drive/folders/1waTy5fqhVeBWXeToU4mY0Rb02TpyM-3y","createdTime":"2018-03-25T12:49:36.619Z","modifiedTime":"2018-03-25T12:49:36.619Z"},{"id":"1oqmmQAw9PQSbyIcwMjGK839B6yz6K3yM","name":"4jmu21rptjpd3hpv47sk1jbets","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/drive/folders/1oqmmQAw9PQSbyIcwMjGK839B6yz6K3yM","createdTime":"2018-03-25T12:16:48.528Z","modifiedTime":"2018-03-25T12:16:48.528Z"},{"id":"13psJzrWUlATNr7rSb3fsiyqllCIwD_fS","name":"9qhe4q71n0s492krh14q6hdeknhtafj8leenjqlbsashb10623gg","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/drive/folders/13psJzrWUlATNr7rSb3fsiyqllCIwD_fS","createdTime":"2018-03-25T12:11:07.822Z","modifiedTime":"2018-03-25T12:11:07.822Z"},{"id":"1TBjmFHqFx96T7qc0F5fXkPjUtysmqAQD","name":"gelc2393roqlvf7030e1eltdoc","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["17hbKSiV5ismafvBrRGFgdyZyCpGJ2vUH"],"webViewLink":"https://drive.google.com/drive/folders/1TBjmFHqFx96T7qc0F5fXkPjUtysmqAQD","createdTime":"2018-03-25T12:05:00.154Z","modifiedTime":"2018-03-25T12:05:00.154Z"},{"id":"1yXAg-M7DFu0OF8irDWe_I1ANBvf8e5H-","name":"7sl5vsngdrb78ai95cfjqibbmc","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1InfHpQy9VhqpgN9dh0DLXu96zmOgi92F"],"webViewLink":"https://drive.google.com/drive/folders/1yXAg-M7DFu0OF8irDWe_I1ANBvf8e5H-","createdTime":"2018-03-25T09:41:57.770Z","modifiedTime":"2018-03-25T09:41:57.770Z"},{"id":"1qnrxTiNHiAaFw7WY5FfdLXFFeZROpFvG","name":"2kua7nrsu303tg3re17caug7cqljip6m30j7i5askna02kdv7kn0","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1InfHpQy9VhqpgN9dh0DLXu96zmOgi92F"],"webViewLink":"https://drive.google.com/drive/folders/1qnrxTiNHiAaFw7WY5FfdLXFFeZROpFvG","createdTime":"2018-03-25T09:31:31.985Z","modifiedTime":"2018-03-25T09:31:31.985Z"},{"id":"17XnW9L7lM6Q5dFVLofLsQy-Weii0_CWO","name":"3ouvepco8b7ah7l0levejmghtv9rnmleafcvetv4t53lmmo422dg","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/drive/folders/17XnW9L7lM6Q5dFVLofLsQy-Weii0_CWO","createdTime":"2018-03-25T09:30:24.485Z","modifiedTime":"2018-03-25T09:30:24.485Z"},{"id":"10xq3FOr0QXyDAa6c-_XxqXZB5Ww_Y6S5","name":"qqkl9gis0tkurs3snfh7njhjbjajo8icqq6p8r9itc96i28erch0","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1InfHpQy9VhqpgN9dh0DLXu96zmOgi92F"],"webViewLink":"https://drive.google.com/drive/folders/10xq3FOr0QXyDAa6c-_XxqXZB5Ww_Y6S5","createdTime":"2018-03-25T09:08:47.534Z","modifiedTime":"2018-03-25T09:08:47.534Z"},{"id":"1p6vLvkiOA1E2dhN2kineK6F5ktcTN_1-","name":"bloesd9q4k2cmqimg0ju5rujbs","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/drive/folders/1p6vLvkiOA1E2dhN2kineK6F5ktcTN_1-","createdTime":"2018-03-25T08:12:55.622Z","modifiedTime":"2018-03-25T08:12:55.622Z"},{"id":"1ZbeQUAv95P8SUQKzHUD4NOZBVdwG2EeF","name":"ait24k35c1ead1hjgfvev9rkds","mimeType":"application/vnd.google-apps.folder","trashed":false,"explicitlyTrashed":false,"parents":["1InfHpQy9VhqpgN9dh0DLXu96zmOgi92F"],"webViewLink":"https://drive.google.com/drive/folders/1ZbeQUAv95P8SUQKzHUD4NOZBVdwG2EeF","createdTime":"2018-03-25T03:38:34.312Z","modifiedTime":"2018-03-25T03:38:34.312Z"},{"id":"1OqwOgo8jiC2g_srOL8Y31SSP2rtQtmnI","name":"nqq7maeeu9a5fsqddiq476n7jk","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/file/d/1OqwOgo8jiC2g_srOL8Y31SSP2rtQtmnI/view?usp=drivesdk","createdTime":"2018-04-07T12:03:23.819Z","modifiedTime":"2017-01-23T13:13:26.371Z","md5Checksum":"cad481abe124a9625881af5de31ff593","size":"6196"},{"id":"1Pb9DVLf90eZV7BMf_M0G9WWpMzz8Fgwb","name":"e0jde83t67bjt0hl0o9o43jhqm77bkoj06nlctqb2vn33s6vsrcg","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1nYNrk8iRLkDXPm89yLX5mrO0-3fMJ7Ge"],"webViewLink":"https://drive.google.com/file/d/1Pb9DVLf90eZV7BMf_M0G9WWpMzz8Fgwb/view?usp=drivesdk","createdTime":"2018-03-25T05:28:29.482Z","modifiedTime":"2016-09-28T07:10:08.538Z","md5Checksum":"30946575ce053823332bdeb63bb3516a","size":"531845"},{"id":"1kJJFMYq2Dea-i_467ov17mD0vJUEStnp","name":"oqaq5htqoounmej0u7slts0pj44gl7crc5efrnactogahiejse5g","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1YYWU5RosgNvb-MElF4XM24aaecvO_p3y"],"webViewLink":"https://drive.google.com/file/d/1kJJFMYq2Dea-i_467ov17mD0vJUEStnp/view?usp=drivesdk","createdTime":"2018-03-25T02:04:35.147Z","modifiedTime":"2015-08-13T07:35:10.000Z","md5Checksum":"bf88d8c4bee2bb84cc0be41134f7e696","size":"9828"},{"id":"1mfugJpP9T2pBYzXAPqTRKVGaZPKEv3KI","name":"n7h7fs8osaoi0i41dsqd11238r0er57lvve23gtictlf9n3jqpp0","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1AJfpicDLfBLA7MehXEZuLdBI_emf9VG_"],"webViewLink":"https://drive.google.com/file/d/1mfugJpP9T2pBYzXAPqTRKVGaZPKEv3KI/view?usp=drivesdk","createdTime":"2018-10-11T17:03:14.763Z","modifiedTime":"2015-05-25T11:48:20.157Z","md5Checksum":"af6655d38178999949ba6a27e272d329","size":"1109198"},{"id":"1n2Efu8qa4zey7hvuz44D-U_DpIjVsuEL","name":"7t1jcu1lql1jqobmokvqsdobvt863ckocr058oqmerqor0v05cpg","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1qgh-TTPFRjmH2mYWVrvEsmC5G-YC3IvX"],"webViewLink":"https://drive.google.com/file/d/1n2Efu8qa4zey7hvuz44D-U_DpIjVsuEL/view?usp=drivesdk","createdTime":"2018-03-25T09:26:31.386Z","modifiedTime":"2015-05-25T11:46:23.606Z","md5Checksum":"9769bd194c3240138eb1b1765458a945","size":"1819104"},{"id":"1uBR_XE1fVyZpp2AANNToUewJzxjB1R86","name":"3r0oudlmi2ve2o4okete0g8fc8brlm7pb2i5ca4tqu02fl8q4v7g","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1qgh-TTPFRjmH2mYWVrvEsmC5G-YC3IvX"],"webViewLink":"https://drive.google.com/file/d/1uBR_XE1fVyZpp2AANNToUewJzxjB1R86/view?usp=drivesdk","createdTime":"2018-03-25T09:26:11.814Z","modifiedTime":"2015-05-25T11:46:23.426Z","md5Checksum":"0c4f81457d8d7d0b18591665cc004ab3","size":"94784"},{"id":"1uWkj-PlVu5IAB3ODnsInOGaKRUHcsXyy","name":"ku9knlcsgmgdsd2468r3gbo5ivlj9q3lv6nm59c6v50eub3eumug","mimeType":"application/octet-stream","trashed":false,"explicitlyTrashed":false,"parents":["1qgh-TTPFRjmH2mYWVrvEsmC5G-YC3IvX"],"webViewLink":"https://drive.google.com/file/d/1uWkj-PlVu5IAB3ODnsInOGaKRUHcsXyy/view?usp=drivesdk","createdTime":"2018-04-07T12:03:36.357Z","modifiedTime":"2015-05-25T11:46:23.000Z","md5Checksum":"39908c3a45c0eaf454c58a86c798b71a","size":"32"}]}
0

I did a test where I used cryptdecrypt to go manually to a path where I know there is duplicates, and that worked.

(@ncw I really wish for a easier way to navigate encrypted folders. Maybe just a "cryptencode" so at least we easily can turn a cleartext path into a crypted path.)

2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/n13hcceclfkhs3rr4oipua8efk: Found 3 files with duplicate names
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/n13hcceclfkhs3rr4oipua8efk: Skipped delete as --dry-run is set
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/n13hcceclfkhs3rr4oipua8efk: Skipped delete as --dry-run is set
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/n13hcceclfkhs3rr4oipua8efk: Deleted 2 extra copies
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/7omo00otaraqkrcof4hog3vjvc: Found 3 files with duplicate names
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/7omo00otaraqkrcof4hog3vjvc: Skipped delete as --dry-run is set
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/7omo00otaraqkrcof4hog3vjvc: Skipped delete as --dry-run is set
2020/09/19 00:03:51 NOTICE: uotjd5ku9jhucf91m5v4eujag61q277k6ljjqckvv1u2bhhiru10/7omo00otaraqkrcof4hog3vjvc: Deleted 2 extra copies

So in summary. It seems like it works, it's just the debug log that don't output anything usefull until it hits a duplicate. And if there is a lot of files you will need to go through a lot of pacing.

(I will try to debug the more strange sync issue next)

random404 · September 18, 2020, 10:43pm

Keep --fast-list and add --tpslimit 10 if the only thing you are running is rclone dedupe. 5~7 if you have other stuff

Animosity022 · September 19, 2020, 1:09am

Yep. That’ll help too. The 403s are pretty harmless in this case other than costing a little time.

I’d be more curious how and why you keep creating dupes as I’ve never made one myself.