How to check what dedupe is doing?

Hi there,

Wanted to dedupe my Google Drive as follows:

rclone dedupe --dedupe-mode newest --tpslimit 1 orschiro:

It's been running for hours nown without any output.

My Google Drive storage did not reduce.

How can I check what/if this command is doing?

Thanks and regards

Adding -v or -vv will make the command more verbose. The first phase is traverse the file system.

Make sure you are using 1.47 or newer and have made your own API key for best results.

1 Like

@ncw thank you very much. Appreciate your help and support!

Do you see anything I can do to optimise based on this output:

[orschiro@x230 ~]$ rclone -vv dedupe orschiro:
2019/05/27 14:16:12 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "-vv" "dedupe" "orschiro:"]
2019/05/27 14:16:12 DEBUG : Using config file from "/home/orschiro/.config/rclone/rclone.conf"
2019/05/27 14:16:12 INFO  : Google drive root '': Looking for duplicates using interactive mode.
2019/05/27 14:16:12 DEBUG : orschiro: Loaded invalid token from config file - ignoring
2019/05/27 14:16:12 DEBUG : orschiro: Saved new token in config file
2019/05/27 14:16:19 DEBUG : Archive/SDG: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/What do you think of the book?: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/Untitled site: No export formats found for "application/vnd.google-apps.site"
2019/05/27 14:16:19 DEBUG : Archive/Or | Robert Orzanna: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/What is your favourite Linux desktop?: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/Survey among SCORAI members: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/SCORAI Conference 2018 Evaluation: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/Systems of Sustainable Consumption and Production Knowledge Action Network: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : Archive/Digital consciousness: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:19 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:16:19 DEBUG : pacer: Rate limited, increasing sleep to 1.856604879s
2019/05/27 14:16:19 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:16:19 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:16:19 DEBUG : pacer: Rate limited, increasing sleep to 1.642715154s
2019/05/27 14:16:19 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:16:21 DEBUG : Archive/Vergleich Zahnärzte in Karlsruhe: No export formats found for "application/vnd.google-apps.site"
2019/05/27 14:16:21 DEBUG : Archive/Lista de productos ecológicos en los supermercados convencionales de España: No export formats found for "application/vnd.google-apps.site"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/Degrowth (Google) Maps: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/SCORAI Library Paperpile File Upload: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/WM2017: Offizielle Registrierung: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/Humane Technology System: Ignoring unknown document type "application/vnd.google-apps.drive-sdk.758379822725"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/Untitled form: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/Copy of Receive Files in Google Drive: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/WM2017: Official Registration: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:22 DEBUG : Archive/2016-2017/Untitled form: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:31 DEBUG : Archive/2016-2017/Archive/Material objects to feel well: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:31 DEBUG : Archive/2016-2017/Archive/Leih dir die Dinge aus: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:36 DEBUG : Archive/2016-2017/Archive/Backup/Ad blocking survey: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:36 DEBUG : Archive/2016-2017/Archive/Google Scripts/Mark Archived Gmail Messages as Read Automatically: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/2015/I'm proud game: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Find emails with no reply: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Appraisal: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Cycling in New Zealand: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Global Membrane bioreactor (MBR) plants: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Route nach Rheinstetten, Karlsruhe: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/happygiving: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Bachelor of travels: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Uithof Sustainable Campus Map: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Outdoor Urban Bouldering Map: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Konsum Global Karlsruhe Stadtplan: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Givebox Karlsruhe: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Dumpster Diving / Foodsharing Utrecht: Ignoring unknown document type "application/vnd.google-apps.map"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Manage Tasks in Gmail: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Samenvatting nulmeting sorteeranalyse 100-100-100.zdoc: Ignoring unknown document type "application/vnd.google-apps.drive-sdk.457660898219"
2019/05/27 14:16:44 DEBUG : Archive/2016-2017/Archive/Backup/Older/Samenvatting nulmeting sorteeranalyse 100-100-100.zdoc: Ignoring unknown document type "application/vnd.google-apps.drive-sdk.457660898219"
2019/05/27 14:16:46 DEBUG : Archive/2016-2017/Archive/Backup/Older/Loconf - Linux config file overview: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:46 DEBUG : Archive/2016-2017/Archive/Backup/Older/Agenda: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:50 DEBUG : Archive/2016-2017/Archive/Backup/Older/justrenaME/Site suggestion: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Login database [Add login ]: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Login database [Add login ]: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Survey: Japanese Car Brands [Form]: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Report abuse [Report abuse or broken accounts]: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Givebox Karlsruhe Newsletter: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:16:56 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Personal Documents/Recyclung Newsletter: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:01 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Writing/Linux Software Popularity/Collection of most popular Linux software: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:01 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Writing/Linux Software Popularity/Linux Software Popularity: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:01 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Science/UU/Untitled form: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:01 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Science/UU/Untitled form: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:01 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Science/UU/Red books for Sustainable Development: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:02 DEBUG : Archive/2016-2017/Archive/Backup/Older/Projects/Konsum Global Karlsruhe/Newsletter Anmeldung: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:02 DEBUG : Archive/2016-2017/Archive/Backup/Older/Projects/ltiwif/Twitter RSS Feed Generator IFTTT: No export formats found for "application/vnd.google-apps.script"
2019/05/27 14:17:07 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Science/UU/Sustainability Science: Modelling and Indicators/Indicator online assessment tool: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:17:07 DEBUG : Archive/2016-2017/Archive/Backup/Older/Docs/Science/UU/What the Fiets/Price of a triangle: No export formats found for "application/vnd.google-apps.form"
2019/05/27 14:18:02 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:02 DEBUG : pacer: Rate limited, increasing sleep to 1.651717518s
2019/05/27 14:18:02 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:18:56 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:56 DEBUG : pacer: Rate limited, increasing sleep to 1.777820073s
2019/05/27 14:18:56 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:56 DEBUG : pacer: Rate limited, increasing sleep to 2.074297191s
2019/05/27 14:18:56 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:56 DEBUG : pacer: Rate limited, increasing sleep to 4.305187347s
2019/05/27 14:18:56 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:18:58 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:58 DEBUG : pacer: Rate limited, increasing sleep to 1.662951842s
2019/05/27 14:18:58 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:58 DEBUG : pacer: Rate limited, increasing sleep to 2.559428345s
2019/05/27 14:18:58 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:18:58 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:18:58 DEBUG : pacer: Rate limited, increasing sleep to 1.420800981s
2019/05/27 14:18:59 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:01 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:01 DEBUG : pacer: Rate limited, increasing sleep to 1.462380174s
2019/05/27 14:19:01 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:01 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:01 DEBUG : pacer: Rate limited, increasing sleep to 1.499407814s
2019/05/27 14:19:01 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 1.278306012s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 2.791589179s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 4.240722804s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 8.812662556s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 16.901168603s
2019/05/27 14:19:31 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:31 DEBUG : pacer: Rate limited, increasing sleep to 16.804485527s
2019/05/27 14:19:31 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:33 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:33 DEBUG : pacer: Rate limited, increasing sleep to 1.98787693s
2019/05/27 14:19:33 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:33 DEBUG : pacer: Rate limited, increasing sleep to 2.008641533s
2019/05/27 14:19:33 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:37 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:37 DEBUG : pacer: Rate limited, increasing sleep to 1.063437131s
2019/05/27 14:19:37 DEBUG : pacer: Reducing sleep to 0s
2019/05/27 14:19:37 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:37 DEBUG : pacer: Rate limited, increasing sleep to 1.807815562s
2019/05/27 14:19:37 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=82478737740, userRateLimitExceeded)
2019/05/27 14:19:37 DEBUG : pacer: Rate limited, increasing sleep to 2.490113041s
2019/05/27 14:19:37 DEBUG : pacer: Reducing sleep to 0s

That looks like it is working OK. I'm guessing you have lots of files so it will take some time. The --ttpslimit will slow it right down so if you want it to run faster you can removing it.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.