What is the problem you are having with rclone?
When syncing to Google Drive on my personal Gmail account, it always errors out after some time with
Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota
At first I used the defaults, but then I tried setting the values of transfers and checkers to 8/8, then 4/4, but it still errored out. Finally I set them to 1/1, and then the operation completed successfully.
The quotas page in Google Cloud Console is set to the maximum/defaults (1000,000,000,000 / 20,000 / 20,000) and the actual requests are nowhere near those numbers.
Changing transfers and checkers to 1 seems to solve the problem. I never had this issue on my work Google-account.
How come it seems to exceed the quota even though the usage stats in Cloud Console show otherwise? And is it normal to have to set transfers and checkers to 1 for Google Drive to work?
The log attached shows excerpts from the log when using transfers=4 and checkers=4. I had to remove hundreds of near-identical lines because of the post max size limitations. One thing I noted in the log is that it's checking hundreds of files in
site1/public/images/features/download/ in less than a second. Not sure how the checking is performed, but could the large number of files in that directory be the cause of the error?
Run the command 'rclone version' and share the full output of the command.
rclone v1.52.3
-
os/arch: darwin/amd64
-
go version: go1.14.7
Which cloud storage system are you using? (eg Google Drive)
Google Drive
The command you were trying to run (eg rclone copy /tmp remote:tmp)
rclone sync "/Users/umberto/dev" "umbertorojas_drive:/dev" \
--delete-excluded \
--exclude .git/ \
--exclude vendor/ \
--drive-use-trash=false \
--transfers=1 \
--checkers=1 \
--retries=2 \
--low-level-retries=2 -vv
The rclone config contents with secrets removed.
[umbertorojas_drive]
type = drive
client_id = secret.apps.googleusercontent.com
client_secret = secret
scope = drive.file
token = {"access_token":"secret","token_type":"Bearer","refresh_token":"secret","expiry":"2022-06-27T12:30:00.357391+02:00"}
root_folder_id = secret
A log from the command with the -vv flag
2022/06/27 11:43:57 DEBUG : rclone: Version "v1.52.3" starting with parameters ["/Users/umberto/Utils/bin/rclone" "sync" "/Users/umberto/PhpstormProjects" "umbertorojas_drive:/PhpstormProjects" "--delete-excluded" "--exclude" ".git/" "--exclude" "vendor/" "--drive-use-trash=false" "--transfers=4" "--checkers=4" "--retries=2" "--low-level-retries=2" "-vv"]
2022/06/27 11:43:57 DEBUG : Using config file from "/Users/umberto/.config/rclone/rclone.conf"
2022/06/27 11:43:58 DEBUG : fs cache: renaming cache item "umbertorojas_drive:/PhpstormProjects" to be canonical "umbertorojas_drive:PhpstormProjects"
2022/06/27 11:43:58 DEBUG : site2-php/.git: Excluded
2022/06/27 11:43:58 DEBUG : site2-php/vendor: Excluded
2022/06/27 11:43:58 DEBUG : workspace.xml: Size and modification time the same (differ by -254.115µs, within tolerance 1ms)
2022/06/27 11:43:58 DEBUG : .DS_Store: Size and modification time the same (differ by -629.291µs, within tolerance 1ms)
2022/06/27 11:43:58 DEBUG : .DS_Store: Unchanged skipping
2022/06/27 11:43:58 DEBUG : workspace.xml: Unchanged skipping
2022/06/27 11:43:58 DEBUG : site3.com/.git: Excluded
2022/06/27 11:43:59 DEBUG : site4-api/composer.json: Size and modification time the same (differ by -767.16µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site4-api/composer.json: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site4-api/.gitignore: Size and modification time the same (differ by -56.572µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site4-api/.gitignore: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site1/.git: Excluded
2022/06/27 11:43:59 DEBUG : site1/vendor: Excluded
2022/06/27 11:43:59 DEBUG : site2-php/.gitignore: Size and modification time the same (differ by -649.629µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/.htaccess: Size and modification time the same (differ by -255.994µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/composer.json: Size and modification time the same (differ by -581.262µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/.htaccess: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/.gitignore: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/composer.phar: Size and modification time the same (differ by -867.142µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/composer.phar: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/composer.json: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/httpd-access_log: Size and modification time the same (differ by -263.351µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/httpd-access_log: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/composer.lock: Size and modification time the same (differ by -250.958µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/composer.lock: Unchanged skipping
2022/06/27 11:43:59 DEBUG : site2-php/httpd-error_log: Size and modification time the same (differ by -915.27µs, within tolerance 1ms)
2022/06/27 11:43:59 DEBUG : site2-php/httpd-error_log: Unchanged skipping
... More of the same ...
2022/06/27 11:45:26 DEBUG : site1/public/js/jquery/jquery-3.5.1.min.js: Size and modification time the same (differ by -975.299µs, within tolerance 1ms)
2022/06/27 11:45:26 DEBUG : site1/public/js/jquery/jquery-3.5.1.min.js: Unchanged skipping
2022/06/27 11:45:26 DEBUG : pacer: low level retry 2/2 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded)
2022/06/27 11:45:26 DEBUG : pacer: Rate limited, increasing sleep to 1.342197294s
limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded
2022/06/27 11:45:28 DEBUG : pacer: Reducing sleep to 0s
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-300x300.png: Size and modification time the same (differ by -378.949µs, within tolerance 1ms)
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-300x300.png: Unchanged skipping
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-300x300.webp: Size and modification time the same (differ by -799.394µs, within tolerance 1ms)
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-300x300.webp: Unchanged skipping
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-512x512.png: Size and modification time the same (differ by -842.156µs, within tolerance 1ms)
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-512x512.png: Unchanged skipping
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-512x512.webp: Size and modification time the same (differ by -275.347µs, within tolerance 1ms)
2022/06/27 11:45:28 DEBUG : site3.com/http/images/site-images/icon-512x512.webp: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site1/public/images/features/backup_local/.gitinclude: Size and modification time the same (differ by -319.965µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site1/public/images/features/backup_local/.gitinclude: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site1/---resources/_tools/get_tiles/make_info/zip/.DS_Store: Size and modification time the same (differ by -645.845µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site1/---resources/_tools/get_tiles/make_info/zip/.DS_Store: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/map.twig: Size and modification time the same (differ by -536.094µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/map.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/won.twig: Size and modification time the same (differ by -794.02µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/won.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/view.twig: Size and modification time the same (differ by -887.261µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/contestss/view.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/change-password.twig: Size and modification time the same (differ by -418.168µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/change-password.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/login.twig: Size and modification time the same (differ by -603.997µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/login.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/signup.twig: Size and modification time the same (differ by -985.713µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/signup.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/social-signin.twig: Size and modification time the same (differ by -329.841µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/social-signin.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/verify-email.twig: Size and modification time the same (differ by -494.239µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site2-php/www/src/View/user/verify-email.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : pacer: low level retry 1/2 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded)
2022/06/27 11:45:30 DEBUG : pacer: Rate limited, increasing sleep to 1.615633s
2022/06/27 11:45:30 DEBUG : pacer: Reducing sleep to 0s
2022/06/27 11:45:30 DEBUG : site1/src/Templates/common/en/404.twig: Size and modification time the same (differ by -951.93µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site1/src/Templates/common/en/404.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : site1/src/Templates/common/sv/404.twig: Size and modification time the same (differ by -238.558µs, within tolerance 1ms)
2022/06/27 11:45:30 DEBUG : site1/src/Templates/common/sv/404.twig: Unchanged skipping
2022/06/27 11:45:30 DEBUG : pacer: low level retry 2/2 (error googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded)
2022/06/27 11:45:30 DEBUG : pacer: Rate limited, increasing sleep to 1.583425172s
2022/06/27 11:45:30 ERROR : site1/src/Templates/www/en: error reading destination directory: couldn't list directory: googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded
2022/06/27 11:45:32 DEBUG : pacer: Reducing sleep to 0s
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/home.twig: Size and modification time the same (differ by -735.485µs, within tolerance 1ms)
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/home.twig: Unchanged skipping
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/qr_popover.twig: Size and modification time the same (differ by -288.711µs, within tolerance 1ms)
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/qr_popover.twig: Unchanged skipping
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/user-data-removal-status.twig: Size and modification time the same (differ by -658.654µs, within tolerance 1ms)
2022/06/27 11:45:32 DEBUG : site1/src/Templates/www/sv/user-data-removal-status.twig: Unchanged skipping
2022/06/27 11:45:34 DEBUG : site1/src/Templates/www/sv/base/footer.twig: Size and modification time the same (differ by -72.286µs, within tolerance 1ms)
2022/06/27 11:45:34 DEBUG : site1/src/Templates/www/sv/base/footer.twig: Unchanged skipping
2022/06/27 11:45:34 DEBUG : site1/src/Templates/www/sv/base/header.twig: Size and modification time the same (differ by -320.508µs, within tolerance 1ms)
2022/06/27 11:45:34 DEBUG : site1/src/Templates/www/sv/base/header.twig: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/.gitinclude: Size and modification time the same (differ by -513.472µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/0030c7bbff4e0bed61e6e1fdc2d08566: Size and modification time the same (differ by -474.097µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/0030c7bbff4e0bed61e6e1fdc2d08566: Unchanged skipping
... More of the same ...
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07bdc85f2eb59d309295a79f4461f17a: Size and modification time the same (differ by -571.457µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07bdc85f2eb59d309295a79f4461f17a: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07c8c72366861900b36e80e14a986c6d: Size and modification time the same (differ by -558.673µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07c8c72366861900b36e80e14a986c6d: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07fabfe7da58104be1df800114ef1957: Size and modification time the same (differ by -781.346µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/07fabfe7da58104be1df800114ef1957: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08207e0f5070fb028360ea7a76cfb76e: Size and modification time the same (differ by -885.102µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08207e0f5070fb028360ea7a76cfb76e: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/083cef04a392c55f81fa45210c56d42c: Size and modification time the same (differ by -638.631µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/083cef04a392c55f81fa45210c56d42c: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/084739d05a037ee826fc989e5d3ba708: Size and modification time the same (differ by -850.886µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/084739d05a037ee826fc989e5d3ba708: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/0853fa9ca5dd995bac2b59a13fbb82bf: Size and modification time the same (differ by -822.766µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/0853fa9ca5dd995bac2b59a13fbb82bf: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/087be77ec64d77159aa463245b024deb: Size and modification time the same (differ by -192.758µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/087be77ec64d77159aa463245b024deb: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08c79d73e54fa26a2349f34babd7987e: Size and modification time the same (differ by -477.044µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08c79d73e54fa26a2349f34babd7987e: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/089f81f858704b17b04533e5c0f90d05: Size and modification time the same (differ by -471.36µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/089f81f858704b17b04533e5c0f90d05: Unchanged skipping
2022/06/27 11:45:42 DEBUG : Google drive root 'PhpstormProjects': Waiting for checks to finish
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08efa616ac4d249a8fa95f345225cc89: Size and modification time the same (differ by -295.404µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/08efa616ac4d249a8fa95f345225cc89: Unchanged skipping
... More of the same ...
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/ffe60573d544c4965810f16cc85a4689: Size and modification time the same (differ by -125.347µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/ff4ad27f0f23dcb5f0d6675f9e575827: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/ffe60573d544c4965810f16cc85a4689: Unchanged skipping
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/ffefcbe1298bfde366e5ce6269339ba5: Size and modification time the same (differ by -26.479µs, within tolerance 1ms)
2022/06/27 11:45:42 DEBUG : site1/public/images/features/download/ffefcbe1298bfde366e5ce6269339ba5: Unchanged skipping
2022/06/27 11:45:42 DEBUG : Google drive root 'PhpstormProjects': Waiting for transfers to finish
2022/06/27 11:45:42 ERROR : Google drive root 'PhpstormProjects': not deleting files as there were IO errors
2022/06/27 11:45:42 ERROR : Google drive root 'PhpstormProjects': not deleting directories as there were IO errors
2022/06/27 11:45:42 INFO : There was nothing to transfer
2022/06/27 11:45:42 ERROR : Attempt 2/2 failed with 10 errors and: march failed with 9 error(s): first error: couldn't list directory: googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded
2022/06/27 11:45:42 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 10 (retrying may help)
Checks: 4230 / 4230, 100%
Elapsed time: 0.0s
2022/06/27 11:45:42 DEBUG : 12 go routines active
2022/06/27 11:45:42 Failed to sync with 10 errors: last error was: march failed with 9 error(s): first error: couldn't list directory: googleapi: Error 403: User Rate Limit Exceeded. Rate of requests for user exceed configured project quota. You may consider re-evaluating expected per-user traffic to the API and adjust project quota limits accordingly. You may monitor aggregate quota usage and adjust limits in the API Console: https://console.developers.google.com/apis/api/drive.googleapis.com/quotas?project=815672436659, userRateLimitExceeded