Second followup on the older topic : Rclone Invokes more number of WorkDrive’s files listing API calls which exceeds the throttling limit

What is the problem you are having with rclone?

Need to discuss further the topic that we already raised in the below forum

We haven't got a reply from "nick@craig-wood.com" for the last comment made by us in the above topic, and that thread got closed hence making it as a new topic to discuss on this further.

Here is our reply to Nick, for the last comment in that thread.

Yes, "ZohoSearch.securesearch.READ" scope is required to access the search-related APIs in WorkDrive. If you can suggest all your users to re-login once and include this scope you can use this API to search the file.

The parentId you are passing seems to be the TeamFolder ID("Root Folder ID"), in this API you are supposed to pass the Folder IDs("Subfolders of a Team Folder"), which is why you are getting this error. If you are ok with adding the ZohoSearch-related API scopes as mentioned in the above point. Please let us know, we will try to support the TeamFolder ID also in this parentId, and after that, you should be able to pass the TeamFolder ID too.

Coming back to the main point

Can you explain the real need for using the listing API or search API to confirm the upload? As I mentioned earlier that can be confirmed using the response status of the upload call. Can you explain the /rcolne workflow to understand the need for this?

Run the command 'rclone version' and share the full output of the command.

not applicable

Which cloud storage system are you using? (eg Google Drive)

Zoho WorkDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

not applicable

The rclone config contents with secrets removed.

not applicable

A log from the command with the -vv flag

not applicable

Hello @ncw,
Hope you are doing good.

As mentioned by @cselvakumar , files listing APIs are called many times within a short span of time and it leads to rate limiting.
Find my test log below:

cybersa@ptpll740:/data/softwares/rclone-v1.66.0-linux-amd64$ ./rclone --config rcloneApp.conf --dump headers copy '/home/cybersa/Pictures/temp' workdrive2: 2>&1 | grep -E '(HTTP/1.1|HTTP/2.0 429)'
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempdojewas3&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempnaribuf8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTemphimusij0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempjanoyek2&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:17 DEBUG : GET /api/v1/files/6ocjh4e4d1a1939b74c3098d9a4f41fbc0c28 HTTP/1.1
2024/05/12 11:46:17 DEBUG : GET /api/v1/files/6ocjh89fb697fac2e4b6b974a98759142cac5 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7ccf94ab42554475a0b960a986879267 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh89fb697fac2e4b6b974a98759142cac5 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh7ccf94ab42554475a0b960a986879267 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh4e4d1a1939b74c3098d9a4f41fbc0c28 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTempyirobuf8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTempsuzubaz1&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjhddbbcfbed3544330a9a65c655cf4820b HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTemplodanic7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhddbbcfbed3544330a9a65c655cf4820b HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjhd02ff3aee12947d6aebe72fddfcb1727 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhd02ff3aee12947d6aebe72fddfcb1727 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh6622428b88824b99aa4e2a0a33b9e9f3 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjhfa1ab722b6404fae8447a424574bd0c8 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempjalaqoq5&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjh6622428b88824b99aa4e2a0a33b9e9f3 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhfa1ab722b6404fae8447a424574bd0c8 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempbogahoc8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh0e5b1cb807ba47b58ace92d2c5799bac HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempcobiruh0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempvijucet9&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:20 DEBUG : PATCH /api/v1/files/6ocjh0e5b1cb807ba47b58ace92d2c5799bac HTTP/1.1
2024/05/12 11:46:20 DEBUG : GET /api/v1/files/6ocjhc73b64e2455040939ad67ed21a881ca8 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh78fcf9150b4e464ab2cdb1c8a2f5bff9 HTTP/1.1
2024/05/12 11:46:21 DEBUG : PATCH /api/v1/files/6ocjh78fcf9150b4e464ab2cdb1c8a2f5bff9 HTTP/1.1
2024/05/12 11:46:21 DEBUG : PATCH /api/v1/files/6ocjhc73b64e2455040939ad67ed21a881ca8 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : POST /api/v1/upload?filename=rcloneTempzifejes4&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh5fe4bfd5b3544afcb804f3b26168ef71 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:22 DEBUG : POST /api/v1/upload?filename=rcloneTempdobikib7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:22 DEBUG : POST /api/v1/upload?filename=rcloneTemphegeloq7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:22 DEBUG : PATCH /api/v1/files/6ocjh5fe4bfd5b3544afcb804f3b26168ef71 HTTP/1.1
2024/05/12 11:46:22 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:22 DEBUG : GET /api/v1/files/6ocjh13ee0b1253d34c18a827599eb7740fdd HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjhce7a9da757104a20a9e1500269822447 HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjh13ee0b1253d34c18a827599eb7740fdd HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjhce7a9da757104a20a9e1500269822447 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTempxamarur0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjhf21735bd6f1048bf83e73cbf10996b3d HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjhf21735bd6f1048bf83e73cbf10996b3d HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTemphemetol8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTempfomanuj8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh203caf61578040938b46b2c906441362 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh29251312f73b4f64b4b418aa80a26efc HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh3af62ccd11524f9991102e7fde010f15 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh29251312f73b4f64b4b418aa80a26efc HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh203caf61578040938b46b2c906441362 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh3af62ccd11524f9991102e7fde010f15 HTTP/1.1
2024/05/12 11:46:24 DEBUG : POST /api/v1/upload?filename=rcloneTemppejusuw4&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjhbaa59633381e465a8f5c4b0a7458d53d HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjhbaa59633381e465a8f5c4b0a7458d53d HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : POST /api/v1/upload?filename=rcloneTemptozurib2&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:25 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjhb5177e7f80434535a6d4913ecbe20061 HTTP/1.1
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:25 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:26 DEBUG : PATCH /api/v1/files/6ocjhb5177e7f80434535a6d4913ecbe20061 HTTP/1.1
2024/05/12 11:46:27 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:27 DEBUG : HTTP/2.0 429 Too Many Requests

I'm trying to copy 20 files from my local into a empty folder in the workdrive. If you see the above HTTP request logs, files listing API was called 5 times without any change in the offset. Then it starts to upload the file with temporary file name. After that, File metadata API was called to read the file size. I guess this is to verify the upload status. Then File rename API was called to rename the file name to it's original name. Again Files listing API was called. I'm not sure why it was called again. Because we already verified the upload status by calling File Metadata API.

Let me know if you need any other details. BTW, I'm using latest version of Rclone.

Thanks.

Hello @ncw,
I went through your comments in the old discussion.

We are currently using 10 for the page size of directory listings. This seems very low (eg S3/Drive/Box use 1000). Should we increase it to the maximum of 50?

List files API also supports cursor pagination and we can use it to fetch 1000 items instead of 50 limit in the offset pagination. We are not using it. Is there any reason for it?
More info:

Thanks.

Hello @ncw ,
New Update on the above cursor pagination comment.
I tried to implement the files listing API using cursor pagination. Comparing rclone:master...cybersa:zoho-fix-listAll-ratelimit-issue · rclone/rclone · GitHub

Cursor pagination is working fine but it does not fix the rate limiting issue which we are facing. Seems like File Listing API has very low ratelimit configuration. I tried to hit the listing API every 1 sec and its fails after 18-20 requests.

So I guess we are forced to implement Search File API to fix this issue. Also I confirmed that Search API does not have this very low rate limit configuration as like listing API.


BTW, Cursor Pagination helps in avoiding one extra API call when compared to Offset pagination. Because with current offset pagination implementation , we make one extra API call even though listed items is less than pagination limit in the last API call response.