Second followup on the older topic : Rclone Invokes more number of WorkDrive’s files listing API calls which exceeds the throttling limit

cselvakumar · April 24, 2024, 4:21am

What is the problem you are having with rclone?

Need to discuss further the topic that we already raised in the below forum

We haven't got a reply from "nick@craig-wood.com" for the last comment made by us in the above topic, and that thread got closed hence making it as a new topic to discuss on this further.

Here is our reply to Nick, for the last comment in that thread.

Yes, "ZohoSearch.securesearch.READ" scope is required to access the search-related APIs in WorkDrive. If you can suggest all your users to re-login once and include this scope you can use this API to search the file.

The parentId you are passing seems to be the TeamFolder ID("Root Folder ID"), in this API you are supposed to pass the Folder IDs("Subfolders of a Team Folder"), which is why you are getting this error. If you are ok with adding the ZohoSearch-related API scopes as mentioned in the above point. Please let us know, we will try to support the TeamFolder ID also in this parentId, and after that, you should be able to pass the TeamFolder ID too.

Coming back to the main point

Can you explain the real need for using the listing API or search API to confirm the upload? As I mentioned earlier that can be confirmed using the response status of the upload call. Can you explain the /rcolne workflow to understand the need for this?

Run the command 'rclone version' and share the full output of the command.

not applicable

Which cloud storage system are you using? (eg Google Drive)

Zoho WorkDrive

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

not applicable

The rclone config contents with secrets removed.

not applicable

A log from the command with the `-vv` flag

not applicable

cybersa · May 12, 2024, 6:18am

Hello @ncw,
Hope you are doing good.

As mentioned by @cselvakumar , files listing APIs are called many times within a short span of time and it leads to rate limiting.
Find my test log below:

cybersa@ptpll740:/data/softwares/rclone-v1.66.0-linux-amd64$ ./rclone --config rcloneApp.conf --dump headers copy '/home/cybersa/Pictures/temp' workdrive2: 2>&1 | grep -E '(HTTP/1.1|HTTP/2.0 429)'
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempdojewas3&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempnaribuf8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTemphimusij0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:16 DEBUG : POST /api/v1/upload?filename=rcloneTempjanoyek2&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:17 DEBUG : GET /api/v1/files/6ocjh4e4d1a1939b74c3098d9a4f41fbc0c28 HTTP/1.1
2024/05/12 11:46:17 DEBUG : GET /api/v1/files/6ocjh89fb697fac2e4b6b974a98759142cac5 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7ccf94ab42554475a0b960a986879267 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh89fb697fac2e4b6b974a98759142cac5 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh7ccf94ab42554475a0b960a986879267 HTTP/1.1
2024/05/12 11:46:18 DEBUG : PATCH /api/v1/files/6ocjh4e4d1a1939b74c3098d9a4f41fbc0c28 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTempyirobuf8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTempsuzubaz1&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:18 DEBUG : GET /api/v1/files/6ocjhddbbcfbed3544330a9a65c655cf4820b HTTP/1.1
2024/05/12 11:46:18 DEBUG : POST /api/v1/upload?filename=rcloneTemplodanic7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhddbbcfbed3544330a9a65c655cf4820b HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjhd02ff3aee12947d6aebe72fddfcb1727 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhd02ff3aee12947d6aebe72fddfcb1727 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh6622428b88824b99aa4e2a0a33b9e9f3 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjhfa1ab722b6404fae8447a424574bd0c8 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempjalaqoq5&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjh6622428b88824b99aa4e2a0a33b9e9f3 HTTP/1.1
2024/05/12 11:46:19 DEBUG : PATCH /api/v1/files/6ocjhfa1ab722b6404fae8447a424574bd0c8 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempbogahoc8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:19 DEBUG : GET /api/v1/files/6ocjh0e5b1cb807ba47b58ace92d2c5799bac HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempcobiruh0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:19 DEBUG : POST /api/v1/upload?filename=rcloneTempvijucet9&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:20 DEBUG : PATCH /api/v1/files/6ocjh0e5b1cb807ba47b58ace92d2c5799bac HTTP/1.1
2024/05/12 11:46:20 DEBUG : GET /api/v1/files/6ocjhc73b64e2455040939ad67ed21a881ca8 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh78fcf9150b4e464ab2cdb1c8a2f5bff9 HTTP/1.1
2024/05/12 11:46:21 DEBUG : PATCH /api/v1/files/6ocjh78fcf9150b4e464ab2cdb1c8a2f5bff9 HTTP/1.1
2024/05/12 11:46:21 DEBUG : PATCH /api/v1/files/6ocjhc73b64e2455040939ad67ed21a881ca8 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : POST /api/v1/upload?filename=rcloneTempzifejes4&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh5fe4bfd5b3544afcb804f3b26168ef71 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:21 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:22 DEBUG : POST /api/v1/upload?filename=rcloneTempdobikib7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:22 DEBUG : POST /api/v1/upload?filename=rcloneTemphegeloq7&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:22 DEBUG : PATCH /api/v1/files/6ocjh5fe4bfd5b3544afcb804f3b26168ef71 HTTP/1.1
2024/05/12 11:46:22 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:22 DEBUG : GET /api/v1/files/6ocjh13ee0b1253d34c18a827599eb7740fdd HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjhce7a9da757104a20a9e1500269822447 HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjh13ee0b1253d34c18a827599eb7740fdd HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjhce7a9da757104a20a9e1500269822447 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTempxamarur0&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjhf21735bd6f1048bf83e73cbf10996b3d HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:23 DEBUG : PATCH /api/v1/files/6ocjhf21735bd6f1048bf83e73cbf10996b3d HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTemphemetol8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : POST /api/v1/upload?filename=rcloneTempfomanuj8&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:23 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh203caf61578040938b46b2c906441362 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh29251312f73b4f64b4b418aa80a26efc HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh3af62ccd11524f9991102e7fde010f15 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh29251312f73b4f64b4b418aa80a26efc HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh203caf61578040938b46b2c906441362 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjh3af62ccd11524f9991102e7fde010f15 HTTP/1.1
2024/05/12 11:46:24 DEBUG : POST /api/v1/upload?filename=rcloneTemppejusuw4&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=10 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjhbaa59633381e465a8f5c4b0a7458d53d HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=20 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : PATCH /api/v1/files/6ocjhbaa59633381e465a8f5c4b0a7458d53d HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : POST /api/v1/upload?filename=rcloneTemptozurib2&override-name-exist=true&parent_id=6ocjh7eb30207146e4ce08c056e8385aa6865 HTTP/1.1
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:24 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:24 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:25 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjhb5177e7f80434535a6d4913ecbe20061 HTTP/1.1
2024/05/12 11:46:25 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:25 DEBUG : HTTP/2.0 429 Too Many Requests
2024/05/12 11:46:26 DEBUG : PATCH /api/v1/files/6ocjhb5177e7f80434535a6d4913ecbe20061 HTTP/1.1
2024/05/12 11:46:27 DEBUG : GET /api/v1/files/6ocjh7eb30207146e4ce08c056e8385aa6865/files?page%5Blimit%5D=10&page%5Boffset%5D=0 HTTP/1.1
2024/05/12 11:46:27 DEBUG : HTTP/2.0 429 Too Many Requests

I'm trying to copy 20 files from my local into a empty folder in the workdrive. If you see the above HTTP request logs, files listing API was called 5 times without any change in the offset. Then it starts to upload the file with temporary file name. After that, File metadata API was called to read the file size. I guess this is to verify the upload status. Then File rename API was called to rename the file name to it's original name. Again Files listing API was called. I'm not sure why it was called again. Because we already verified the upload status by calling File Metadata API.

Let me know if you need any other details. BTW, I'm using latest version of Rclone.

Thanks.

cybersa · May 20, 2024, 10:58am

Hello @ncw,
I went through your comments in the old discussion.

We are currently using 10 for the page size of directory listings. This seems very low (eg S3/Drive/Box use 1000). Should we increase it to the maximum of 50?

List files API also supports cursor pagination and we can use it to fetch 1000 items instead of 50 limit in the offset pagination. We are not using it. Is there any reason for it?
More info:

Thanks.

cybersa · May 22, 2024, 5:05am

Hello @ncw ,
New Update on the above cursor pagination comment.
I tried to implement the files listing API using cursor pagination. Comparing rclone:master...cybersa:zoho-fix-listAll-ratelimit-issue · rclone/rclone · GitHub

Cursor pagination is working fine but it does not fix the rate limiting issue which we are facing. Seems like File Listing API has very low ratelimit configuration. I tried to hit the listing API every 1 sec and its fails after 18-20 requests.

So I guess we are forced to implement Search File API to fix this issue. Also I confirmed that Search API does not have this very low rate limit configuration as like listing API.

BTW, Cursor Pagination helps in avoiding one extra API call when compared to Offset pagination. Because with current offset pagination implementation , we make one extra API call even though listed items is less than pagination limit in the last API call response.

cybersa · May 24, 2024, 6:07am

Hello @ncw

New Update:

I went through the rclone code. I guess this is the flow for copying files from local into workdrive.

File listing API is called to filter the files based on files needs to be uploaded and files needs to be skipped (Which are already in the workdrive)
Then Upload process starts from the filtered files list.
Before uploading the particular file, rclone check the workdrive again using listing API to make sure the file which are going to be uploaded are not in the drive.
If it is not there, it will do the upload process with overwrite option.
if it is there, it will call the update instead of upload process. But in the update fucn will call the upload func with overwrite option.

Steps from 3 to 5 are repeated until all the filtered files are uploaded.

Code Reference:
Put func which calls Upload or Update after listing API is called: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub
Upload func always do the overwrite: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub
Update func always going to call the upload func: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub

So there is no use in calling listing files API before uploading each file.

Solution:
Skip calling Listing API before uploading for each file and Update func can be removed since it is called only from Put func.

cselvakumar · May 29, 2024, 1:00pm

@cybersa Thank you for your detailed analysis in Rclone and for providing the insights.

To answer you query

Ans : As I already mentioned in one of my earlier reply

WorkDrive's Current API throttling limit for files listing API "/api/v1/files/<folder_id>/files" is

If the same pagination numbers passed with the same folder_id from the same user -> only 20 calls will be allowed for 1 minute
Other than the above case there is no daily limit or account/license specific limit. If you hit the listing API for different folders or the same folder with different pagination values then there shouldn't be any restriction.

As you stated, calling the files listing query with the same page numbers multiple times within a short span of time is the cause of the problem.

@ncw Can you check this and revisit the upload confirmation logic and reduce the files listing API calls from /rclone service?

please reply to this thread or get back to us at support@zohoworkdrive.com

decarbonise · June 6, 2024, 10:01am

This Workdrive issue is a massive pain point for us. @cselvakumar is there a timeline on getting a fix? Zoho TrueSync is not supported on Linux so Rclone is our only option. If Zoho is going to rely on the fantastic open source solution that is Rclone, perhaps it should contribute more actively to the development of the Zoho integration, or fund other developers to do so?

asdffdsa · June 6, 2024, 11:55am

welcome to the forum,

great point, i have been thinking that, ever since zoho started posting in the forum

zoho should sponsor the project, same as the other providers have done...

ncw · June 11, 2024, 5:06pm

The problem is caused here by the Put function.

It checks to see if the file exists - if it doesn't then it puts it directly, if it does it calls Update.

Unfortunately the only way to check whether a file exists or not is to do a listing.

Possible solutions

Use a different API for the call
- This needs a new permission which means all users will have to login
- The API call didn't work on the root folder (may be fixed now)
Do the upload anyway so make Put not check for existing files as suggested by @cybersa - that would probably work fine. Checking that the file exists already stops rclone uploading duplicate names but I don't think this is a problem with Zoho
Cache the listing read so we don't have to do it multiple times in a row.

Option 2 sounds like the best option but needs testing...

ncw · June 11, 2024, 8:26pm

OK I had a go at fixing some of these problems

zoho: sleep for 60 seconds if rate limit error received
zoho: remove simple file names complication which is no longer needed
zoho: retry reading info if size wasn't returned
zoho: fix throttling problem when uploading files
zoho: use cursor listing for improved performance

Combined these allow me to upload 1000 small files just fine. I discovered that zoho has rate limiting on the upload URL too so if you upload files too fast you'll get rate limited!

The first commit, sleeping for 60s if we get a rate limit error is a bit of a sledghammer to crack a nut but it does seem to allow the operations to complete. I may/may not leave it in depending on feedback.

The problem with zoho's rate limits is that once you've hit them, you then have to wait about 5 minutes for them to reset. Other providers do the rate limiting more gradually (eg Google Drive) and other providers send a Retry-After header which would be really useful so we know when it is safe to send more requests.

With the 60 second timeout I managed to get the short integration tests to pass which is a step forward!

Please give this a go.

v1.67.0-beta.8015.b1343a534.fix-zoho-nocheck on branch fix-zoho-nocheck (uploaded in 15-30 mins)

decarbonise · June 12, 2024, 11:10am

@ncw thanks for this! Just did a very quick test upload and it didn't trigger the slowdowns it used to

ncw · June 12, 2024, 3:43pm

Thanks for testing @decarbonise

I've decided to merge those fixes for v1.67 which will be out in a few days, so any last minute testing much apprecated!

Debeka · July 9, 2024, 9:22am

Hello @ncw

This is Debeka, a backend developer, from Zoho WorkDrive. We appreciate the changes made with rename flow with file uploads. Yet the listing API can be completely removed for the file upload

As suggested here , there is no specific need for checking the files status [existing / new file] using the listing API. We already have such a provision in file upload / folder creation itself. Also the 60s throttling added in the previous fix might not always work since we have dynamic throttling lock time in place for listing API.

The /upload API in upload(…) is already being triggered with “override-name-exist” header as true which should update the file if it already exists.

We understand this might need testing efforts. However, doing those changes should make the throttling issue retire making it possible for the uploads work seamlessly.

When it uploads a file, it wants to confirm the file has been uploaded properly - rclone is all about the verification.

Regarding this comment in the previous post, you can trust that once you get a resource Id in the response, you can assume that the file is uploaded successfully already and there is no way that the file upload had issue.

Just for you reference, adding the folder creation API with duplicate name check details also here

POST api/v1/files

Request Body Parameters

name : string Mandatory The name of the new folder to be created.
parent_id : string Mandatory The unique ID of the destination folder/team folder.

To create a folder in My Folders, use myfolder_id fetched using step 3 given here as the parent_id.

Adding “checkduplicatename” in the request header with true / false will restrict the folder creation if a folder with same name already exists / allow folder to be created with a date - timestamp appended to the folder name respectively.

Please do your concerns here if any.

ncw · July 9, 2024, 10:15pm

Thanks for your reply @Debeka

I just uploaded 50 files to zoho with latest rclone.

It took 54 GET requests and 51 POST requests

Of these requests there were for each of the 50 files

1 POST request to upload the file
1 GET request to read the file status

And a few other requests.

So rclone is pretty efficient here.

Rclone doesn't do this (I think I made a mistake when I said it did!) and does use the “override-name-exist” header as you can see here

github.com

rclone/rclone/blob/27267547b903fe5e72aaa71cc1f45456661d0422/backend/zoho/zoho.go#L646-L650


      
          func (f *Fs) upload(ctx context.Context, name string, parent string, size int64, in io.Reader, options ...fs.OpenOption) (*api.Item, error) {
          	params := url.Values{}
          	params.Set("filename", name)
          	params.Set("parent_id", parent)
          	params.Set("override-name-exist", strconv.FormatBool(true))

Rclone only does this because the response from the upload call is missing useful information like the size of the file and the modification time Zoho workdrive has applied to it.

The file integrity in Zoho workdrive is quite weak compared to other backends. For example S3 and Google drive both send back an MD5SUM of the file data received which rclone can compare to see if the correct data was received. Checking the size received by Zoho gives a small amount of comfort the file has been received correctly.

Here is what the response from the upload POST looks like.

Should I be reading stuff out of the "File INFO" field? That appears to be a string with encoded JSON in it which has useful info in it, but I can't see it documented on the API docs page for upload: Zoho WorkDrive API Documentation - is it documented somewhere?

{
  "data": [
    {
      "attributes": {
        "Permalink": "https://workdrive.zoho.eu/file/ezxak202f5d869a4049eba3bf2aec300e399a",
        "File INFO": "{\"ORG_ID\":\"4xrzuf5bbda0ee9724864a75bd4b8408a6d88\",\"RESOURCE_ID\":\"ezxak202f5d869a4049eba3bf2aec300e399a\",\"LIBRARY_ID\":\"4xrzu9ece9a669c3c4d38a009ca405a67cce9\",\"PARENT_MODEL_ID\":\"ezxaka2ba0202358f4699ac69a797f4b7e95b\",\"PARENT_ID\":\"ezxaka2ba0202358f4699ac69a797f4b7e95b\",\"RESOURCE_TYPE\":2002,\"WMS_SENT_TIME\":\"1720562158932\",\"TAB_ID\":\"-1\",\"OWNER\":\"20073456615\",\"RESOURCE_GROUP\":\"FILE\",\"PARENT_MODEL_NAME\":\"file\",\"size\":8,\"OPERATION\":\"UPLOAD\",\"EVENT_ID\":\"-1\",\"AUDIT_INFO\":{\"resource\":{\"owner\":\"20073456615\",\"created_time\":1720562158836,\"creator\":\"20073456387\",\"service_type\":4,\"extension\":\"txt\",\"resource_type\":2002,\"name\":\"file50.txt\"},\"parentInfo\":{\"parentName\":\"50files\",\"parentId\":\"ezxaka2ba0202358f4699ac69a797f4b7e95b\",\"parentType\":1001},\"libraryInfo\":{\"libraryName\":\"General\",\"libraryId\":\"4xrzu9ece9a669c3c4d38a009ca405a67cce9\",\"libraryType\":14},\"statusCode\":\"D201\"},\"ZUID\":20073456387,\"TEAM_ID\":\"4xrzuf5bbda0ee9724864a75bd4b8408a6d88\"}",
        "parent_id": "ezxaka2ba0202358f4699ac69a797f4b7e95b",
        "FileName": "file50.txt",
        "resource_id": "ezxak202f5d869a4049eba3bf2aec300e399a"
      },
      "type": "files"
    }
  ]
}

Rclone needs to read the size, to confirm the file got uploaded properly, and the modification time once the file gets uploaded. If this data came back from the upload call, rclone would use it and use one less HTTP transaction.

This is the way google drive and amazon s3 work.

Debeka · July 17, 2024, 9:15am

Hi @ncw

Thanks for the details. This makes us understand the necessity of the listing API for upload confirmation. We are working to provide the required information in the Upload API response. The modified response will hold the file checksum, file size and modified time of the file. Along with this, information on the File INFO Json usage will be added to the API Documentation.

We will update here once the changes are ready to use. Please feel free to add any other such requirement in the response to completely remove the listing API dependency.

ncw · July 17, 2024, 2:15pm

Thank you @Debeka - much appreciated.

decarbonise · August 14, 2024, 11:19am

@Debeka, any updates on the API changes you're working on so that this can progress?

Debeka · August 19, 2024, 7:03am

Hi @decarbonise , we have already rolled out the discussed API changes to use. We are yet to add the File INFO Json usage details in the API documentation. Once this is done, we will update here to proceed.

Saravanapandi · August 19, 2024, 10:30am

Hi, @Debeka & @ncw. You can find the sample Java code snippet to read the upload response below. We will check the possibility of adding the same in our API documentation.

String response = "<Upload API response>";
			JSONObject json = new JSONObject(response);
        String infoStr = json.getJSONArray("data").getJSONObject(0).getJSONObject("attributes").getString("File INFO");
        JSONObject infoJson = new JSONObject(infoStr);
        String checksum = infoJson.getString("MD5_CHECKSUM");
        long size = infoJson.getLong("size");
        long modifiedTime = -1l;
        //Normal upload
        if(infoJson.getJSONObject("AUDIT_INFO").has("resource")) {
          modifiedTime = infoJson.getJSONObject("AUDIT_INFO").getJSONObject("resource").getLong("status_change_time");
        }
        //Revision upload
        else {
          modifiedTime = infoJson.getJSONObject("AUDIT_INFO").getJSONObject("resourceInfo").getLong("status_change_time");
        }
        System.out.println("checksum - " + checksum + " size - " + size + " modifiedTime - " + modifiedTime);

ncw · August 27, 2024, 9:09am

Thanks @Saravanapandi for the update

I've managed to read the size and the modified time out of the return which is all I need and that removes the listing after upload which has been causing all the problems.

What most providers do is return the same thing that you get when you do a listing after you do an upload.

So when I do a listing I get an object like this

type Item struct {
	ID         string `json:"id"`
	Attributes struct {
		Name         string `json:"name"`
		Type         string `json:"type"`
		IsFolder     bool   `json:"is_folder"`
		CreatedTime  Time   `json:"created_time_in_millisecond"`
		ModifiedTime Time   `json:"modified_time_in_millisecond"`
		UploadedTime Time   `json:"uploaded_time_in_millisecond"`
		StorageInfo  struct {
			Size        int64 `json:"size_in_bytes"`
			FileCount   int64 `json:"files_count"`
			FolderCount int64 `json:"folders_count"`
		} `json:"storage_info"`
	} `json:"attributes"`
}

And I'd like exactly that back when I do an upload somewhere in the response. This is how Google Drive, Box and Dropbox work anyway.

What you've done works for me. It is a bit weird as it seems to be internal API stuff so I understand your reluctance to document it.

How stable is this API response?

I noticed the response has a MD5 checksum. Rclone could use this but it doesn't come back in the normal file listings and I don't know how to query it.

Here is a beta to try with the improvements. It does upload much quicker now - it uploaded 400 very small (1-100 bytes) files in about 10s before it throttled which is a great improvement. I guess it is hitting a throttle on the file upload URL now, but it sleeps and retries and gets there eventually in batches of 400 files.

Any feedback from the beta much appreciated. Hopefully this will go into 1.68

v1.68.0-beta.8241.4d4a6da26.fix-zoho-api on branch fix-zoho-api