Followup on the older topic : Rclone Invokes more number of WorkDrive’s files listing API calls which exceeds the throttling limit

[cselvakumar (Selvakumar Chokkalingam)]

What is the problem you are having with rclone?

Need to discuss further the topic that we already raised in the below forum

Rclone Invokes more number of WorkDrive's files listing API calls which exceeds the throttling limit

What is the problem you are having with rclone? More Rclone users are complaining about the file syncing using the rclone to WorkDrive and reaching WorkDrive support. I am a developer from Zoho WorkDrive and handled such customers in our support. While analyzing we have found that rclone is making repeated files/<folder_id>/files calls in a short span of time with the same folder id, with the same pagination parameters for the same user. In a valid use case, users aren't required to fetch the f…

As the above thread was automatically closed after 30 days, we are unable to continue the discussion further in that thread. Sorry we have missed to respond to that thread within 30 days.

We have sent a mail to "nick@craig-wood.com" on "26 Dec 2023" on this, but we haven't got any reply.

Here is our reply to Nick, for the last comment in that thread.

`Hi Nick,

Please to meet you.

Currently, WorkDrive doesn't have the API limits based on the account or its license so there is no need to upgrade your account to increase the API usage.

WorkDrive enforced stricter API limits for the API calls requesting the same page("page%5Blimit%5D=10&page%5Boffset%5D=0") in the files listing API "/api/v1/files/<folder_id>/files". But if you pass the different page numbers as you mentioned in one of your screenshots the requests should not throttle.

Our current API throttling limit for files listing API "/api/v1/files/<folder_id>/files" is

• If the same pagination numbers passed with the same folder_id from the same user -> only 20 calls will be allowed for 1 minute
• Other than the above case there is no daily limit or account/license specific limit. If you hit the listing API for different folders or the same folder with different pagination values then there shouldn't be any restriction.

As per our understanding, the /rclone app is invoking the files listing API for the base folder for every action that is happening in rclone. Say, if the user did 20 different actions in the same folder (copy/upload 20+ files) for every action the app hits the listing API 20+ times(starting from 1st page to last) within 1 minute timeframe. So it crosses the above-mentioned API threshold limit.

Can we avoid hitting the listing API for each action, instead can we handle the batch actions and hit only once for every batch and make sure it is within the above-mentioned threshold?

If we are missing something, we can discuss further to understand your API usage and enhance this integration.

Including our support email address("support@eu.zohoworkdrive.com") in this mail, for further conversation.`

Run the command 'rclone version' and share the full output of the command.

Not applicable

Which cloud storage system are you using? (eg Google Drive)

Zoho WorkDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Not applicable

The rclone config contents with secrets removed.

Not applicable

A log from the command with the -vv flag

Not applicable

[ncw (Nick Craig-Wood)]

@cselvakumar

As per our understanding, the /rclone app is invoking the files listing API for the base folder for every action that is happening in rclone.

I figured out why rclone is doing this...

When it uploads a file, it wants to confirm the file has been uploaded properly - rclone is all about the verification.

Currently it does this by listing the directory to find the file after the upload. This is super inefficient, however we don't have to do this for most cloud providers.

Does Zoho have an API to find a file by name in a directory without doing a listing?

The rclone code suggests it doesn't, or at least didn't when the code was written in 2020. I had a quick look through the workdrive docs and I couldn't find such an API.

If the folder list API had a search file name parameter to look for we could use that (that is how a lot of the other providers do it).

Questions for you:

1. Can you suggest an API which shows how to get info about a single file in a directory by name given a directory id? If so then we can solve this inefficiency

2. We are currently using 10 for the page size of directory listings. This seems very low (eg S3/Drive/Box use 1000). Should we increase it to the maximum of 50?

Thanks!

[cselvakumar (Selvakumar Chokkalingam)]

@ncw

When it uploads a file, it wants to confirm the file has been uploaded properly - rclone is all about the verification.

The file upload confirmation can be done based on the response status of that particular upload request itself. Is there any specific need to confirm it in a separate call?

If you want to find a file using its name in a particular directory, you can use the below search API.

Zoho WorkDrive API Documentation

https://www.zohoapis.com/workdrive/api/v1/teams/{team_id}/records?search[name]={query}&filter[parentId]={parentfolder_id}&page[limit]=50&page[offset]=0

@ncw

1. We are currently using 10 for the page size of directory listings. This seems very low (eg S3/Drive/Box use 1000). Should we increase it to the maximum of 50?

Yes, you can increase the limit to 50. As per our above discussions please refrain from using this listing API for confirmation of the upload. Use the listing API with 50 as the limit where it is required to get the list of files other than the upload confirmation.

Please reach us at "support@eu.zohoworkdrive.com" for further communication on this.

[ncw (Nick Craig-Wood)]

@cselvakumar

If you want to find a file using its name in a particular directory, you can use the below search API.

Zoho WorkDrive API Documentation

https://www.zohoapis.com/workdrive/api/v1/teams/{team_id}/records?search[name]={query}&filter[parentId]={parentfolder_id}&page[limit]=50&page[offset]=0

I had a go with this.

Unfortunately this needs a new scope which rclone didn't ask for originally
ZohoSearch.securesearch.READ so using this API means all rclone users will have to re-login. I guess that can't be helped.

However I couldn't get the API to work properly it kept giving {"errors":[{"id":"R602","title":"Invalid parent Id"}]} when I pass the root ID.

2024/03/11 11:24:44 DEBUG : HTTP REQUEST (req 0xc000b16100)
2024/03/11 11:24:44 DEBUG : GET /api/v1/teams/4xrzuf5bbda0ee9724864a75bd4b8408a6d88/records?filter%5BparentId%5D=4xrzu9ece9a669c3c4d38a009ca405a67cce9&page%5Blimit%5D=1&page%5Boffset%5D=0&search%5Bname%5D=50.txt HTTP/1.1
Host: workdrive.zoho.eu
User-Agent: rclone/v1.67.0-beta.7775.024a27ecf.fix-zoho
Accept: application/vnd.api+json
Authorization: XXXX
Accept-Encoding: gzip

And the reply

2024/03/11 11:24:44 DEBUG : HTTP RESPONSE (req 0xc000b16100)
2024/03/11 11:24:44 DEBUG : HTTP/2.0 500 Internal Server Error
Cache-Control: private,no-cache,no-store,max-age=0,must-revalidate
Content-Type: application/vnd.api+json;charset=UTF-8
Date: Mon, 11 Mar 2024 11:24:44 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Server: ZGS
Set-Cookie: f32142af9e=XXX; Path=/
Set-Cookie: zpcc=XXX;path=/;SameSite=None;Secure;priority=high
Set-Cookie: _zcsr_tmp=XXX;path=/;SameSite=Strict;Secure;priority=high
Set-Cookie: JSESSIONID=XXX; Path=/
Vary: accept-encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{"errors":[{"id":"R602","title":"Invalid parent Id"}]}

A 500 error doesn't look good. When I leave out the filter[parentId] then the query works and returns lots of stuff.

2024/03/11 11:30:28 DEBUG : HTTP REQUEST (req 0xc0008cf500)
2024/03/11 11:30:28 DEBUG : GET /api/v1/teams/4xrzuf5bbda0ee9724864a75bd4b8408a6d88/records?page%5Blimit%5D=1&page%5Boffset%5D=0&search%5Bname%5D=50.txt HTTP/1.1
Host: workdrive.zoho.eu
User-Agent: rclone/v1.67.0-beta.7775.024a27ecf.fix-zoho
Accept: application/vnd.api+json
Authorization: XXXX
Accept-Encoding: gzip

Gives

2024/03/11 11:30:28 DEBUG : HTTP RESPONSE (req 0xc0008cf500)
2024/03/11 11:30:28 DEBUG : HTTP/2.0 200 OK
Cache-Control: private,no-cache,no-store,max-age=0,must-revalidate
Content-Type: application/vnd.api+json;charset=UTF-8
Date: Mon, 11 Mar 2024 11:30:28 GMT
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Server: ZGS
Set-Cookie: f32142af9e=XXX; Path=/
Set-Cookie: zpcc=XXX;path=/;SameSite=None;Secure;priority=high
Set-Cookie: _zcsr_tmp=XXX;path=/;SameSite=Strict;Secure;priority=high
Set-Cookie: JSESSIONID=XXX; Path=/
Strict-Transport-Security: max-age=63072000
Vary: accept-encoding
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

{"data":[{"id":"1blhh19fa53470f634704967f41e51d5f888e",...
...
"meta":{"search_result_count":1}}

@cselvakumar

Please reach us at "support@eu.zohoworkdrive.com" for further communication on this.

I will try emailing this to there too.

[cselvakumar (Selvakumar Chokkalingam)]

@ncw

Unfortunately this needs a new scope which rclone didn't ask for originally
ZohoSearch.securesearch.READ so using this API means all rclone users will have to re-login. I guess that can't be helped.

Yes, "ZohoSearch.securesearch.READ" scope is required to access the search-related APIs in WorkDrive. If you can suggest all your users to re-login once and include this scope you can use this API to search the file.

@ncw

However I couldn't get the API to work properly it kept giving {"errors":[{"id":"R602","title":"Invalid parent Id"}]} when I pass the root ID.

The parentId you are passing seems to be the TeamFolder ID("Root Folder ID"), in this API you are supposed to pass the Folder IDs("Subfolders of a Team Folder"), which is why you are getting this error. If you are ok with adding the ZohoSearch-related API scopes as mentioned in the above point. Please let us know, we will try to support the TeamFolder ID also in this parentId, and after that, you should be able to pass the TeamFolder ID too.

Coming back to the main point

@cselvakumar

The file upload confirmation can be done based on the response status of that particular upload request itself. Is there any specific need to confirm it in a separate call?

Can you explain the real need for using the listing API or search API to confirm the upload? As I mentioned earlier that can be confirmed using the response status of the upload call. Can you explain the /rcolne workflow to understand the need for this?