Second followup on the older topic : Rclone Invokes more number of WorkDrive’s files listing API calls which exceeds the throttling limit

Suspected Bug
6

Hello @ncw

New Update:

I went through the rclone code. I guess this is the flow for copying files from local into workdrive.

  1. File listing API is called to filter the files based on files needs to be uploaded and files needs to be skipped (Which are already in the workdrive)
  2. Then Upload process starts from the filtered files list.
  3. Before uploading the particular file, rclone check the workdrive again using listing API to make sure the file which are going to be uploaded are not in the drive.
  4. If it is not there, it will do the upload process with overwrite option.
  5. if it is there, it will call the update instead of upload process. But in the update fucn will call the upload func with overwrite option.

Steps from 3 to 5 are repeated until all the filtered files are uploaded.

Code Reference:
Put func which calls Upload or Update after listing API is called: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub
Upload func always do the overwrite: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub
Update func always going to call the upload func: rclone/backend/zoho/zoho.go at master · rclone/rclone · GitHub

So there is no use in calling listing files API before uploading each file.

Solution:
Skip calling Listing API before uploading for each file and Update func can be removed since it is called only from Put func.

1 Like