Box bug blocks rclone cleanup

Hi Nick:

Box has an interesting set of bugs that keep rclone cleanup from working (as well as keeping their own web interface from working) if you have a lot in your trash and, in particular, if a lot of the items are in folders owned by someone else but deleted by you.

First, just doing the normal fetch with limit and offset fails, by just sitting there forever with no server-side response. (Their web interface will eventually throw an error, rclone just sits there forever)

The workaround for this is to replace the limit and offset with ?usemarker=true and then the API will return the first page of results.

The second problem is the API will return items that the current account has deleted but that are owned by another account and these items cannot be removed from the trash by the current account. Using the Box API, it is possible to include the owner in the fields being returned and then skip over them or rclone could just attempt to delete them and play through the error and go onto the next row. Unfortunately, there doesn't seem to be any way to sort or filter what is returned so, in order for rclone cleanup to get past this it would have to...

  1. Fetch the trash with ?usemarker=true

  2. Attempt to delete each item returned in that first fetch but continue if the delete fails

  3. At the end of the page, use the marker value to fetch the next page, (instead of offset)


2021/08/13 11:40:45 DEBUG : rclone: Version "v1.56.0-beta.5482.732bc08ce" starting with parameters ["/home/bcxxxxer/rclone/rclone" "-vvv" "--dump" "headers,requests,responses" "cleanup" "joelbox:"]
2021/08/13 11:40:45 DEBUG : Creating backend with remote "joelbox:"
2021/08/13 11:40:45 DEBUG : Using config file from "/hxxxxnf"

2021/08/13 11:41:16 DEBUG : HTTP REQUEST (req 0xc00021ae00)
2021/08/13 11:41:16 DEBUG : GET /2.0/folders/trash/items?fields=type&fields=id&limit=1000&offset=0 HTTP/1.1

hi,

so the bugs are with box, not rclone?

Yes. What rclone does currently SHOULD work and will work in the simplest cases (small number of files in trach all deleted from folders actually located in the client's box account). When there are a large number of files and/or they are deleted from shared folders, Box's own web site fails and so does rclone.

With the workaround to use the markers, both box's web site and the API will at least see the files located in trash. But even Box's "empty trash" button doesn't work and files from shared folders cannot be deleted.

So, it seems that the Rclone Box backend would need two workarounds if we want cleanup to save the day.

have you contacted box and reported these bugs to them?

since this is not a rclone bug, best to change the category from suspected bug to feature
as you want new rclone features to work around multiple bugs with box api/website.

Hi Jojo,

Yes. Like so many Box bugs, Box considers these to be "normal" even though they make it impossible to empty the trash by any mechanism Box supports.

I changed this to a "feature"

-Joel

i do not use box, perhaps this will spark an idea...

"my favourite way of cleaning up directories"
All files get deleted - #6 by brdlgwmpft

Would it be a good idea for rclone to switch to using marker for all listings do you think?

I found the docs here

https://developer.box.com/guides/api-calls/pagination/marker-based/

There doesn't seem to be any downsides as far as I can see.

If we did that then at least rclone would progress!

Do we get an error code we can test against to note that the delete failed because the files was owned by someone else. If we have that then we can make the whole process seamless.

Looks like we do

 curl -i -X DELETE "https://api.box.com/2.0/files/67269120225/trash"    -H "Authorization: Bearer 1REDACTEDmH6I4wth"
HTTP/1.1 403 Forbidden
Date: Sun, 15 Aug 2021 15:41:29 GMT
Content-Type: application/json
Transfer-Encoding: chunked
Connection: keep-alive
x-envoy-upstream-service-time: 226
box-request-id: 15f1f151feb2282227a86e8d26e12b3bf
cache-control: no-cache, no-store
strict-transport-security: max-age=31536000

{"type":"error","status":403,"code":"access_denied_insufficient_permissions","help_url":"http:\/\/developers.box.com\/docs\/#errors","message":"Access denied - insufficient permission","request_id":"998p1gthwi4nx4k"}

@Joel_Peshkin I think we've reached a point where we need an issue to track this!

Can you open a new issue on Github, please, summarising the above with a link to this forum post.

I think that

  • changing the listing to use next marker would be easy
  • we could ignore that delete error and report them all at the end

Created Enhance rclone cleanup to work around Box "quirks" · Issue #5545 · rclone/rclone · GitHub

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.