Please help to test out new API for OneDrive (less throttling, better Sharepoint support, etc.)

Feature
1

Hi all,

I’m trying to get rclone migrated from the old, deprecated OneDrive API to the new Graph API based on @olihey 's branch olihey/onedrive_graph. My pull request can be found here. Now the work is almost done and I would like you to help test it out. The expected advantages of this migration should be:

  • Less API throttling
  • Possibility to apply for your own Client ID / Key easily through (Docs)
  • More Unified OneDrive Personal / Business / Sharepoint Support
  • And More…

Update: The code has been merged into master, so all v1.43-beta versions that can be found on https://beta.rclone.org/ will contain this change. Thank you for your support!

2 Likes
2

Awesome that you finished that and sorry I couldn’t be of more help. Great job! :+1:

2 Likes
3

Thanks for all of your hard work on this!

Just tried this out and ran into a few issues:

I tried to setup a new Sharepoint site via rclone config. I selected OneDrive as the storage type, then after the auto config succeeded I chose “2 / Sharepoint site”. I then got this error:

2018/08/27 13:41:20 Failed to query available drives: HTTP error 429 (429 ) returned body: "{\r\n  \"error\": {\r\n    \"code\": \"activityLimitReached\",\r\n    \"message\": \"The request has been throttled\",\r\n    \"innerError\": {\r\n      \"request-id\": \"9962c0d6-aff7-4c8d-a65b-4848922f695b\",\r\n      \"date\": \"2018-08-27T03:40:48\"\r\n    }\r\n  }\r\n}"

This appears to relate to the OneDrive API’s request throttling, so I’m not sure how much can be done about it from rclone’s side, but some useful docs can be found here:

https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

I managed to get around it by selecting ‘5 / Search a Sharepoint site’ and selecting the site that way.

Second issue occurred when I tried to sync some files from Dropbox to the Sharepoint site. I started getting errors like this:

2018/08/27 13:53:05 ERROR : Document.docx: corrupted on transfer: sizes differ 17225 vs 24280
2018/08/27 13:53:05 INFO  : Document.docx: Removing failed copy

This only seems to happen for certain types of files (Office files like docx, msg, txt). I’m pretty sure this is the same issue as the one discussed by just-ak here:

In that Sharepoint modifies files after upload, rclone interprets the size difference as a corrupt transfer and discards it.

4

Hi, thanks for your help.

I would reply to the throttling first: I’ve been uploading stuff to my OneDrive for a few hours, so I don’t know if I’m using up the quota of rclone’s default Client ID / Key :confused: Microsoft is not quite generous on this. But I’ll suggest that you have a look at the Getting your own Client ID and Key section in the new docs https://github.com/Cnly/rclone/blob/993130495a8d2a8834f6b97aedd69718e4d23ffa/docs/content/onedrive.md#getting-your-own-client-id-and-key and have a try on that.

5

Thanks @Cnly, I followed the steps in that article and the error doesn’t appear now. Although the returned list of sites for option 2 isn’t what I expected - I think it only sees document libraries within the top level site collection (orgname.sharepoint.com). It doesn’t seem to pick up on any subsites - I can only see document libraries for the top level. So to find the document library in the team site I wanted to use, I had to use the search function (option 5), which works fine.

6

Good to hear the throttling is going away.

You’re right that option 2 only lists the drives of the root site - it’s actually a drive selector (for the root site), not a site selector. Maybe we can change this option to a site & drive selector.

7

Yes, I think either reword option 2 to make it clear that it is just listing the root site, or add the drive selector.

Sharepoint modifying files after upload appears to be the main blocker for use at the moment, I haven’t found a way around it yet.

8

Got everything configured but I just keep getting throttled really fast. After about 500 file checks. Anything I can do?

9

Have you tried getting your own Client ID / Key? If you had, then I’m afraid we can’t do more about it. Microsoft isn’t quite generous on this.