I'm struggling with my OneDrive mounts due to almost immediate throttling from Microsoft when mounting OneDrive.
Looking at debug logs, a call for each folder and subfolder is needed, resulting in a lot of back to back API calls and that is triggering the throttling unless I use a very low TPSLimit.
Checking OneDrive API documentation, seems there is a particular way to pull all the file structure, and then reuse it for polling:
Wonder if this is possible to be implemented when mounting? It might need a flag as it might take a longer time for command to execute and might be best to use alongside a vfs/refresh command.
The onedrive backend does support changenotify to get the deltas by calling the /sites/{siteId}/drive/root/delta endpoint.
We don't currently call it with no parameters to get a listing of the entire drive. This would indeed be what we need to implement ListR. It would only implement ListR for the root directory though and often we mount not the root directory.
If it did implement ListR then we'd need another VFS call to call it and use the results to fill in the VFS cache. We don't have that at the moment but it could be possible to do. Ideally we'd rework vfs/refresh to do that (currently it will list directory by directory).
I might be wrong, but it seems the delta allows for a subfolder to be called directly.
Delta query returns an array of driveItems. If you know ahead of time that you want specific information, you can include it on a select statement with the delta query. You can additionally follow up the delta call with a request for a specific driveItem should you need it. Delta query will still help narrow down the number of items you have to query against for changes, making your application more efficient.
Unfortunately I'm not a Go developer so I would be completely lost, but wonder if this is something worth pursuing or not really.
Yes, I think you are right. You can just call delta on a driveItem
We don't do this for changenotify and we probably should as it would make it more efficient.
ListR for onedrive is worth doing, yes. It speeds up a lot of things by making more efficient API calls.
Can you open a new issue on Github about that and link to this forum post?
The idea of getting vfs/refresh to use ListR is a good one too. That would use more memory as it will have to build the listing in memory first before applying it to the vfs. Probably not too tricky provided using more memory is OK. Worth another issue too I think!
Apologies for not adding any value to the post but I just wanted to ask what your tpslimits are? I have a Sharepoiht Union mount and just like yourself there is a lot of throttling. I am currently using Tpslimit 2 which seems to be working ok.
Not a problem. I'm using my own client id for each mount and --tpslimit 3 --tpslimit-burst 0 flags. That pretty much removed all throttling, but tpslimit 4 does trigger them so it's right on the edge.
Been testing the new build made by Nick and it's been working great. Wonder if anybody else wants to take a look and see if they get a better experience or face any issues. Anybody facing throttling should benefit from this as based on my initial finding seems Microsoft does not throttle the delta API command, giving room for the other commands not to choke and throttle everything.
Keep an eye that my mount command has debug logging and dump-bodies which generated HUGE log files and can't really be left running for a long time, but it's helpful to see what is going on and if any API error shows up.
I have incorporated these commands without the tpslimit flags. So far it is way quicker than before so I am quite happy with that.
Have enabled debug logging as well to see how it performs.
I'll try it with a new sharepoint crypt remote before switching in general.
e.g. onedrive:remote and will play with some mount options especially "tps-limits"...
I'll report some results.
It should work fine on non root drives. Its just that the implementation lists everything under the root and discards the items that aren't in the path you speficied. That is the way the API works.
So if you've got a lot of files not under private it will be a little inefficient, but if you don't it then it will work very efficiently.
Been rocking and using this beta version for a couple of days. Been testing multiple commands and seems it really helps on throttling.
I notices cleanup command still triggers very quickly a throttle event. Not sure if this change might be able to help on that as I'm not familiar on how it works at all.
Otherwise I have yet to face a single issue with the implementation, personally it's now a no brainer to use this flag and method.
Thanks a lot for your help Nick, let me know if you have any concerns that would like to be further tested before merging.
I could change the implementation here to use ListR. Can you give a test and use the --fast-list flag to see if that helps - that will use ListR but it will built it in memory first which isn't needed? If that works I'll switch the implementation of cleanup over to use ListR.