Feature request: Cloud storage quota querying

[i336]

Just started playing around with rclone, it’s nice. I’ll be using it to offload files onto a bunch of Google Drive accounts (the free kind that get created with new email addresses).

One thing that I cannot currently do with rclone, and which seems relevant to a tool like this, is a df-like command that figures out how much storage I have available and how much I’ve used.

I can use --drive-auth-owner-only (when using gdrive specifically) to filter for files only I’ve created [and which count towards my quota], sum the sizes, manually subtract that from my quota limit, and get a result. I could reasonably automate this, too - my quota limit isn’t going to get much bigger anytime soon. But it’s definitely clunky - and actually inaccurate.

Looking around in the Drive UI, I noticed (at the bottom-left in the desktop version) a breakdown showing my usage between GDrive, Gmail and Google Photos. The breakdown is nice, but the fact that it’s counting my usage across all services is very relevant - I could have 2GB of email attachments, 5GB of photos, and only a few hundred MB in Drive, making the tallying idea above completely wrong.

After quite a few minutes’ sweeping up and down the Network tab in the devtools I finally found where Drive requests this breakdown info. The exact HTTP request Drive makes is to a v2internal URL, but I found that the API method it uses is publicly available!

There are two versions.

https://developers.google.com/drive/v2/reference/about/get is the v2 version. …I’m not seeing any deprecation warnings, FWIW.

https://developers.google.com/drive/v3/reference/about/get is the v3 version. Unlike v2 it doesn’t list per-app usage; it just gives you the total capacity, data used by all apps, and total used by Drive. Hmph. (A bare-minimum implementation might subtract usage-by-everything from total-capacity, and drop usage-by-Drive on the floor.) NB. The API explorer is currently incorrect about there being no required parameters, you must expand Request Parameters and put storageQuota in fields.

Both APIs only require the “drive” scope, FYI (you can uncheck all the other scopes in the API explorer and it still works).

Obviously this only handles Drive, and not any other service (including Cloud Storage). I would be very very surprised (okay, but maybe not really all that surprised, cynically speaking…) if all other services rclone supports don’t support quota capacity/usage reporting.

I was going to file this in github but came here on the new-issue template’s advice. Should I move this info over to github?

[ncw (Nick Craig-Wood)]

One thing that I cannot currently do with rclone, and which seems relevant to a tool like this, is a df-like command that figures out how much storage I have available and how much I’ve used.

You can use rclone size but far more fun rclone ncdu.

However neither of these is a quota command. There is one in development though which is exactly what you want.

The development of that feature appears to have stalled though - I wanted the original poster to generalise it so it could be useful for other backends. Maybe you’d like to take over?

[i336]

Ah, size, missed that. I definitely saw ncdu though - I’ve actually used “real” ncdu's export format for quick hacks such as viewing the result of indexing remote HTTP/FTP servers and such. It’s a small utility/idea, but a very cool one. Nice reimplementation!

about looks awesome. I’ve commented on it over in the thread you linked, thanks!

I could tentatively say yes to taking over, but two caveats make me hesitant. a) I work unbelievably slowly at things (ADHD), and I’m trying to prioritize and squash a couple of existing projects before I start anything new; and b) I’m actually in the same position as the other person in that thread, in that Go is (still) on my really-do-need-to-get-around-to-it list, and I’d be playing scramble-catchup-orientate learning the language while I used it.

I’m honestly not sure whether distracted-me would nail Go or not. I know the process of me adding this feature would probably be less noisy if I knew what I was doing.

If the deadline was “next year” or “6 months” I’d say sure, but I know that this is sufficiently trivial it mean less time taken and less fuss involved if I pass. No sweat.

[ncw (Nick Craig-Wood)]

I could tentatively say yes to taking over, but two caveats make me hesitant. a) I work unbelievably slowly at things (ADHD), and I’m trying to prioritize and squash a couple of existing projects before I start anything new; and b) I’m actually in the same position as the other person in that thread, in that Go is (still) on my really-do-need-to-get-around-to-it list, and I’d be playing scramble-catchup-orientate learning the language while I used it.

I’m honestly not sure whether distracted-me would nail Go or not. I know the process of me adding this feature would probably be less noisy if I knew what I was doing.

If the deadline was “next year” or “6 months” I’d say sure, but I know that this is sufficiently trivial it mean less time taken and less fuss involved if I pass. No sweat.

No worries! I think what I’ll do is merge the about as it stands then generalise it in time for v1.41 - that isn’t too much work…