Issue with "rclone about" and data sizes returned creating account issues

What is the problem you are having with rclone?

Suspected issue with rclone about retrieving correct quota information OR a problem with the provider's information provision.

Yesterday (before getting more storage assigned by Dropbox) I was getting upload errors due to no space. rclone about claimed in the region of 13Tb free. Dropbox's admin panel claimed zero bytes. When Dropbox added more space the values increased in both rclone about AND Dropbox's own web interface, but the discrepency still is present, in other words rclone reports more space than is available and I believe Dropbox should know what space they have, especially if they block access to writes when they believe it is full.

The Dropbox account being used does not have a user quota in itself (e.g. no user x has xTb, user y has xTb and the like) so if the space exists it can be used. This rules out an internal quota issue.

Even when I look at space used by other users, it does not add up to the discrepancy, by several Tb).

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

  • os/version: debian 11.5 (64 bit)
  • os/kernel: 5.10.0-17-amd64 (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.18.5
  • go/linking: static
  • go/tags: none

[also problem noted with MacOS use with same version]

Which cloud storage system are you using? (eg Google Drive)

Dropbox (Business/Corporate account)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone about [REMOTENAME]:

rclone about [REMOTENAME]:

The rclone config contents with secrets removed.

[REMOTENAME]
type = crypt
remote = XXX:rc/enc
password = XXXXX

A log from the command with the -vv flag

2022/09/16 05:48:09 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "about" "XXX-ENC:" "-vv"]
2022/09/16 05:48:09 DEBUG : Creating backend with remote "XXX-ENC:"
2022/09/16 05:48:09 DEBUG : Using config file from "/home/xxxxx/.rclone.conf"
2022/09/16 05:48:09 DEBUG : Creating backend with remote "XXX:rc/enc"
Total:   483.176 TiB
Used:    418.640 TiB
Free:    64.536 TiB
2022/09/16 05:48:10 DEBUG : 6 go routines active
2022/09/16 05:48:10 INFO  : Dropbox root 'rc/enc': Commiting uploads - please wait...

This was made last night to further give (hopefully) evidence:

2022/09/15 19:46:35 NOTICE: --dump-bodies is obsolete - please use --dump bodies instead
2022/09/15 19:46:35 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "about" "XXX-ENC:" "-vv" "--dump-bodies" "--log-file=Dropbox.txt"]
2022/09/15 19:46:35 DEBUG : Creating backend with remote "XXX-ENC:"
2022/09/15 19:46:35 DEBUG : Using config file from "/home/bastuman/.rclone.conf"
2022/09/15 19:46:35 DEBUG : Creating backend with remote "XXX:rc/enc"
2022/09/15 19:46:35 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/09/15 19:46:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:35 DEBUG : HTTP REQUEST (req 0xc000150200)
2022/09/15 19:46:35 DEBUG : POST /2/files/get_metadata HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"path":"/rc/enc","include_media_info":false,"include_deleted":false,"include_has_explicit_shared_members":false}
2022/09/15 19:46:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : HTTP RESPONSE (req 0xc000150200)
2022/09/15 19:46:36 DEBUG : HTTP/2.0 200 OK
Content-Length: 120
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 15 Sep 2022 19:46:35 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: f3808403667e415f8ad3c2072a8a379d
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 96

{".tag": "folder", "name": "enc", "path_lower": "/rc/enc", "path_display": "/rc/enc", "id": "id:XXXXXXXXXXXXXX"}
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : HTTP REQUEST (req 0xc000150600)
2022/09/15 19:46:36 DEBUG : POST /2/users/get_space_usage HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip

2022/09/15 19:46:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : HTTP RESPONSE (req 0xc000150600)
2022/09/15 19:46:36 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 15 Sep 2022 19:46:35 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 388b15b17c0a4b1b899407148c5709d5
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 93

{"used": 459998867321730, "allocation": {".tag": "team", "used": 476683729280959, "allocated": 531257389744128, "user_within_team_space_allocated": 0, "user_within_team_space_limit_type": {".tag": "off"}, "user_within_team_space_used_cached": 459755495502467}}
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : 6 go routines active
2022/09/15 19:46:36 INFO : Dropbox root 'rc/enc': Commiting uploads - please wait...

Thats exactly what I would have recommended doing.

Expanding that out we get

{
  "used": 459998867321730,  // rclone is reporting this value as used
  "allocation": {
    ".tag": "team",
    "used": 476683729280959,
    "allocated": 531257389744128, // rclone is reporting this value as allocated
    "user_within_team_space_allocated": 0,
    "user_within_team_space_limit_type": {
      ".tag": "off"
    },
    "user_within_team_space_used_cached": 459755495502467 // not sure what this is
  }
}

Those should be the numbers rclone about reports.

How do they differ from what you are expecting?

Thanks. Now via the web (admin interface, total view over entire plan) 54.6Tb free. Via rclone about 64.223Tb. Yesterday when we ran out of space according to Dropbox, rclone thought we had c 13Tb or so free.

There's presently a spare licence if you wish to access & see if there's something to troubleshoot.

From the numbers above, rclone will be reporting

what bytes TiB
total 531257389744128 483.1
used 459998867321730 418.3
free 71258522422398 64.8

So that is where the numbers came from. Whether they are correct or not, if you take the other used number 476683729280959 then you get 531257389744128-476683729280959 = 54573660463169 which is 49.6 TiB - maybe this is closer to the correct number?

The python SDK has a bit more on these numbers: dropbox.users – Users — Dropbox for Python 0.0.0 documentation

The docs aren't very helpful as to what the team/user parts of the get space response mean and I'm not sure exactly which bit rclone should be returning as part of rclone about

The current logic is here

To return a free space, we need to subtract a used from an allocated...

We seem to have 3 possible used values though and I'm not sure which one rclone should be using!

  • "used": 459998867321730 = 418.3 TiB
  • "team.allocation.used" = 476683729280959 = 433.5 TiB
  • "team.allocation.user_within_team_space_used_cached" = 459755495502467 = 418.1 TB