Issue with "rclone about" and data sizes returned creating account issues

DarrenPIngram · September 16, 2022, 5:52am

What is the problem you are having with rclone?

Suspected issue with rclone about retrieving correct quota information OR a problem with the provider's information provision.

Yesterday (before getting more storage assigned by Dropbox) I was getting upload errors due to no space. rclone about claimed in the region of 13Tb free. Dropbox's admin panel claimed zero bytes. When Dropbox added more space the values increased in both rclone about AND Dropbox's own web interface, but the discrepency still is present, in other words rclone reports more space than is available and I believe Dropbox should know what space they have, especially if they block access to writes when they believe it is full.

The Dropbox account being used does not have a user quota in itself (e.g. no user x has xTb, user y has xTb and the like) so if the space exists it can be used. This rules out an internal quota issue.

Even when I look at space used by other users, it does not add up to the discrepancy, by several Tb).

Run the command 'rclone version' and share the full output of the command.

rclone v1.59.1

os/version: debian 11.5 (64 bit)
os/kernel: 5.10.0-17-amd64 (x86_64)
os/type: linux
os/arch: amd64
go/version: go1.18.5
go/linking: static
go/tags: none

[also problem noted with MacOS use with same version]

Which cloud storage system are you using? (eg Google Drive)

Dropbox (Business/Corporate account)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone about [REMOTENAME]:

rclone about [REMOTENAME]:

The rclone config contents with secrets removed.

[REMOTENAME]
type = crypt
remote = XXX:rc/enc
password = XXXXX

A log from the command with the `-vv` flag

2022/09/16 05:48:09 DEBUG : rclone: Version "v1.59.2" starting with parameters ["rclone" "about" "XXX-ENC:" "-vv"]
2022/09/16 05:48:09 DEBUG : Creating backend with remote "XXX-ENC:"
2022/09/16 05:48:09 DEBUG : Using config file from "/home/xxxxx/.rclone.conf"
2022/09/16 05:48:09 DEBUG : Creating backend with remote "XXX:rc/enc"
Total:   483.176 TiB
Used:    418.640 TiB
Free:    64.536 TiB
2022/09/16 05:48:10 DEBUG : 6 go routines active
2022/09/16 05:48:10 INFO  : Dropbox root 'rc/enc': Commiting uploads - please wait...

DarrenPIngram · September 16, 2022, 5:58am

This was made last night to further give (hopefully) evidence:

2022/09/15 19:46:35 NOTICE: --dump-bodies is obsolete - please use --dump bodies instead
2022/09/15 19:46:35 DEBUG : rclone: Version "v1.59.1" starting with parameters ["rclone" "about" "XXX-ENC:" "-vv" "--dump-bodies" "--log-file=Dropbox.txt"]
2022/09/15 19:46:35 DEBUG : Creating backend with remote "XXX-ENC:"
2022/09/15 19:46:35 DEBUG : Using config file from "/home/bastuman/.rclone.conf"
2022/09/15 19:46:35 DEBUG : Creating backend with remote "XXX:rc/enc"
2022/09/15 19:46:35 DEBUG : You have specified to dump information. Please be noted that the Accept-Encoding as shown may not be correct in the request and the response may not show Content-Encoding if the go standard libraries auto gzip encoding was in effect. In this case the body of the request will be gunzipped before showing it.
2022/09/15 19:46:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:35 DEBUG : HTTP REQUEST (req 0xc000150200)
2022/09/15 19:46:35 DEBUG : POST /2/files/get_metadata HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 113
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"path":"/rc/enc","include_media_info":false,"include_deleted":false,"include_has_explicit_shared_members":false}
2022/09/15 19:46:35 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : HTTP RESPONSE (req 0xc000150200)
2022/09/15 19:46:36 DEBUG : HTTP/2.0 200 OK
Content-Length: 120
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 15 Sep 2022 19:46:35 GMT
Server: envoy
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: f3808403667e415f8ad3c2072a8a379d
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 96

{".tag": "folder", "name": "enc", "path_lower": "/rc/enc", "path_display": "/rc/enc", "id": "id:XXXXXXXXXXXXXX"}
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : HTTP REQUEST (req 0xc000150600)
2022/09/15 19:46:36 DEBUG : POST /2/users/get_space_usage HTTP/1.1
Host: api.dropboxapi.com
User-Agent: rclone/v1.59.1
Content-Length: 0
Authorization: XXXX
Accept-Encoding: gzip

2022/09/15 19:46:36 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : HTTP RESPONSE (req 0xc000150600)
2022/09/15 19:46:36 DEBUG : HTTP/2.0 200 OK
Accept-Encoding: identity,gzip
Cache-Control: no-cache
Content-Type: application/json
Date: Thu, 15 Sep 2022 19:46:35 GMT
Server: envoy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Dropbox-Request-Id: 388b15b17c0a4b1b899407148c5709d5
X-Dropbox-Response-Origin: far_remote
X-Frame-Options: SAMEORIGIN
X-Server-Response-Time: 93

{"used": 459998867321730, "allocation": {".tag": "team", "used": 476683729280959, "allocated": 531257389744128, "user_within_team_space_allocated": 0, "user_within_team_space_limit_type": {".tag": "off"}, "user_within_team_space_used_cached": 459755495502467}}
2022/09/15 19:46:36 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2022/09/15 19:46:36 DEBUG : 6 go routines active
2022/09/15 19:46:36 INFO : Dropbox root 'rc/enc': Commiting uploads - please wait...

ncw · September 16, 2022, 3:43pm

Thats exactly what I would have recommended doing.

Expanding that out we get

{
  "used": 459998867321730,  // rclone is reporting this value as used
  "allocation": {
    ".tag": "team",
    "used": 476683729280959,
    "allocated": 531257389744128, // rclone is reporting this value as allocated
    "user_within_team_space_allocated": 0,
    "user_within_team_space_limit_type": {
      ".tag": "off"
    },
    "user_within_team_space_used_cached": 459755495502467 // not sure what this is
  }
}

Those should be the numbers rclone about reports.

How do they differ from what you are expecting?

DarrenPIngram · September 16, 2022, 5:27pm

Thanks. Now via the web (admin interface, total view over entire plan) 54.6Tb free. Via rclone about 64.223Tb. Yesterday when we ran out of space according to Dropbox, rclone thought we had c 13Tb or so free.

DarrenPIngram · September 16, 2022, 5:29pm

There's presently a spare licence if you wish to access & see if there's something to troubleshoot.

ncw · September 17, 2022, 2:03pm

From the numbers above, rclone will be reporting

what	bytes	TiB
total	531257389744128	483.1
used	459998867321730	418.3
free	71258522422398	64.8

So that is where the numbers came from. Whether they are correct or not, if you take the other used number 476683729280959 then you get 531257389744128-476683729280959 = 54573660463169 which is 49.6 TiB - maybe this is closer to the correct number?

The python SDK has a bit more on these numbers: dropbox.users – Users — Dropbox for Python 0.0.0 documentation

users.TeamSpaceAllocation.used – The total space currently used by the user’s team (bytes).
users.TeamSpaceAllocation.allocated – The total space allocated to the user’s team (bytes).
users.TeamSpaceAllocation.user_within_team_space_allocated – The total space allocated to the user within its team allocated space (0 means that no restriction is imposed on the user’s quota within its team).
users.TeamSpaceAllocation.user_within_team_space_limit_type – The type of the space limit imposed on the team member (off, alert_only, stop_sync).
users.TeamSpaceAllocation.user_within_team_space_used_cached – An accurate cached calculation of a team member’s total space usage (bytes).

The docs aren't very helpful as to what the team/user parts of the get space response mean and I'm not sure exactly which bit rclone should be returning as part of rclone about

The current logic is here

github.com

rclone/rclone/blob/1c99661d8cfbafa86ea186a3d0c8f07993a122a9/backend/dropbox/dropbox.go#L1276-L1289


      
          	var total uint64
          	if q.Allocation != nil {
          		if q.Allocation.Individual != nil {
          			total += q.Allocation.Individual.Allocated
          		}
          		if q.Allocation.Team != nil {
          			total += q.Allocation.Team.Allocated
          		}
          	}
          	usage = &fs.Usage{
          		Total: fs.NewUsageValue(int64(total)),          // quota of bytes that can be used
          		Used:  fs.NewUsageValue(int64(q.Used)),         // bytes in use
          		Free:  fs.NewUsageValue(int64(total - q.Used)), // bytes which can be uploaded before reaching the quota
          	}

To return a free space, we need to subtract a used from an allocated...

We seem to have 3 possible used values though and I'm not sure which one rclone should be using!

"used": 459998867321730 = 418.3 TiB
"team.allocation.used" = 476683729280959 = 433.5 TiB
"team.allocation.user_within_team_space_used_cached" = 459755495502467 = 418.1 TB

system · October 17, 2022, 2:04pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.