Rclone size showing totally incorrect and absurd results in some remotes

What is the problem you are having with rclone?

I have created a remote named Index whose actual size is 7.22 TB
rclone size Index: always giving incorrect results
( Well to be honest not just slightly incorrect but does so by a very huge margin and even most absurdly gives different results each time )
I ran it 5 times
It gave results ranging from 80 gb to 127 gb
Thankfully rclone ncdu Index: works perfectly and gives me 7.22 TB every time i run it
( Posting pics of these both as proof )


Now if it was just a matter of checking size , i do always use ncdu and not bother much ( although ideally even size should give correct info as everyone is not aware of ncdu )
But the bigger indirect issue which comes out is that if we use rclone copy command , i have noticed rclone copies content what it detects according to size command . I won't call cloning is bugged just that it clones what content size command detects .. If rclone size debugging becomes tricky , i do reckon rclone copy use can algorithm to detect files to be cloned using ncdu as its 100% perfect in every scenario whatsoever

Now as i have mentioned in question , this issue doesn't happen in every case. To reproduce make sure that while creating a new remote.

  1. You explicitly enter the root_folder id ( ie don't skip it )
    ( Could be any folder your account has access to )
  2. Don't set this Remote as team drive
  3. and ofc don't use service account

Then check size using rclone size/ncdu remote: respectively
I tried this with some other folders too and got similar results

What is your rclone version (output from rclone version)

rclone v1.51.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

  • os/arch: android/arm64
  • go version: go1.13.7

Which cloud storage system are you using? (eg Google Drive)

Google Drive

Here are the logs for rclone size command

rclone size Index: -vv
2020/05/08 14:11:54 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "size" "Index:" "-vv"]                     2020/05/08 14:11:54 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf"                    2020/05/08 14:11:54 DEBUG : Index: Loaded invalid token from config file - ignoring                                                 2020/05/08 14:11:55 DEBUG : Index: Saved new token in config file
Total objects: 403
Total size: 96.457 GBytes (103570393644 Bytes)
2020/05/08 14:11:58 DEBUG : 21 go routines active
2020/05/08 14:11:58 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "size" "Index:" "-vv"]

================================================
Another test
Remote name - test2
Actual folder size - 167 GB


rclone ncdu test2: gives accurate results as always ( pic uploaded )

rclone size test2: fails for even such a small file shows 143 gb instead of 167 ( log below)

rclone size test2: -vv
2020/05/08 15:13:51 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "size" "test2:" "-vv"]
2020/05/08 15:13:51 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf"
Total objects: 254
Total size: 143.149 GBytes (153704723191 Bytes)
2020/05/08 15:13:56 DEBUG : 19 go routines active
2020/05/08 15:13:56 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "size" "test2:" "-vv"]

Is it a team drive? What does your rclone.conf look like for that remote without keys/passwords in it?

I run the same thing on a larger Google Drive and get the exact same results each time as I'm running on Linux.

Can you run the size command a few times with -vv as well and share the log?

Yes that remote is folder from a team drive
Here's what the config looks like

[Index]
type = drive
scope = drive
root_folder_id = 17_A4A0h09mKhsGQo-S-0ottF########
token = {"access_token":"ya29.a0Ae4lvC2HpK2t5Q.......}

Surely i can give the log , here it is

rclone size Index: -vv
2020/05/08 14:11:54 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "size" "Index:" "-vv"]                     2020/05/08 14:11:54 DEBUG : Using config file from "/data/data/com.termux/files/home/.config/rclone/rclone.conf"                    2020/05/08 14:11:54 DEBUG : Index: Loaded invalid token from config file - ignoring                                                 2020/05/08 14:11:55 DEBUG : Index: Saved new token in config file
Total objects: 403
Total size: 96.457 GBytes (103570393644 Bytes)
2020/05/08 14:11:58 DEBUG : 21 go routines active
2020/05/08 14:11:58 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone" "size" "Index:" "-vv"]

I also edited a second test , might wanna have a look

When you run the test a few times, does the object count change?

Team drives are a bit annoying at times as they can take some time to index and catch things up if you are moving to it.

and I assume you are sure no one else is changing / moving things while it's going on? Feels thats a silly question but wanted to ask.

I can't get it to break running a number of times on a regular Google Drive.

Yes in case of rclone size remote: , the object count changes like crazy (for reference check in question , that pic where 5 rclone size commands are present and object count kept changing )

ncdu has never given any wrong result yet , bet it td folder or shared . 100% accurate in every test case
size seems bugged if you enter a folder id and not set it up as team drive ( for me atleast , i mean from all of testings i can conclude this only. Because size command too gives accurate result if i set up remote as team drive )

Yes , 100% no one is touching any file , noone except me has access to them

Yeah, that's strange, I'll see if I can get some time and test on a team drive and see as you don't seem to have too many objects either.

Sure just make sure you enter the folder root id manually and don't set it up as team drive. And preferably test it on folder have decent content in it ( i mean not like just 1 big file )

Ok, you confused me.

Is it a team drive or no?

Lemme explain once more on how to setup the remote to reproduce the issue
Create a new remote , and

  1. when it asks to enter root folder id , just paste folder id of any folder you have access too.
    a-Could be a folder from inside your team drive ,
    b-could be a shared folder
    c-could be any public folder on internet ..
    Important is don't skip this step
  2. When it asks to set up remote as team drive - Select No
  3. Now test that remote size using rclone size remote: && rclone ncdu remote:

@ncw @calisro can you please have a look

What's the reason you select not a team drive when using a team drive though?

They both read the forums and I'm just trying to make sure I got the flow you are using and why you are using that.

1 Like

I'm not sure. I'll try to take a look when I have a moment though

1 Like

Yes i understand that logically if its a folder from inside td , one wouldn't set Remote up as non td after entering folder id
Mainly this could be shared folder or public folders on internet to recreate the issue
i just included td folder to setup as non td along with other two to cover every possibility on paper

When you experience the issue though, is it on a regular Google Drive or Shared (Team) Drive?

I've tried to recreate on a Drive and I cannot, which is why I''m trying to understand if it's always on a Shared (Team) drive as they do some funky things with indexing and such compared to a regular Drive.

Thanks for trying but i haven't tested it on a folder from my drive yet and i am not certain about my drive's folders

Although i can surely recreate it on a shared folder, public folder and team drive folder set as non td .

Lemme make this easy and share with you a public folder to try
1uXCWl9rGsndFg_jpQ6VHq59OcQYb-Et1
Create a remote and enter this when asked folder id and choose NO for setup as team drive

Then check size using rclone size and rclone ncdu
Its actual size is 6.54 TB which ncudu perfectly showed as always
Although rclone size is absurdly telling me its just 100 gb -150 gb ( yes as i had told originally , its not only massively incorrect but also showing variations each time in rclone size output )

I don't think it has anything to do with the size command.

I can run a ls and get different results so the drive is reporting back different things.

felix@gemini:/opt/rclone$ rclone size Test: --fast-list
Total objects: 446
Total size: 24.722 GBytes (26545144293 Bytes)
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
56
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
6
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
111
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
6
felix@gemini:/opt/rclone$ rclone size Test: --fast-list
Total objects: 481
Total size: 168.237 GBytes (180642797845 Bytes)
felix@gemini:/opt/rclone$ rclone size Test: --fast-list
Total objects: 14
Total size: 3.360 GBytes (3607796268 Bytes)
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
26
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
2
felix@gemini:/opt/rclone$ rclone ls Test: | wc -l
339

About does a different API call which I think is why it returns properly.

felix@gemini:/opt/rclone$ rclone about Test:
Used:    115.366T
Trashed: 0
Other:   185.297M
felix@gemini:/opt/rclone$ rclone about Test:
Used:    115.366T
Trashed: 0
Other:   185.297M
felix@gemini:/opt/rclone$ rclone about Test:
Used:    115.366T
Trashed: 0
Other:   185.297M

I'll take look at some of the replies a bit more.

Ah nice
Finally it gave different results for you as well while using rclone size and ls

And regarding using rclone about remote:
Please note that this command will not output the size of the public folder id which you have entered but rather about command always outputs the storage status of the google account you have authenticated the config...

Used: 115.366T
Trashed: 0
Other: 185.297M

This size is not of the public folder whose remote you have created but rather the storage status of your google account which you used to log in at end .. Similarly when i use rclone about, i get storage status of my google drive (15 gb basic )which i have used to authenticate rclone and not the public folder .So obviously there's no reason for rclone about remote: to be inaccurate as its not even touching the troubled area

rlcone size , ls these commands seem bugged for public and shared folders, while ncdu has never failed
Also i hope you tried rclone ncdu Test:
It hopefully might have shown you correct output

On another note , i also made some of my friends go through the test and they too reported similarly

I'm pretty sure if you disable --fast-list (which is used automatically by rclone size) with --disable ListR you'll get 100% consistent results.

I also think this is fixed in the latest beta - can you give that a go?

Sure , i will test this in the latest beta and let you know
Also just curious , this --fast-list seem to have caused this issue it seems , i ran the rclone size with this --disable ListR along with rclone size and it gave accurate results

So is it possible it can skip files if someone uses this flag --fast-list while cloning ( rclone copy ...) ?
i had read it can improve speeds while rclone copy but now i am not so sure if its worth using it

I wonder if the shared items act more like team/drives.

There is this item which is related to fast list and items: