Google Drive: disk usage information is inconsistent

Suspected Bug
1

What is the problem you are having with rclone?

Disk usage information is inconsistent.

What is your rclone version (output from rclone version)

rclone v1.52.1-061-gfb06427c-beta
- os/arch: linux/amd64
- go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Devuan 3.0, 64 bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

# cd /mnt/remote
# find
.
# rclone size drive:
Total objects: 0
Total size: 0 Bytes (0 Bytes)
# df .
Filesystem         1K-blocks        Used     Available Use% Mounted on
drive:         1115718439112 16206811336 1099511627776   2% /mnt/remote
# cd

I'd expect Used in the df output to match Total size in the rclone size output, after some accounting for unit conversions and rounding.

What is going on here is that Used in the df output includes space used by the drive-with-appfolder remote which stores its data in the Application Data folder. However, since the drive remote in use here does not reference the Application Data folder, I'd say that the rclone size output is more correct than the df one. In any case, I think the numbers should be much closer to each other.

The rclone config contents with secrets removed.

[drive-with-appfolder]
type = drive
scope = drive,drive.appfolder
service_account_file = <file path>
impersonate = <email address>
root_folder_id = appDataFolder
server_side_across_configs = true

[drive]
type = drive
scope = drive,drive.appfolder
service_account_file = <file path>
impersonate = <email address>
root_folder_id = <string>
server_side_across_configs = true

A log from the command with the -vv flag

# rclone mount drive: /mnt/remote --poll-interval 15s -vv
2020/06/13 19:07:35 DEBUG : rclone: Version "v1.52.1-061-gfb06427c-beta" starting with parameters ["rclone" "mount" "drive:" "/mnt/remote" "--poll-interval" "15s" "-vv"]
2020/06/13 19:07:35 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2020/06/13 19:07:35 DEBUG : Google drive root '': Mounting on "/mnt/remote"
2020/06/13 19:07:35 DEBUG : Adding path "vfs/forget" to remote control registry
2020/06/13 19:07:35 DEBUG : Adding path "vfs/refresh" to remote control registry
2020/06/13 19:07:35 DEBUG : Adding path "vfs/poll-interval" to remote control registry
2020/06/13 19:07:35 DEBUG : : Root: 
2020/06/13 19:07:35 DEBUG : : >Root: node=/, err=<nil>
2020/06/13 19:07:40 DEBUG : /: Attr: 
2020/06/13 19:07:40 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/06/13 19:07:44 DEBUG : /: Attr: 
2020/06/13 19:07:44 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/06/13 19:07:44 DEBUG : /: ReadDirAll: 
2020/06/13 19:07:44 DEBUG : /: >ReadDirAll: item=0, err=<nil>
2020/06/13 19:07:50 DEBUG : Google drive root '': Checking for changes on remote
2020/06/13 19:08:05 DEBUG : Google drive root '': Checking for changes on remote
2020/06/13 19:08:20 DEBUG : Google drive root '': Checking for changes on remote
2020/06/13 19:08:27 DEBUG : /: Attr: 
2020/06/13 19:08:27 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxr-xr-x, err=<nil>
2020/06/13 19:08:27 DEBUG : : Statfs: 
2020/06/13 19:08:27 DEBUG : : >Statfs: stat={Blocks:278929609778 Bfree:274877906944 Bavail:274877906944 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
2020/06/13 19:08:35 DEBUG : Google drive root '': Checking for changes on remote
^C2020/06/13 19:08:40 DEBUG : rclone: Version "v1.52.1-061-gfb06427c-beta" finishing with parameters ["rclone" "mount" "drive:" "/mnt/remote" "--poll-interval" "15s" "-vv"]
2

Interesting! What does "rclone about" say on your drive?

That is the info that is used to show the df results.

I think the appdata stuff is part of your 15gb quota though isn't it?

3

It's 15TB, and I don't know of any "15gb" quota. I'm on G Suite for Business.

# rclone about drive:
Used:    15.094T
Trashed: 3.838M
Other:   2.476G
4

What rclone is reporting here is the quotas on your account. You have unlimited quota so you don't get a total field. However the 15T is counting against your unlimited quota.

There isn't a quick way of finding the number from rclone size other than looking at every file which can take a long time.

Rclone uses the quota amounts to give the df results. Just like if you did df of an empty directory you'd see the results for the whole disk, rclone about shows you the data for the whole account even though those files aren't visible.

5

Well, I just ran rclone size drive-with-appfolder:, and it took almost a minute to complete, so I guess that explains why you're hesitant to use it. I still think the df output is incorrect, though. df is supposed to show information on filesystems. /mnt/remote in my case is the mountpoint of one filesystem. df /mnt/remote should show information on that filesystem and it doesn't!

6

I can't quite figure this out as all my stuff shows identical:

GD:             1.2P  118T  1.0P  11% /home/felix/test
GDAPP:          1.2P  118T  1.0P  11% /home/felix/test2
felix@gemini:~$ rclone about GD:
Used:    117.648T
Trashed: 0
Other:   185.297M
felix@gemini:~$ rclone about GDAPP:
Used:    117.648T
Trashed: 0
Other:   185.297M
felix@gemini:~$ rclone size GD:
Total objects: 45791
Total size: 117.648 TBytes (129354910288946 Bytes)
felix@gemini:~$ rclone size GDAPP:
Total objects: 45791
Total size: 117.648 TBytes (129354910288946 Bytes)
7

That is what I'd expect. It shows the usage for the account. The appfolder is like a separate drive in the same account.

1 Like
8

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.