Regarding S3 root size

Help and Support
1

Hi,

What is the problem you are having with rclone?

Probably none.

I’m just trying to assert the root size of a S3-compatible storage, and I’d like to know if I’m understanding things correctly.

Which cloud storage system are you using?

S3-compatible

The command you were trying to run

Here are the commands that ran successfully, in a separate way:

rclone ncdu "My S3_Storage:"

rclone lsf --format "ps" "My S3_Storage:" -R -d

rclone size "My S3_Storage:"

rclone tree "My S3_Storage:" --size

Apparently, the first 3 commands do not indicate the root size; rclone size does show a number of total objects that does not take into account the default folder, but only the files in it.

The command rclone tree " My S3 _ Storage :" --size does explicitly show the root size above the default folder size.
So my question: Is the root (/) size shown via this command reliable, and is this the best way to assert the root (/) size of a S3-compatible storage?

Thanks for your help.

2

Not sure what you mean by root(/) size? and what is default folder?

S3 storage is bucket based. You can not have any files in what you call root(/). And in S3 storage there is no concept of folders/directories.

What you have is a list of buckets containing objects.

3

I’m sorry, I was imprecise in naming it a folder, it is, of course, the bucket that is named default.
Here’s the result of the command rclone tree "My S3_Storage:" --size :

[842889430072]  /
└── [842889430072]  default
    ├── [220117073920]  *File_1*
    ├── [        270]  *File_2*
    ├── [    2097152]  *File_3*
    ├── [        268]  *File_4*
    ├── [257698037760]  *File_5*
    ├── [306016419840]  *File_6*
    ├── [        276]  *File_7*
    ├── [59055800320]  *File_8*
    └── [        266]  *File_9*

1 directories, 9 files

So, as far as I understand, rclone itself uses this concept of folder/directory, but for the bucket. However, ‘Amazon S3 console supports the folder concept as a means of grouping objects’ and ‘you can have folders within folders’.
(see Organizing objects in the Amazon S3 console by using folders - Amazon Simple Storage Service)

But fundamentally, you’re right: there are only buckets and objects.

Regarding my initial question, I’m still unsure because another software confuses buckets for folders, while another one clearly states that my default is a bucket, but what I see above is a root containing one bucket named default. If I had two buckets, they would be both under the / according to the rclone tree.
Is the line ‘[842889430072] /’ valid? Or am I totally misunderstanding?

4

OK - so it is clear for both of us:)

Correct

It shows sum of all buckets sizes. Not sure what you mean by valid? Have you encountered some situation where its value is incorrect?

5

Have you encountered some situation where its value is incorrect?

No, it’s just that I have been asked by the support of my S3 storage to show the root size of my S3 storage, and that rclone is the only software I tried that shows me a size for /, others are just showing the bucket size and don’t go any further up.

So 1) I just want to be sure that the first line really represents what I think is root.

And 2) I’d like to know if I used the correct command to assert it, or if there is another one more suited to get root size.

Also, the support asked me to show them the result of the command rclone ncdu in order to show the root size, but as far I understand, it doesn’t make sense, since ncdu only permits to navigate inside the bucket and so the root size is never shown.

My query is in fact really simple, it’s just that I did not find much info regarding root size for S3 storage. Sorry if I've complicated things.

6

To get sum size of all buckets (which is what I guess is what you mean by root size) I would use rclone size S3:

But you can use rclone tree or rclone ncdu as well.

And given that you want to provide info to some support people I would list all buckets and include every bucket size - it should make all clear.

7

Thanks.
As I said, rclone size is not explicit enough for me:


Total objects: 9 (9)

Total size: 785.002 GiB (842889430072 Byte)

Since there is twice the total of 9 objects, does it mean one is the total of the bucket and the other one the total of all buckets, and thus the total under /?

If you think that rclone tree --size is alright, I’ll validate your response as the solution.

Regarding rclone ncdu, I do not know if the total usage at the bottom of the window is showing the total of / or of the default bucket, since /default is selected and there is nothing else to select.

I still think the rclone tree --size is the most explicit, since I have only one bucket.

Thanks for your assistance.

1 Like
8

I do think that any command would do.

Here example for mulltiple buckets remote:

$ rclone size S3-test:
Total objects: 1.285k (1287)
Total size: 3.306 GiB (3549546562 Byte)

vs. ncdu

rclone ncdu v1.64.2 - use the arrow keys to navigate, press ? for help
-- S3-test: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    3511910176 [##########] /test-info
      37303167 [          ] /test-uk
        333211 [          ] /test
             8 [          ] /test2

Total usage: 3549546562, Objects: 1287

They both provide exactly the same data. rclone tree --size as well but with many objects it is not nice to list.

IMO rclone size is the clearest as it simply tells me total size of all objects (in all buckets) but clear or not can be indeed subjective term:)

In your case I guess the best is one your support will understand and accept.

1 Like
9

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.