Can 'lsf' with '--fast-list' still cause large Data Transfer Out fees with S3?

[Mel_Matsuoka]

I’m a bit of an S3 newbie, so I’m sure this might be user error, but is it possible for simple ‘lsf’ commands on an Amazon S3 bucket to generate enough calls that my AWS bill would significantly spike because of it?

I have about 11 Terabytes of personal photos and videos in an S3 bucket (Glacier Deep Archive storage class), and was using rclone to generate a CSV listing of all the files and folders in that bucket, so that I could import the listing into a media cataloguing system on my Mac.

There’s 230,000 files in the S3 bucket, and I was using the following rclone command to dump the file list:

rclone lsf -R --csv --absolute --format ps --files-only --fast-list --exclude .DS_Store mys3remote:/photolibrary.bucket

To my shock, at the end of the month, my AWS bill was nearly $60 for “data transfer out”, even though I did not actually download any files from the bucket. I checked my bucket security settings and all public access to the bucket was disabled.

I was under the impression that —fast-list minimized the number of transactions when performing directory listings? But even if I didn’t use —fast-list, is it really possible that my rclone command for getting a file listing could really be downloading over half a terabyte of data (I did about 5-10 test runs of the command, as I was integrating it into a python script, so the “data transfer out” size is probably a cumulative count of all my test runs)

Unfortunately I forgot to enable logging on the bucket in question, so I’m not exactly sure why I’m getting billed for 655 GB of Data Transfer Out when all I was doing was listing the bucket contents using —fast-list in rclone?

I contacted AWS support about this, and they just replied with links to their “Understanding data transfer charges” documentation page.

My AWS bill shows that I got charged $59.03 for Data Transfer Out, and an additional $16.82 for "USE2-Requests-Tier1". I know AWS charges for every API call, but this seems a bit excessive to me for just pulling down file listings (and not the files themselves). But I'm willing to be schooled on this if this is totally my fault!

My rclone version:

rclone v1.62.2

• os/version: darwin 14.2 (64 bit)
• os/kernel: 23.2.0 (x86_64)
• os/type: darwin
• os/arch: amd64
• go/version: go1.20.2
• go/linking: dynamic
• go/tags: none

[asdffdsa (jojothehumanmonkey)]

welcome to the forum,

fwiw, i have been using deep glacier for over five years with rclone, never had that happen.
i find glacier is a confusing beast, with complex pricing.

@Mel_Matsuoka

I’m a bit of an S3 newbie

me too, tho i thought with deep glacier, that you cannot simply download data on the fly.
first, you have to start the restore process, then wait for the files to be restored, then download.

@Mel_Matsuoka

230,000 files

i could be wrong but using fast list, one api call gets 1,000 files.
230,000 files /1,000 = 230 api calls.

@Mel_Matsuoka

Unfortunately I forgot to enable logging on the bucket in question

do you have the rclone debug log?

going forward, if you want to see what api calls rclone makes, use --dump flags

with my deep glacier, i only upload, never download.
i just checked an invoice and did not see data transfer cost.