Best command to purge S3 bucket

Help and Support
1

I'm trying to find the best/effective way to PURGE an S3 bucket containing hundreds of thousands of files but any command I try takes ages (hours), like : rclone purge s3:bucket --no-check-dest --no-traverse --fast-list

Is there any optimized command to achieve this simple task : destroy the S3 bucket and all it's contents? :thinking:

Or maybe rclone is not the best tool for this?

Thanks. :slight_smile:

2

hello,

how about empty an Amazon S3 bucket using a lifecycle configuration rule

3

I'm not using Amazon, but an S3 compatible service which does not implement it.

4

which providre are you using?

5

I'm not looking for a provider specific solution, but a general S3 solution trough rclone :slight_smile:
I use many providers for different purposes, like MinIO on local, Scaleway, OVH, Wasabi...but not Amazon.

6

understood,
i use aws s3 deep glacier and wasabi,

as far as i know, there is no --magic flag, as rclone uses the aws s3 sdk.

tho rclone can use custom headers.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
maybe using quiet mode, will speed up the process.

7

Sure, but maybe there are optimized params that I'm not aware of.
Hence, this question.

8

--fast-list isn't always the fastest option, it depends on the characteristics of the data (and the server).
--no-check-dest and --no-traverse has no effect when purging.

So this command may be quicker due to a higher level of concurrency:

rclone purge s3:bucket --checkers=16

If so, then try doubling the number of checkers until you reach maximum speed.

Note, it will use more API calls and therefore increase your costs.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.