I'm trying to find the best/effective way to PURGE an S3 bucket containing hundreds of thousands of files but any command I try takes ages (hours), like :
rclone purge s3:bucket --no-check-dest --no-traverse --fast-list
Is there any optimized command to achieve this simple task : destroy the S3 bucket and all it's contents?
Or maybe rclone is not the best tool for this?
I'm not using Amazon, but an S3 compatible service which does not implement it.
which providre are you using?
I'm not looking for a provider specific solution, but a general S3 solution trough rclone
I use many providers for different purposes, like MinIO on local, Scaleway, OVH, Wasabi...but not Amazon.
i use aws s3 deep glacier and wasabi,
as far as i know, there is no
--magic flag, as rclone uses the aws s3 sdk.
tho rclone can use custom headers.
maybe using quiet mode, will speed up the process.
Sure, but maybe there are optimized params that I'm not aware of.
Hence, this question.
--fast-list isn't always the fastest option, it depends on the characteristics of the data (and the server).
--no-check-dest and --no-traverse has no effect when purging.
So this command may be quicker due to a higher level of concurrency:
rclone purge s3:bucket --checkers=16
If so, then try doubling the number of checkers until you reach maximum speed.
Note, it will use more API calls and therefore increase your costs.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.