I found a past topic on this that didn't get traction. I'd value this option a lot, especially for local filesystems.
Acknowledging that this could be expensive for a cloud target, maybe just warn people of that in the man page and/or only allow the feature for remotes that won't take a cost hit (ie local).
For cloud targets, maybe it just talks about MiB/s deleted or some other metric that doesn't require a remote walk before or during the operation?
I can put in a separate feature topic for it but in the same vein, is there any way to control the parallelization of rclone purge? How parallel is it by default? It doesn't complain when I put in --transfers=N but I suspect it's just ignoring the flag. Thanks!!
I don't understand going out of your way to point this out when it has nothing whatsoever to do with my question. I use the online docs for rclone which should always be the latest and it doesn't mention the root of my question.
In any case, I upgraded my version to the latest and there is zero change in functionality.
Online docs are for the latest version (v1.65.2 now) - you are using v1.57.0 beta... Nobody is interested really in troubleshooting some many years old rclone release.
Just a thought but for local filesystem why to use something built to work with cloud providers (rclone)? Wouldn't simple rm -rf do the the job better?
rclone is not trying to replace every imaginable local tool/command possible. Its focus is to excel when dealing with cloud. OS provided tools are in most cases much better and faster.
It would be pure waste of time to try to provide some stats for local filesystem operations which have no meaning for online storages... IMO
Hmm that should have helped... I had a look at the code and I discovered that the local backend implements its own Purge rather than using the rclone fallback which has the nice stats and parallel running.
I did a bit of code archaeology and discovered it was implemented in 2014 for v1.02 and doesn't appear to have any special reasons for existing. Unfortunately I didn't write in the commit message anything useful and I certainly don't remember after 10 years!
Deleting the Purge implementation causes stats to appear and --checkers to have an effect!
I think you'll find that the rclone purge in the branch above is faster than rm -rf now (I didn't time it) as it runs --checkers deletes at once, and all I had to do was delete 25 lines of code
I have several thoughts on this...first it's pretty clear that rclone wasn't only built to work with "cloud providers". It's a a powerful, parallelized tool with lots of types of remotes, including support for several filesystems and file-share platforms that technically aren't cloud.
Second, most POSIX commands aren't natively parallelized so when you're interacting with filesystems it's useful to have a tool that you can thread to do something like removal of millions of files. I have other tools to do highly-parallel file/dir removals on Linux but I like rclone so I wanted to see why this feature didn't seem to have much flesh and it seems like Nick digging into it helped a lot. If I hadn't seen rclone purge listed in the documentation I wouldn't have asked them to write it. Someone had clearly already put some time into it though so I wanted to understand why the behavior was different.
