I would like to be able to use a flag, something like --cut-dirs 1
or --add-dirs 1
. It would work like this:
$ rclone copy --add-dirs 1 --include '2021/07/**' 'REMOTE:.CDN_ACCESS_LOGS/Container {A,C}/' ./cdn_logs/
Instead of the files being merged:
./cdn_logs/2021/07/01/0000.log
./cdn_logs/2021/07/01/0000.log
./cdn_logs/2021/07/01/0001.log
./cdn_logs/2021/07/01/0001.log
It would add 1 (or however many) of the parent directories to the target, like so:
./cdn_logs/Container A/2021/07/01/0000.log
./cdn_logs/Container C/2021/07/01/0000.log
./cdn_logs/Container A/2021/07/01/0001.log
./cdn_logs/Container C/2021/07/01/0001.log
--cut-dirs
would be the reverse, where it removes the first directory from the target name. This option exists in wget
: GNU Wget 1.21.1-dirty Manual
I am also aware that you can move the directory filter to the --include
, but in the case of --fast-list
the entire parent folder gets traversed, e.g. every single subfolder in Container B
, Container D
, etc. Like this:
$ rclone copy --fast-list --include 'Container {A,C}/2021/07/' 'REMOTE:.CDN_ACCESS_LOGS/' ./cdn_logs/
Using --fast-list
is crucial in this case, since the directories actually go down to the minute, so without it the traversal is very slow while traversing many directories that are not needed. Here's an example of a real filename: Container A/2021/07/27/00/2ca13205ac8790d9f87aecac8ba50684.log.gz
.
These options would also help reduce verbosity and duplication in some cases, e.g.:
$ rclone copy 'REMOTE:.CDN_ACCESS_LOGS/Container A/2021/07/27/00/*.log.gz' './cdn_logs/Container A/2021/07/27/00/'
could be replaced by
$ rclone copy --add-dirs 5 'REMOTE:.CDN_ACCESS_LOGS/Container A/2021/07/27/00/*.log.gz' './cdn_logs/'
Thank you for your time.