Include or exclude one or more parent dirs when writing to target

I would like to be able to use a flag, something like --cut-dirs 1 or --add-dirs 1. It would work like this:

$ rclone copy --add-dirs 1 --include '2021/07/**' 'REMOTE:.CDN_ACCESS_LOGS/Container {A,C}/' ./cdn_logs/

Instead of the files being merged:

./cdn_logs/2021/07/01/0000.log
./cdn_logs/2021/07/01/0000.log
./cdn_logs/2021/07/01/0001.log
./cdn_logs/2021/07/01/0001.log

It would add 1 (or however many) of the parent directories to the target, like so:

./cdn_logs/Container A/2021/07/01/0000.log
./cdn_logs/Container C/2021/07/01/0000.log
./cdn_logs/Container A/2021/07/01/0001.log
./cdn_logs/Container C/2021/07/01/0001.log

--cut-dirs would be the reverse, where it removes the first directory from the target name. This option exists in wget: GNU Wget 1.21.1-dirty Manual

I am also aware that you can move the directory filter to the --include, but in the case of --fast-list the entire parent folder gets traversed, e.g. every single subfolder in Container B, Container D, etc. Like this:

$ rclone copy --fast-list --include 'Container {A,C}/2021/07/' 'REMOTE:.CDN_ACCESS_LOGS/' ./cdn_logs/

Using --fast-list is crucial in this case, since the directories actually go down to the minute, so without it the traversal is very slow while traversing many directories that are not needed. Here's an example of a real filename: Container A/2021/07/27/00/2ca13205ac8790d9f87aecac8ba50684.log.gz.

These options would also help reduce verbosity and duplication in some cases, e.g.:

$ rclone copy 'REMOTE:.CDN_ACCESS_LOGS/Container A/2021/07/27/00/*.log.gz' './cdn_logs/Container A/2021/07/27/00/'

could be replaced by

$ rclone copy --add-dirs 5 'REMOTE:.CDN_ACCESS_LOGS/Container A/2021/07/27/00/*.log.gz' './cdn_logs/'

Thank you for your time.

Bump. No interest in this?

hi,

seems like a narrow use-case and could be solved by a bash/python script.

You can always add a feature request on github and if enough action on it, it'll get picked up..

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.