Include-from intersection of patterns

Is there a way to use --include-from and --include, and require that the --include pattern be met AND at least one of the patterns in --include-from? As far as I can tell, the way rclone evaluates --include and --include-from is that if one or the other condition is met, the file is included.

My use case is this: I am trying to use include-from to choose files in particular directories of certain file extensions.
For example, I want to select ONLY .bam and .g.vcf for a subset (A, C) of the directories below.

DIRECTORY A
---A.bam
---A.r1.fastq.gz
---A.r2.fastq.gz
---A.vcf
---A.g.vcf
DIRECTORY B
---B.bam
---B.r1.fastq.gz
---B.r2.fastq.gz
---B.vcf
---B.g.vcf
DIRECTORY C
---C.bam
---C.r1.fastq.gz
---C.r2.fastq.gz
---C.vcf
---C.g.vcf

So I tried the command
rclone ls remote: --include="**.{bam,g.vcf}" --include-from=folder_names.txt
But as far as I can tell, this will include every file from A and C, not just the .bam and .g.vcf files.
The desired list of files would be
A.bam
A.g.vcf
C.bam
C.g.vcf

folder_names.txt
*A*
*C*

Thanks for your help!

Edit: I also should add that the number of folders is large and changing, and the number of file types is also large and could change. Both of these lead me to think using --exclude would not be a very robust solution.

If you use --include-from I believe this will override all over includes or excludes, so that is probably why you get weird results. (I think there is an issue logged to fix this from being possible to prevent confusion).

**.{bam,g.vcf} looks sensible to me to match for the file extensions
and if you wanted to match for a certain folder or group of folders and those extensions when why not just put that into the filter? like...
patternA/**.{bam,g.vcf}
That should match for those extensions in all subfolders of any directory that matches "patternA".
Hard to give a more exact example for what "patternA" might look like since your example is pretty abstracted.

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.