--files-from not working, all files and folders end up being excluded

Help and Support
1

What is the problem you are having with rclone?

Trying to copy a folder with a selection of subfolders and its contents

What is your rclone version (output from rclone version)

1.50

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64 bit

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy --files-from listoffiles.txt remote:myfolder <local_directory> -P -v

My listoffiles.txt looked like this:
A/**
B/**
C/**

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

All the subfolders and its contents in the folder myfolder ends up to be excluded.

2

Can you please run the command with that and share the contents of the listoffiles.txt.

3

--files-from takes names of files. If you want patterns then you want --include-from most likely.

4

yea the input format here looks wrong.

If you want an idea of what the input format looks like - run rclone lsf myremote:
That is a compatible format (I have recently learned).

5

So what about folders?

I tried using --include-from but it's not working as well (all folders gets excluded)

However when I tried to list files using ls with --files-from, it works.

6

You can match for folders too if you want...
For simple patterns use
--include GLOB-PATTERN

for complex patterns (many rules) you can set the rules in a file and use
--include-from mypatterns.txt

If you want an example of how to match for a folder, you either need to specify exactly what it should or should not match (patterns need to be very spesific) - or you can just show me an example of a folder structure and point to one that you want the pattern to match, and I'll try to provide an example.

But generally, a folder match would look something along the lines of
--include **foldername*/**
That should match any folder that has "foldername" anywhere in it's name (a partial match), and then transfer all files inside of it, including files in any of it's subfolders.
But there are many variations here depending on what exactly you need...

7

OK so I just wanted to transfer a selection of folders from a main folder. My main folder (call it main) has these subfolders:

AAA/subfolder/contents
BBB/contents
CCC/subfolder1/subfolder2/contents
DDD/contents
EEE/contents

Now I just want to transfer folder AAA, BBB and CCC along with all the contents and subfolders in the folder AAA, BBB and CCC. How should I go around with this?

Note that it needs to be in a text file as the list is quite big.

@thestigma

8

if you were to use just --include (which I know you didn't ask for, but bear with me):
--include **AAA/** --include **BBB/** --include **CCC/**

I haven't had a need to use the --include from file, but the file should just need to look like this I think:

   + **AAA/** 
   + **BBB/**
   + **CCC/**

The same patterns in other words. + or - for include/exclude

https://rclone.org/filtering/#include-from-read-include-patterns-from-file

Technically this matches any folder that ends in AAA (even if it has more at the start).
A direct match would be **/AAA/** , but that will not work for any folders that are in level 1 (in the root).
I don't know of a way to get it both ways (direct match but even on level 1).

9

That didn't work either - all files just got excluded

I think I got it somewhat - it seems like I'll need to use exclude instead of include

10

Also @ncw can you try to include an option call include-only? Because I managed to solve my issue by excluding the folders that I don't need, but I wanted an option that downloads the only files required.

11

I do not get what was your problem regards on --include-from, since I have the exact commands and they are doing just fine.

12

What did you come up with in the end?

13

I'll just use --exclude-from, and excluding the folders that I don't need like this:

AAA/**
BBB/**

@ncw

14

are you using plus and minus signs to indicate include or exclude in the file? (see my example + the example in the documentation link I provided)

I'm not actually sure what happens if you use no sign, but you might end up getting just one of the two modes by default.

15

No, I didn't use the plus and minus signs.

16

Give that a try then :slight_smile:

17

I did, but the problem still persists - so I have to use the exclude method. Although it's quite tedious and I may end up excluding the wrong folder :confused:

18

Just to check something basic, you aren't confusing --files-from with --include-from right? I've made that brainfart before too.

--files-from requires a full list of all files to include
--include-from contains rules in GLOB format, and is what would be appropriate to use there.

The logic for exclude and include works the same way. If include is not working as you'd expect and you aren't confusing it with --files-from then it's almost certainly a problem in the matching pattern you are specifying.

19

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.