Copy multiple directories wildcard?

Help and Support
1

What is the problem you are having with rclone?

I need to copy a directory list. I have
dir/2021-01-01/dir2
dir/2021-01-02/dir2
dir/2021-01-03/dir3
...
The dates go from 2021-01-01 until 2021-08-31
Is there any way to do this with only one command, or with filters file.txt?

What is your rclone version (output from rclone version)

1.54.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian, 64b

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy remote:dir/*/dir2 /local_dir

The rclone config contents with secrets removed.

A log from the command with the -vv flag

2

you can use
https://rclone.org/filtering/#include-from-read-include-patterns-from-file

not sure exactly what you are asking for..
if you want a one-line command, this could work.
rclone ls . --include='/dir/2021-0{1,2,3,4,5,6,7,8}-??/**'

3

Hello, and thanks. I have seen an error on my message.
I need to copy a directory list. I have
dir/2021-01-01/dir2
dir/2021-01-02/dir2
dir/2021-01-03/dir2
...
dir/2021-08-31/dir2

Note that the dir2 is the same on all cases. And the data are in Google Drive.
So, is it correct?
rclone copy remote --include='/dir/2021-0{1,2,3,4,5,6,7,8}-??/dir2/**'

Another question is the order of the execution. I need run first the more recent dates:

  1. rclone copy remote:dir/2021-08-31/dir2 destination
  2. rclone copy remote:dir/2021-08-30/dir2 destination
  3. rclone copy remote:dir/2021-08-29/dir2 destination
    ...
    rclone copy remote:dir/2021-01-01/dir2 destination

Is it possible run with an order?
Thanks!!

4

I have try
rclone copy remote: destination --include='/dir/2021-0{1,2,3,4,5,6,7,8}-??/dir2/'
and it runs ok, but I obtain multiple destination folders of "dir2"
destination/dir/2021-01-01/dir2/
..
destination/dir/2021-08-31/dir2/**

But I wish to combine all on the same "dir2" folder. I'd like to have all files and folders on
destination/dir2/**

Could you help me?
Thanks!

5

need an example,
can you post a detailed example of the source
and what you want the dest to look like

6

Yes,
I have on SOURCE (Google Drive):
temp/2021-08-31/folderA
temp/2021-08-30/folderA
temp/2021-08-29/folderA
...
More than 200 "folderA" in different dates

And I want to merge all "folderA" in one folder in local. With all the contents (subdirs and files)
On DESTINATION I wish: Only "folderA".

Note:
I want to save the more recent file (if the file is duplicate).
Example:
SOURCE
temp/2021-08-31/folderA/file1.doc
temp/2021-03-30/folderA/file1.doc
Then I want on DESTINATION
folderA/file1.doc (the file1 of 2021-08-31)

Many thanks

7

try rclone ls /path/to/temp --include="/2021-??-??/folderA/**"
get that working first

for the dest, just choose a location like gdrive:folderA

as for most recent files, `rclone copy, by default compares size and mod-time.
if you want to use only mod-time, then add https://rclone.org/docs/#ignore-size

8

Better, but the problem whit your command is that the result are multiple folders and I only want one folder (merge).
In the example I have got:
temp/2021-08-31/folderA/file1.doc
temp/2021-03-30/folderA/file1.doc

But I'd like to obtain:
folderA/file1.doc

Thanks!!!

9

post your command.

10

rclone copy remote:proves --include="/2021-??-??/folderA/**" folderA --ignore-size

11

you are copying from gdrive to local?

12

yes. Remote is Google drive

13

still not sure exactly what you want, need a better example of what the dest should looke like?
does folderA have subfolders? and if yes, what do you want to do with the subfolders?

perhaps try
`--include="/2021-??-??/folderA/*"

14

Ok, I will try to put an easy example.
SOURCE (Google Drive)
proves/2021-01-01/folderA/myfile.jpg
proves/2021-01-01/folderA/subfolderX/document.doc
proves/2021-01-02/folderA/myfile.jpg
proves/2021-01-03/folderA/myfile.jpg

The folderA can have subfolders

All I want on the DESTINATION (local) is:
folderA/myfile.jpg (only the most recent)
folderA/subfolderX/document.doc

15

I only want one folderA
And I obtain multiple folderA

By now I have obtained the same structure that on SOURCE:
folderA/2021-01-01/folderA/myfile.jpg
folderA/2021-01-01/folderA/subfolderX/document.doc
folderA/2021-01-02/folderA/myfile.jpg
folderA/2021-01-03/folderA/myfile.jpg

17

not a linux exper but you want some like

rclone lsf --format=p --include="2021-??-??/folderA/**" gdrive:folderA --recursive --files-only > files.txt

then cut the start of each line to remove the 2021-08-31/folderA/

rclone copy --files-from-raw files.txt gdrive:folderA /path/to/local

18

I'm not a linux expert either, unfortunately. Just playing around until one appears...

I think something like this should work. It will not be one rclone command, though, it will execute a rclone copy command for each unique directory at level proves/2021-01-01/folderA.

for f in $(rclone lsf --recursive --include="/2021-??-??/folderA/**" --files-only gdrive:proves | cut -d/ -f-2 | sort -ru); do rclone copy $f $(echo $f | cut -d/ -f2) --ignore-existing; done
19

Thank you both.

20

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.