How do I copy several different dirs to on location

[nelson777 (Nelson Teixeira)]

What is the problem you are having with rclone?

I want to copy several different folders to one remote folder but couldn't find a way to do it in one command.

I have tried to use interactive mode but it would include the (many) dot folders. Also it doesn't let me choose the dirs beforhand which leads to making me have to wait for the files to be copied in each dir and there are 1 million+ files there which makes waiting for each dir very impratical.
I tried at least to get rid of the dot folders and others with --exclude=.** or --exclude=/.** but it didn't work.

I found a very ugly solution of using the following script:

#!/bin/bash

rclone -P -v --transfers=10 --fast-list copy ~/bin gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/Images gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/Music gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/Desktop gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/bkp gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/data gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/dev gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/Documents gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/Downloads gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log
rclone -P -v --transfers=10 --fast-list copy ~/progs gdrive:bkp/desktop-2024-05-09 --log-file=copylog.log

but I'm sure there should be something better than this...

What I want is a command that let me specify the dirs I want to copy and a unique destination.

Run the command 'rclone version' and share the full output of the command.

rclone v1.53.3-DEV

• os/arch: linux/amd64
• go version: go1.18.1

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -i -P -v --transfers=10 --fast-list copy ~/Images ~/bin ~/Documents  gdrive:bkp/test

Please run 'rclone config redacted' and share the full output. If you get command not found, please make sure to update rclone.

[gdrive]
type = drive
client_id = XXX
client_secret = XXX
scope = drive
token = XXX

A log from the command that you were trying to run with the -vv flag

2024/05/09 03:16:30 DEBUG : rclone: Version "v1.66.0" starting with parameters ["rclone" "-i" "-P" "-vv" "--transfers=10" "--fast-list" "copy" "/home/nelson/Images" "/home/nelson/bin" "/home/nelson/Documents" "gdrive:bkp/test"]
Usage:
  rclone copy source:path dest:path [flags]
(... help page )

[kapitainsky]

You can use filter:

rclone copy /home/nelson gdrive:bkp/test \
--filter "+ /Images/**" \
--filter "+ /bin/**" \
--filter "+ /Documents/**" \
--filter "- *"

include what you need and exclude everything else. If you have a lot of rules you have to use then you can simplify this command by using --filter-from

BTW:

@nelson777

rclone v1.53.3-DEV

You are using ancient version of rclone - remove it and install the latest one directly from Install

[nelson777 (Nelson Teixeira)]

This made a mess of the copy that was already in the middle. It put all files and subfolders of the subfolders in one dir not creating the original dirs.
I took for granted that it was obvious that when I said copy to one location should be each dir to that location. Not the lower files. Sorry for that.
I this this way will be more clear what I'm trying to accomplish here:

I have this in

/home/nelson
├── Android
├── Desktop
├── bin
├── bkp
├── data
├── dev
├── Documents
├── Downloads
├── Images
├── Models
├── Music
├── Postman
├── AppData
├── progs
├── Public
├── t
├── Videos
└── VirtualBox VMs

And I want this in remote:

/bkp/desktop-2024-05-09
├── Desktop
├── bin
├── bkp
├── data
├── dev
├── Documents
├── Images
├── Music
└── progs

Is there a any to do that in one command ?

[kapitainsky]

It includes what is explicitly included only:

--filter "+ /Images/**" \
--filter "+ /bin/**" \
--filter "+ /Documents/**" \

so for example everything in /Images/ directory.

and excludes everything else:

--filter "- *"

You can always add additional exclusions if needed. Just keep in mind that rules are processed in the order they are defined. So pay attention to their order. Arrange the order of filter rules with the most restrictive first and work down.

Experiment with rules until you define exactly what you need. You can always use --dry-run for testing.

[kapitainsky]

@nelson777

Is there a any to do that in one command ?

Yes. Exactly one I showed you - use filters.

@nelson777

This made a mess of the copy that was already in the middle. It put all files and subfolders of the subfolders in one dir not creating the original dirs.

Not sure what you mean here. It does creates original dirs. Post your exact command so we will have a look.

[nelson777 (Nelson Teixeira)]

I already tried. It copyed the content of the folder... I wanted to copy the folder itself please see the trees I mentioned in the post above.

[kapitainsky]

Tried what? What is your exact command?

This should be:

rclone copy /home/nelson gdrive:bkp/test \
--filter "+ /Images/**" \
--filter "+ /bin/**" \
--filter "+ /Documents/**" \
--filter "- *"

BTW - this is one line. I use "\" only to make it easier to read.

[nelson777 (Nelson Teixeira)]

This is the command I'm using:

rclone -P -v --transfers=10 --fast-list --drive-chunk-size=256M copy /home/nelson gdrive:bkp/desktop-2024-05-09 \ 
--filter "+ /bin /**" \ 
--filter "+ /Images /**" \ 
--filter "+ /Music /**" \ 
--filter "+ /Desktop /**" \ 
--filter "+ /bkp /**" \ 
--filter "+ /data /**" \ 
--filter "+ /des /**" \ 
--filter "+ /Documents /**" \ 
--filter "+ /Downloads /**" \ 
--filter "+ /progs /**"  

It's not limiting itself to the dirs... it's copying the dot dirs in the root folder.

Please note that I don´t want to exclude dot files in the backuped dirs

[kapitainsky]

Because this is different... to what I posted:)

For example you use:

--filter "+ /bin /**" \ 

when I use:

--filter "+ /bin/**" \ 

In addition you omitted:

--filter "- *"

So your command is doing something totally different - not even going to check it.

@nelson777

It's not limiting itself to the dirs... it's copying the dot dirs in the root folder.

Yes - it does what your filter defines. You copy everything.

Use my example without any modification and it will work:)

[nelson777 (Nelson Teixeira)]

Yes it´s working... thanks. Sorry for my confusion.

[kapitainsky]

Filtering can be confusing. But it works when all done correctly.