Fix `--exclude` to not touch directories

stdedos · January 5, 2021, 12:33pm

With the following sequence of commands:

rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' "${MY_DIR}" gdrive:"${MY_DIR}"
rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' gdrive:"${MY_DIR}" "${MY_DIR}"

Will resolve in all the --exclude folders being purged.
Can you make them work also the other way (i.e. not touching them also at destination)?

Animosity022 · January 5, 2021, 12:40pm

STOP and READ:
Please show the effort you've put in to solving the problem and please be specific -- people are volunteering their time to help you! Low effort posts are not likely to get good answers!

Did you miss the template and all the actual useful information that we need to collect to give you a hand and validate what's going on?

stdedos · January 5, 2021, 12:55pm

Actually, I just started it as a Feature, and then I decided it's actually a bug (after having read some "maybe duplicate" topics).
That's probably why the template was missing.

I'll fill it later, as I am on a different device right now.

What is the problem you are having with rclone?

What is your rclone version (output from `rclone version`)

$ rclone --version
rclone v1.53.2
- os/arch: linux/amd64
- go version: go1.15.3

or later

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 18.04

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Paste command here

The rclone config contents with secrets removed.

Custom API from tutorial

A log from the command with the `-vv` flag

Paste  log here

Animosity022 · January 5, 2021, 1:23pm

You can use rclone ls to test filter flags and see what the output is.

I'm still really not following on what you are trying to do and what the issue is.

If you exclude something in a filter, it won't modify the destination with said filter.

felix@gemini:~/test1$ rclone sync --exclude test4.out /home/felix/test1 /home/felix/test2 --dry-run
2021/01/05 08:21:18 NOTICE: test1.out: Skipped copy as --dry-run is set
2021/01/05 08:21:18 NOTICE: test2.out: Skipped copy as --dry-run is set
2021/01/05 08:21:18 NOTICE: test3.out: Skipped copy as --dry-run is set
felix@gemini:~/test1$ rclone sync --exclude test4.out /home/felix/test1 /home/felix/test2 -v
2021/01/05 08:21:25 INFO  : test1.out: Copied (new)
2021/01/05 08:21:25 INFO  : test2.out: Copied (new)
2021/01/05 08:21:25 INFO  : test3.out: Copied (new)
2021/01/05 08:21:25 INFO  :
Transferred:          300M / 300 MBytes, 100%, 1.327 GBytes/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:         0.2s

felix@gemini:~/test1$ ls ../test2
test1.out  test2.out  test3.out  test4.out
felix@gemini:~/test1$ ls
test1.out  test2.out  test3.out  test4.out
felix@gemini:~/test1$

Another example with a directory.

felix@gemini:~/test1$ rclone sync --exclude test4.out --exclude test4/** /home/felix/test1 /home/felix/test2 -vv
2021/01/05 08:23:31 DEBUG : rclone: Version "v1.53.3" starting with parameters ["rclone" "sync" "--exclude" "test4.out" "--exclude" "test4/**" "/home/felix/test1" "/home/felix/test2" "-vv"]
2021/01/05 08:23:31 DEBUG : Creating backend with remote "/home/felix/test1"
2021/01/05 08:23:31 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/01/05 08:23:31 DEBUG : Creating backend with remote "/home/felix/test2"
2021/01/05 08:23:31 DEBUG : test4.out: Excluded
2021/01/05 08:23:31 DEBUG : test4: Excluded
2021/01/05 08:23:31 DEBUG : test4.out: Excluded
2021/01/05 08:23:31 DEBUG : test1.out: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/01/05 08:23:31 DEBUG : test1.out: Unchanged skipping
2021/01/05 08:23:31 DEBUG : test2.out: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/01/05 08:23:31 DEBUG : test2.out: Unchanged skipping
2021/01/05 08:23:31 DEBUG : Local file system at /home/felix/test2: Waiting for checks to finish
2021/01/05 08:23:31 DEBUG : test3.out: Size and modification time the same (differ by 0s, within tolerance 1ns)
2021/01/05 08:23:31 DEBUG : test3.out: Unchanged skipping
2021/01/05 08:23:31 DEBUG : Local file system at /home/felix/test2: Waiting for transfers to finish
2021/01/05 08:23:31 DEBUG : Waiting for deletions to finish
2021/01/05 08:23:31 INFO  : There was nothing to transfer
2021/01/05 08:23:31 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 3 / 3, 100%
Elapsed time:         0.0s

2021/01/05 08:23:31 DEBUG : 2 go routines active
felix@gemini:~/test1$ ls ../test2
test1.out  test2.out  test3.out  test4  test4.out

stdedos · January 13, 2021, 2:44pm

I have a directory (python git repo) that I am manually syncing from gdrive.
When I am pushing, there is no need to sync some stuff (.gitignore typical stuff), and therefore I invoke:

rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' "${MY_DIR}" gdrive:"${MY_DIR}"

Afterwards, if I pull with either:

rclone sync gdrive:"${MY_DIR}" "${MY_DIR}"
rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' gdrive:"${MY_DIR}" "${MY_DIR}"

Will result in the --excluded folders missing.

I am not sure how you do it and you cannot replicate it; maybe it's because both of your directories are local? Maybe both of them somehow have been including the files and the files are miraculously not touched?

Animosity022 · January 13, 2021, 3:15pm

Can you run a rclone ls and share the output that comes up? That's normally how I test filters.

stdedos · January 13, 2021, 3:50pm

In both of these examples, you are not showing me how to run a rclone ls; instead you run a sync.

Do you want me to run it against the local or remote directory?
Is it rclone ls "${MY_DIR}" or something more?

Animosity022 · January 13, 2021, 4:22pm

Change the word 'sync' to 'ls' and change the MY_DIR to whatever location you want. Rclone ls just lists out the files and you can test if your filters are working properly and share what the output is.

I normally use ls to test filters and once I'm settled that it does what I want, I use copy/sync/etc.

stdedos · January 16, 2021, 11:28am

Yeah, my filters work. And I have verified it (I won't paste the list, though).

My problem is: Those --excluded items, will be removed as soon as I do either of:

rclone sync gdrive:"${MY_DIR}" "${MY_DIR}"
rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' gdrive:"${MY_DIR}" "${MY_DIR}"

these commands

Animosity022 · January 16, 2021, 11:40am

Can you share the output of the rclone ls with the filters and we can take a look?

stdedos · January 16, 2021, 12:13pm

u@h test_dir_1 → ls -lah
total 48K
drwxr-xr-x  3 u u 4,0K Ιαν  16 13:47 .
drwxr-xr-x 26 u u 4,0K Ιαν  16 13:45 ..
-rw-r--r--  1 u u 1009 Ιαν  16 13:47 Makefile
-rwxr-xr-x  1 u u  445 Ιαν  16 13:45 pull-command.sh
-rwxr-xr-x  1 u u  278 Ιαν  16 13:45 push-command.sh
drwxr-xr-x  5 u u 4,0K Ιαν  16 13:47 .venv
u@h test_dir_1 → rclone ls --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' test_dir_1
     1009 Makefile
      445 pull-command.sh
      278 push-command.sh
u@h projects → rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' test_dir_1 gdrive:'test_dir_1'
u@h projects → rclone ls --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' gdrive:'test_dir_1' 
     1009 Makefile
      445 pull-command.sh
      278 push-command.sh
u@h projects → rclone sync --exclude '.venv/**' --exclude '__pycache__/**' --exclude '.pytest_cache/**' gdrive:'test_dir_1' test_dir_1
u@h projects → ls -lah test_dir_1 
total 48K
drwxr-xr-x  3 u u 4,0K Ιαν  16 14:00 .
drwxr-xr-x 26 u u 4,0K Ιαν  16 13:57 ..
-rw-r--r--  1 u u 1009 Ιαν  16 13:47 Makefile
-rwxr-xr-x  1 u u  445 Ιαν  16 13:45 pull-command.sh
-rwxr-xr-x  1 u u  278 Ιαν  16 13:45 push-command.sh
drwxr-xr-x  5 u u 4,0K Ιαν  16 13:47 .venv

I guess I must have left sometime some of the filters out, which caused the mess.

I depended too much on the backend (fs / Google Drive), that loosing files just blew my head too much.
I assume some kind of visual verification before committing (like e.g. https://github.com/bcpierce00/unison/issues/272#issuecomment-524035318) is outside project's resources?

Animosity022 · January 16, 2021, 12:19pm

Ok, I'm confused now as I'm not sure how the github link you provided fits in with rclone.

Your ls command shows the filter works and excludes.

If you exclude something, it doesn't delete it via sync on the destination, because you've excluded it.

Running the commands with -vv is super helpful as that shares more information and why I've used them in every command.

I can't see anything wrong with what you've shared as every rclone command works.

stdedos · January 16, 2021, 12:22pm

Feature request:

I assume some kind of visual verification before committing (like e.g. https://github.com/bcpierce00/unison/issues/272#issuecomment-524035318) is outside project's resources?

... because ....

... it's not fun to read it when the files are north of 100. Not even the --dry-run output.

I didn't make it clear enough I guess.
I cannot replicate it, and I don't think it's a bug - I guess ...

I guess I must have left sometime some of the filters out, which caused the mess.

system · March 18, 2021, 8:22am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.