Strange problem with --exclude-from in Gdrive

Hello everyone,

I'm telling you that I have the following somewhat strange problem when ignoring files.

I have 3 Gdrives, on 2 the --exclude-from= function works well for me but on another I can't get it to work. But on the other hand --files-from does work fine on all three Gdrives

I see the logs of the 2 commands, and --files-from ignores the names of those that are not in the .txt (it works correctly),
but --exclude-from does not ignore anything that is in the .txt

Do you know what can happen?

Nobody can see it but you...

What about instead of these vague words posting all details as asked by Help and Support template? Then somebody can try to figure out what the issue is?

OK

The command executed that works is this:
rclone lsf -vv MyDrive:/"My Folder" --files-from="/Rclone SYNCS/file_names.txt" --log-file="/Rclone SYNCS/works.txt"

Log:

2023/10/23 18:45:12 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "lsf" "-vv" "MyDrive:/MyFolder" "--files-from=/Rclone SYNCS/file_names.txt" "--log-file=/Rclone SYNCS/works.txt"]
2023/10/23 18:45:12 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/10/23 18:45:12 DEBUG : Creating backend with remote "MyDrive:/MyFolder"
2023/10/23 18:45:13 DEBUG : fs cache: renaming cache item "MyDrive:/MyFolder" to be canonical "MyDrive:/MyFolder"
2023/10/23 18:45:14 DEBUG : 1.iso: Excluded
2023/10/23 18:45:14 DEBUG : 2.iso: Excluded
2023/10/23 18:45:14 DEBUG : 3.iso : Excluded
2023/10/23 18:45:14 DEBUG : 4.iso: Excluded
2023/10/23 18:45:14 DEBUG : 5.iso: Excluded
2023/10/23 18:45:14 DEBUG : 6.iso: Excluded
2023/10/23 18:45:14 DEBUG : 7.iso: Excluded
2023/10/23 18:45:14 DEBUG : 8.iso: Excluded
2023/10/23 18:45:14 DEBUG : 10.iso: Excluded
2023/10/23 18:45:14 DEBUG : 11.iso: Excluded
2023/10/23 18:45:14 DEBUG : 12.iso: Excluded
2023/10/23 18:45:14 DEBUG : 4 go routines active

And I get files 13 and 14 in the list that are in files_names.txt, this OK

But with this --exclude_from command it doesn't work

rclone lsf -vv MyDrive:"/MyFolder" --exclude-from="/Rclone SYNCS/file_names.txt" --log-file="/Rclone SYNCS/no_works.txt"

Log:

2023/10/23 18:45:25 DEBUG : rclone: Version "v1.53.3-DEV" starting with parameters ["rclone" "lsf" "-vv" "MyDrive:/MyFolder" "--exclude-from=/Rclone SYNCS/file_names.txt" "--log-file=/Rclone SYNCS/no_works.txt"]
2023/10/23 18:45:25 DEBUG : Using config file from "/root/.config/rclone/rclone.conf"
2023/10/23 18:45:25 DEBUG : Creating backend with remote "MyDrive:MyFolder"
2023/10/23 18:45:26 DEBUG : fs cache: renaming cache item "MyDrive:/MyFolder" to be canonical "MyDrive:MyFolder"
2023/10/23 18:45:27 DEBUG : 4 go routines active

And I get a listing of all the files including names 13 and 14. Then it doesn't work :frowning_face:

And what's in that file?

Super super old version. You'd really, really want to update that.

In file_names.txt there is only this

13.iso
14.iso

I'm going to try to install the latest version on my OrangePI 5 with Debian

Seems fine in my testing.

[felix@gemini test]$ ls
12.iso  13.iso  14.iso  1.iso  test.txt
[felix@gemini test]$ rclone ls /home/felix/test
        0 1.iso
        0 12.iso
        0 13.iso
        0 14.iso
       14 test.txt
[felix@gemini test]$ rclone ls /home/felix/test --exclude-from test.txt
        0 1.iso
        0 12.iso
       14 test.txt
[felix@gemini test]$ cat test.txt
13.iso
14.iso
[felix@gemini test]$ rclone ls /home/felix/test --exclude-from test.txt -vv
2023/10/23 13:38:18 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2023/10/23 13:38:18 DEBUG : rclone: Version "v1.64.2-DEV" starting with parameters ["rclone" "ls" "/home/felix/test" "--exclude-from" "test.txt" "-vv"]
2023/10/23 13:38:18 DEBUG : Creating backend with remote "/home/felix/test"
2023/10/23 13:38:18 DEBUG : Using config file from "/opt/rclone/rclone.conf"
        0 1.iso
        0 12.iso
       14 test.txt
2023/10/23 13:38:18 DEBUG : 4 go routines active
[felix@gemini test]$

I have updated to the latest version that is available

rclone v1.64.2
- os/version: debian 11.7 (64 bit)
- os/kernel: 5.10.110-rockchip-rk3588 (aarch64)
- os/type: linux
- os/arch: arm64 (ARMv8 compatible)
- go/version: go1.21.3
- go/linking: static
- go/tags: none

Yes, the --exclude-from function works correctly for me on 2 of my 3 Gdrives, but the surprising thing is that the --files-from function works well on all 3

If you want to share any examples of what's not working, happy to help.

It generally works everywhere..

I have tried it on Windows with version rclone v1.61.1 and it is not working for me either.

I have already put the Logs of what is not working for me, do you need anything else?

Yeah, pretty much share what I've asked.

You can see my commands and what works without a problem.

Update rclone
rclone config redacted - share that output

show a rlcone ls before
show a rclone ls with filter
show the contents of the filter file
rclone ls with the filter used

[MyDrive]
type = drive
scope = drive
token = XXX
team_drive = XXX
root_folder_id =

1.iso
2.iso
3.iso
4.iso
5.iso
6.iso
7.iso
8.iso
10.iso
11.iso
12.iso
13.iso
14.iso

The filter I am using is solely and exclusively the .txt with
13.iso
14.iso

and only the --files-from function works for me

Can you see my output above? I can't see what you are running only output.

Pretty pretty please, make it look like mine :slight_smile:

I wonder if someone else might have another idea.

The problem the OP was hitting was the file names had special characters in it and using exclude-from which you have to escape the special characters. You could mangle the exclude-from and escape it as that requires more scripting as my understanding is the OP was building the file by just a simple grep command.

Is there a different way to approach the problem I might not be thinking of?

[felix@gemini test]$ touch "test (12) [blah].txt"
[felix@gemini test]$ touch "test (13) [blah].txt"
[felix@gemini test]$ touch "test (14) [blah].txt"
[felix@gemini test]$ rclone ls /home/felix/test
        0 test (12) [blah].txt
        0 test (13) [blah].txt
        0 test (14) [blah].txt
[felix@gemini test]$ rclone ls /home/felix/test --exclude 'test (12) [blah].txt'
        0 test (12) [blah].txt
        0 test (13) [blah].txt
        0 test (14) [blah].txt
[felix@gemini test]$ rclone ls /home/felix/test --exclude 'test \(12\) \[blah\].txt'
        0 test (13) [blah].txt
        0 test (14) [blah].txt

There is a file from raw, which does no processing, but nothing on the exclude side.

1 Like

Thank you for your help and your time @Animosity022

I'm not sure if there's a simpler method for what I'm really looking for because what I want is to create an 'ls' after a 'sync,' and then, if I move those files to another folder, I don't want them to be copied again when I launch the automated 'rclone copy' command.

I've worked with the "sed" function when doing an 'ls,' exactly like this:
sed 's//\\[/g; s//\]/g'
With this, I manage to rename the [blah] to /[blah/] in the .txt file that I later use with '--exclude-from.'

It works well for me, but I think the method is too complicated, and I can't find a simpler one.

THANK YOU to everyone.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.