Why rclone `--filter` does not filter out `.gitkeep` file?

Hongbo-Miao · April 28, 2023, 7:11pm

What is the problem you are having with rclone?

I have multiple format files in a folder, however, --filter does not filter out with .gitkeep. Any idea? Thanks!

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1
- os/version: darwin 13.2.1 (64 bit)
- os/kernel: 22.3.0 (arm64)
- os/type: darwin
- os/arch: arm64
- go/version: go1.19.4
- go/linking: dynamic
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

S3

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

Neither of these works

rclone sync --progress tdms/data hm-s3:hongbomiao-bucket/iot --filter="+ *.parquet"
rclone sync --progress tdms/data hm-s3:hongbomiao-bucket/iot --filter="+ *.parquet" --filter="- .gitkeep"

The rclone config contents with secrets removed.

I created by

rclone config create hm-s3 s3 provider=AWS access_key_id=xxx secret_access_key=xxx region=us-west-2 storage_class=STANDARD

A log from the command with the `-vv` flag

➜ rclone sync --progress tdms/data hm-s3:hongbomiao-bucket/iot --filter="+ *.parquet" -vv
2023/04/28 12:09:38 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "sync" "--progress" "tdms/data" "hm-s3:hongbomiao-bucket/iot" "--filter=+ *.parquet" "-vv"]
2023/04/28 12:09:38 DEBUG : Creating backend with remote "tdms/data"
2023/04/28 12:09:38 DEBUG : Using config file from "/Users/hongbo-miao/.config/rclone/rclone.conf"
2023/04/28 12:09:38 DEBUG : fs cache: renaming cache item "tdms/data" to be canonical "/Users/hongbo-miao/Clouds/Git/hongbomiao.com/tdms/data"
2023/04/28 12:09:38 DEBUG : Creating backend with remote "hm-s3:hongbomiao-bucket/iot"
2023-04-28 12:09:38 DEBUG : S3 bucket hongbomiao-bucket path iot: Waiting for checks to finish
2023-04-28 12:09:38 DEBUG : S3 bucket hongbomiao-bucket path iot: Waiting for transfers to finish
2023-04-28 12:09:38 DEBUG : .gitkeep: md5 = d41d8cd98f00b204e9800998ecf8427e OK
2023-04-28 12:09:38 INFO  : .gitkeep: Copied (new)
2023-04-28 12:09:44 DEBUG : motor-2.parquet: md5 = c72f5e6a722b43053878a5f7bee12376 OK
2023-04-28 12:09:44 INFO  : motor-2.parquet: Copied (new)
2023-04-28 12:09:44 DEBUG : motor-1.parquet: md5 = 0756da16e4c24c6bc17b243aae7682af OK
2023-04-28 12:09:44 INFO  : motor-1.parquet: Copied (new)
2023-04-28 12:09:44 DEBUG : Waiting for deletions to finish
Transferred:   	   58.687 MiB / 58.687 MiB, 100%, 9.432 MiB/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:         6.7s
2023/04/28 12:09:44 INFO  :
Transferred:   	   58.687 MiB / 58.687 MiB, 100%, 9.432 MiB/s, ETA 0s
Transferred:            3 / 3, 100%
Elapsed time:         6.7s

2023/04/28 12:09:44 DEBUG : 12 go routines active

Animosity022 · April 28, 2023, 7:20pm

It's easier to test with rclone ls

Here are a few examples.

[felix@gemini test]$ rclone ls /home/felix/test
        0 .gitkeep
        0 blah
        0 motor-2.parquet
[felix@gemini test]$ rclone ls /home/felix/test --filter '- *.gitkeep'
        0 blah
        0 motor-2.parquet
[felix@gemini test]$ rclone ls /home/felix/test --filter '- *.gitkeep' --filter '+ *.parquet'
        0 blah
        0 motor-2.parquet
[felix@gemini test]$ rclone ls /home/felix/test --filter '- *.gitkeep' --filter '+ *.parquet' --filter '- *'
        0 motor-2.parquet

Hongbo-Miao · April 28, 2023, 8:00pm

Thanks @Animosity022 !

I think I had misunderstanding, and made some mistakes in my original question. In my case, rclone sync --progress tdms/data hm-s3:hongbomiao-bucket/iot --filter="+ *.parquet" --filter="- .gitkeep" works now. I might make some other mistakes at that time.

But I thought

rclone ls tdms/data --filter="+ *.parquet"

is same with

rclone ls tdms/data --include="*.parquet"

But seems not

Hongbo-Miao · April 28, 2023, 8:03pm

Oh, now I get it!

rclone ls tdms/data --include="*.parquet"

is same with

rclone ls tdms/data --filter="+ *.parquet" --filter="- *"

Animosity022 · April 28, 2023, 8:04pm

include has an exclude ** included at the end of it without doing anything.

That does not as you have to explicitly define it as it is implicit with include/exclude.

 rclone ls /home/felix/test --include="*.parquet" --dump filters
2023/04/28 16:04:15 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)[^/]*\.parquet$
- ^.*$
--- Directory filter rules ---
+ ^.*$
- ^.*$
--- end filters ---
2023/04/28 16:04:15 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "/home/felix/test" "--include=*.parquet" "--dump" "filters"]
2023/04/28 16:04:15 DEBUG : Creating backend with remote "/home/felix/test"
2023/04/28 16:04:15 DEBUG : Using config file from "/opt/rclone/rclone.conf"
        0 motor-2.parquet
2023/04/28 16:04:15 DEBUG : 2 go routines active

and

 rclone ls /home/felix/test --filter="+ *.parquet" --dump filters
2023/04/28 16:04:41 NOTICE: Automatically setting -vv as --dump is enabled
--- start filters ---
--- File filter rules ---
+ (^|/)[^/]*\.parquet$
--- Directory filter rules ---
+ ^.*$
--- end filters ---
2023/04/28 16:04:41 DEBUG : rclone: Version "v1.62.2" starting with parameters ["rclone" "ls" "/home/felix/test" "--filter=+ *.parquet" "--dump" "filters"]
2023/04/28 16:04:41 DEBUG : Creating backend with remote "/home/felix/test"
2023/04/28 16:04:41 DEBUG : Using config file from "/opt/rclone/rclone.conf"
        0 .gitkeep
        0 blah
        0 motor-2.parquet
2023/04/28 16:04:41 DEBUG : 2 go routines active

system · May 1, 2023, 8:05pm

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.