Filter-from command

What is the problem you are having with rclone?

I am trying to run a rclone copy with a filter from file. It seems like I can exclude the directories, however aside from that it is downloading everything including things not in the filter.

You will see from the log that rclone is downloading files that don't include what my filter file has.

Run the command 'rclone version' and share the full output of the command.

rclone v1.61.1

  • os/version: linuxmint 21 (64 bit)
  • os/kernel: 5.15.0-56-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.4
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Google Drive (remote) to download locally

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy "remote:backup" "/mnt/12tb" --checksum --drive-chunk-size 128M --max-backlog 999999 --fast-list --filter-from "$script_directory/filter.txt" --no-update-modtime --progress --ignore-existing 

filter.txt

- /demo/
- /demo_updates/
+ *[01007EF00011E000]*
+ *[0100225000FEE000]*
+ *[0100BCE000598000]*
+ *[0100B42001DB4000]*
+ *[01009E20141AD004]*
+ *[0100A6301214F001]*
- **

The rclone config contents with secrets removed.

[remote]
type = drive
client_id = *
client_secret = *
scope = drive
token = *
team_drive = *
root_folder_id = 

A log from the command with the -vv flag

2022/12/28 23:11:39 DEBUG : rclone: Version "v1.61.1" starting with parameters ["rclone" "copy" "remote:backup" "/mnt/12tb" "--checksum" "--drive-chunk-size" "128M" "--max-backlog" "999999" "--fast-list" "--filter-from" "/home/joey/scripts/scripts/filter.txt" "--no-update-modtime" "--progress" "--ignore-existing"]
2022/12/28 23:11:39 DEBUG : Creating backend with remote "remote:backup"
2022/12/28 23:11:39 DEBUG : Using config file from "/home/joey/.config/rclone/rclone.conf"
2022/12/28 23:11:39 DEBUG : remote: detected overridden config - adding "{OgfZc}" suffix to name
2022/12/28 23:11:39 DEBUG : remote{OgfZc}: Loaded invalid token from config file - ignoring
2022/12/28 23:11:39 DEBUG : Saving config "token" in section "remote" of the config file
2022/12/28 23:11:39 DEBUG : remote{OgfZc}: Saved new token in config file
2022/12/28 23:11:39 DEBUG : fs cache: renaming cache item "remote:backup" to be canonical "remote{OgfZc}:nsz"
2022/12/28 23:11:39 DEBUG : Creating backend with remote "/mnt/12tb"
2022/12/28 23:11:39 DEBUG : demo_updates: Excluded
2022/12/28 23:11:40 DEBUG : demo_updates: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/ć‚Æćƒ¬ć‚¤ć‚øćƒ¼ć‚­ćƒ£ćƒ—ćƒćƒ£ćƒ¼ [010077B012D4A000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Pixel Cup Soccer - Ultimate Edition [0100F08019784000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Witch on the Holy Night [0100697019538000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Machinika Museum [0100950019452000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/ē™½ę˜¼å¤¢ć®é’写ēœŸ [0100FB7019BC8000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Ace Angler Fishing Spirits [01004D901984C000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Yomawari Lost in the Dark [010022B019786000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Fragment's Note+ [0100AF801910C000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Shutter Nyan! Enhanced Edition [0100221019468000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/ćƒ‰ćƒ©ćˆć‚‚ć‚“ ć®ć³å¤Ŗ恮ē‰§å “ē‰©čŖž 大č‡Ŗē„¶ć®ēŽ‹å›½ćØćæ悓ćŖć®å®¶ [0100712018062000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:40 DEBUG : demo/Youropa [0100A7701943A000][v0].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/ćƒ¢ćƒ«ć‚µćƒƒć‚«ćƒ¼ć‚Ŗćƒ³ćƒ©ć‚¤ćƒ³ Lite [010034501756C800][v196608].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/惜ć‚Æć‚‰ć®ć‚¹ć‚Æćƒ¼ćƒ«ćƒćƒˆćƒ« [0100B56010A34800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/HARVESTELLA [0100A280187BC800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/Resident Evil Village Cloud [0100CA0018C52800][v131072].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/BIOHAZARD VILLAGE Z Version - Cloud Version [01005C9018C58800][v196608].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/Quest Hunter [01001D300FEC2800][v458752].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/60 Seconds! Reatomized [01004A8015DD0800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/ēøćŒć‚ē”·å­ćØć‘ć‚‚ć®ćŸć‚“ [010079801502C800][v393216].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/Yu-Gi-Oh! Master Duel [010039E012F40800][v524288].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/ęœŗć§ć‚µćƒƒć‚«ćƒ¼ 体éؓē‰ˆ [010025000B836800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/恂恏恂悊恆悀怂 [01004160191C8800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/CRSED F O A D [0100236015946800][v458752].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/Automachef [01006B700EA6A800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/What Lies in the Multiverse [01006E3018574800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:11:41 DEBUG : demo_updates/A Plague Tale Requiem - Cloud Version Demo [0100148018130800][v65536].nsz: Excluded from sync (and deletion)
2022/12/28 23:12:07 DEBUG : Local file system at /mnt/12tb: Waiting for checks to finish
2022/12/28 23:12:07 DEBUG : Local file system at /mnt/12tb: Waiting for transfers to finish
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: Starting multi-thread copy with 4 parts of size 480.188Mi
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: multi-thread copy: stream 4/4 (1510539264-2013926486) size 480.067Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: multi-thread copy: stream 2/4 (503513088-1007026176) size 480.188Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: multi-thread copy: stream 1/4 (0-503513088) size 480.188Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: multi-thread copy: stream 3/4 (1007026176-1510539264) size 480.188Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Panty & Stocking with Garterbelt [050050CE11E68000][v0].nsz: Starting multi-thread copy with 4 parts of size 424.250Mi
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Panty & Stocking with Garterbelt [050050CE11E68000][v0].nsz: multi-thread copy: stream 4/4 (1334575104-1779174573) size 424.003Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Panty & Stocking with Garterbelt [050050CE11E68000][v0].nsz: multi-thread copy: stream 3/4 (889716736-1334575104) size 424.250Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Panty & Stocking with Garterbelt [050050CE11E68000][v0].nsz: multi-thread copy: stream 1/4 (0-444858368) size 424.250Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Panty & Stocking with Garterbelt [050050CE11E68000][v0].nsz: multi-thread copy: stream 2/4 (444858368-889716736) size 424.250Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - SteinsGate [050050CE11E98000][v0].nsz: Starting multi-thread copy with 4 parts of size 768.500Mi
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - SteinsGate [050050CE11E98000][v0].nsz: multi-thread copy: stream 4/4 (2417491968-3223098444) size 768.286Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - SteinsGate [050050CE11E98000][v0].nsz: multi-thread copy: stream 1/4 (0-805830656) size 768.500Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - SteinsGate [050050CE11E98000][v0].nsz: multi-thread copy: stream 3/4 (1611661312-2417491968) size 768.500Mi starting
2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - SteinsGate [050050CE11E98000][v0].nsz: multi-thread copy: stream 2/4 (805830656-1611661312) size 768.500Mi starting
2022/12/28 23:12:08 DEBUG : homebrew/Anime Movie Player - Your Lie in April [0500E8500A780000][v0].nsz: Starting multi-thread copy with 4 parts of size 803.750Mi
2022/12/28 23:12:08 DEBUG : homebrew/Anime Movie Player - Your Lie in April [0500E8500A780000][v0].nsz: multi-thread copy: stream 4/4 (2528378880-3371143239) size 803.723Mi starting
2022/12/28 23:12:08 DEBUG : homebrew/Anime Movie Player - Your Lie in April [0500E8500A780000][v0].nsz: multi-thread copy: stream 2/4 (842792960-1685585920) size 803.750Mi starting
2022/12/28 23:12:08 DEBUG : homebrew/Anime Movie Player - Your Lie in April [0500E8500A780000][v0].nsz: multi-thread copy: stream 1/4 (0-842792960) size 803.750Mi starting
2022/12/28 23:12:08 DEBUG : homebrew/Anime Movie Player - Your Lie in April [0500E8500A780000][v0].nsz: multi-thread copy: stream 3/4 (1685585920-2528378880) size 803.750Mi starting

Add in ** after the directory names as Iā€™m assuming you want to exclude everything g under them.

- /demo/**

Thanks for the reply, however if you look at the log all /demo/ and /demo_updates/ files are excluded regardless of the wildcard. I did add them just to see if it would fix the rest but it did the same output.

if you look at the log:

2022/12/28 23:12:07 DEBUG : homebrew/Anime Movie Player - Made in Abyss [050050CE11E48000][v0].nsz: Starting multi-thread copy with 4 parts of size 480.188Mi

This file does not contain any of the following:

+ *[01007EF00011E000]*
+ *[0100225000FEE000]*
+ *[0100BCE000598000]*
+ *[0100B42001DB4000]*
+ *[01009E20141AD004]*
+ *[0100A6301214F001]*

Which is where my problem comes from.

Thanks again.

Sorry as Iā€™m a bit confused on what you are trying to match then.

The [] are ranges of characters.

Is that what you want? Can you explain in simple terms what you are trying to match and copy?

You figured out the problem! I didn't realize [] mean't a range. Apparently I'm blind since I just noticed it in the docs. I changed it to:

+ *\[01007EF00011E000\]*

And it works!

Thanks for the help!

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.