Copy/sync Multiple Directories from same drive with 1 rclone command. Filters not applying

What is the problem you are having with rclone?

What i'm trying to achieve: I was going to replicate from this forum: but on a larger scale to keep things organized.
I'm trying to backup with copy for the first time of specific folders then move to sync for changes on c drive, d: drive, e:drive and x:drive to a gdrive mount. :media for testing.
I'm going to be using 4 different scripts shown here: . just need help with the first to apply to the rest. I was unable to use '- *' as it would just make nothing transfer. But as of now all it does is ignore all exclude rules and include rules as well as deny access to certain files as shown in the log file.

What is your rclone version (output from rclone version)

rclone v1.52.2-207-g1fdcc195-vfs-beta

  • os/arch: windows/amd64
  • go version: go1.14.4

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Windows 10 64bit

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy c:\ Media:\test --filter-from "X:\.rclone\Directories\C-Drive.txt" -P --ignore-case --log-file "X:\.rclone\backup\log.txt" -L --cache-tmp-wait-time 7s --drive-chunk-size 64m --buffer-size 512m --dump filters -vv --dry-run

The rclone config contents with secrets removed.

rclone config: 

[GDrive]
type = drive
scope = drive
token = {"access_token":"y
team_drive = 
root_folder_id = 
skip_checksum_gphotos = false
keep_revision_forever = true
stop_on_upload_limit = true
client_id = 
client_secret = 

[Media]
type = crypt
remote = GDrive:/Media
filename_encryption = off
directory_name_encryption = false
password = 
password2 = 

A log from the command with the -vv flag

log file: for -v and -vv: 

pastebin can't read it (it crashes) Log File

hello and welcome to the forum,

as rclone is not a backup program using VSS, rclone cannot access certain files on your system.
you might have better luck using my wiki
https://github.com/rclone/rclone/wiki/How-to-enable-VSS-for-rclone
really, nothing you can do about that other than using a real backup program.

looks like the filters might be working
2020/09/17 16:49:53 DEBUG : Users/xxxxx/.VirtualBox VMs/Android 10/Logs/VBox.log: Excluded

ive been using robocopy to copy files from c drive to a large poll on my 4tb hard drive.(e) for the longest time but that was all locally done and did not involve cloud storage. now as for x: for videos/tv etc. would rclone be enough to move those to the cloud as id just be playing them from the cloud anyways?

the filters work for excluding .log's but it doesn't exclude anything past that like windows/recycle bin/desktop etc.

did you post the --filter-from files?

reason im using rclone is to 1, be an alternative of manually dragging files everynight thru filezilla and accidently forgetting things. but also to use the sync command on rclone, and 2. to encrypt and mount the share's so i can stream the videos from the drive without keeping them locally as a sort of "local plex" but threw mkvplayer instead of plex.

Oh i didn't i forgot to, i did it should be:

- *.log**
- desktop.ini**
- C:/$WinREAgent/**
- C:/.Move To Desktop/**
- C:/ffmpeg-4.2.1-win64-static/**
- C:/KeyLock/**
- C:/nvflash_5.620.0/**
- C:/PerfLogs/**
- "C:/Program Files/**"
- "C:/Program Files (x86)/**"
- C:/ProgramData/**
+ C:/RealBench_v2.43/**
- C:/Users/**
- C:/Windows/**
+ C:/Games/**
+ "C:/Program Files/Rockstar Games/**"
+ "C:/Program Files (x86)/Black Desert Online/**"
+ "C:/Program Files (x86)/MSI Afterburner/**"
+ C:/Users/UwU/.config/**
+ C:/Users/UwU/.rclone/**
+ C:/Users/UwU/.VirtualBox/**
+ C:/Users/UwU/.VirtualBox VMs/**
+ C:/Users/UwU/AppData/Local/CodeVein/**
+ C:/Users/UwU/AppData/Local/Corsair/**
+ C:/Users/UwU/AppData/Local/CorsairOsdLauncher/**
+ C:/Users/UwU/AppData/Local/Origin/**
+ C:/Users/UwU/AppData/Local/Programs/Python/**
+ C:/Users/UwU/AppData/Local/Steam/**
+ C:/Users/UwU/AppData/Local/Warframe/**
+ C:/Users/UwU/AppData/Roaming/BetterDiscord/**
+ C:/Users/UwU/AppData/Roaming/Citra/**
+ C:/Users/UwU/AppData/Roaming/Corsair/**
+ C:/Users/UwU/AppData/Roaming/Origin/**
+ C:/Users/UwU/AppData/Roaming/yuzu/**
+ C:/Users/UwU/AppData/Roaming/qBittorrent/**
+ C:/Users/Public/Documents/Steam/**
+ C:/ProgramData/Origin/**

i grabbed the same directories from my robocopy job as a replacement as it didn't have a great excluding feature to ignore the .log files and desktop.ini's
I tried using robocopy to copy everything to google drive but it "crashed and burned" by filling up my drives due to it caching from writing to the mounted drive instead of doing it threw rclone with the copy/sync command.

i am not a expert of filtering but each filter is relative to the source.
instead of C:/$WinREAgent/** try /$WinREAgent/**

would the source be relevant to the original script of: rclone copy "c:\" then?

in that command, the source is c:\

let me do a dry run real quick and see if it works. one momment

I think its working? but i'm not sure id have to wait to see if anything did not get copied over that it wasn't supposed to. but if it does would I have to exclude every root folder for everything else since the -* doesn't register/excludes all files.

edit: never mind it still ignores the filters. and copies files that aren't in the include syntax

if you use an include filter, then you would not need to exclude every root folder.
https://rclone.org/filtering/#include-from-read-include-patterns-from-file

include filter should be + correct?

i made a smaller test filter for it to see if it would work going to go test it.

- .log**
- /$WinREAgent**
- /.Move To Desktop**
- /ffmpeg-4.2.1-win64-static**
- /KeyLock**
- /nvflash_5.620.0**
- /PerfLogs**
- "/Program Files"**
- "/Program Files (x86)"**
- /ProgramData**
- /RealBench_v2.43**
- /Users**
- /Windows**
+ "/Program Files/Rockstar Games"**
+ "/Program Files (x86)/Black Desert Online"**
+ /Users/UwU/.VirtualBox VMs**

nope it still ignores it: https://i.imgur.com/QGXBELZ.png it transfers recycle bin files now. and with the -* it doesnt transfer anything at all.

2020/09/17 17:58:54 INFO  : There was nothing to transfer
2020/09/17 17:58:54 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

for:

- .log**
- /$WinREAgent**
- /.Move To Desktop**
- /ffmpeg-4.2.1-win64-static**
- /KeyLock**
- /nvflash_5.620.0**
- /PerfLogs**
- "/Program Files"**
- "/Program Files (x86)"**
- /ProgramData**
- /RealBench_v2.43**
- /Users**
- /Windows**
+ "/Program Files/Rockstar Games"**
+ "/Program Files (x86)/Black Desert Online"**
+ /Users/UwU/.VirtualBox VMs**
- *

from what I read it showed for the include rule (in a fitler file) would be + and exclude shows as - on: --filter - Add a file-filtering rule

i gave you the link, did you read it?
https://rclone.org/filtering/#include-from-read-include-patterns-from-file

I guess i'm doing it backwards and still going threw filter because of what was said in the other thread. I guess i should just be using the include from file. instead of filtering it all? then adding a sepereate exclude from to exclude .log files and desktop.ini's?

filters can be very confusing and i mentioned i am no expert.
if you want to include and exclude, then --filter-from should work.

also, you can do this
https://rclone.org/filtering/#repeating-options

yes i was originally doing that and it either throws out the giant error i was getting at the start or:

2020/09/17 18:11:42 DEBUG : rclone: Version "v1.52.2-207-g1fdcc195-vfs-beta" starting with parameters ["rclone" "copy" "c:\\" "Media:\\test" "--include-from" "X:\\.rclone\\Directories\\C-Drive.txt" "-P" "--ignore-case" "--log-file" "X:\\.rclone\\backup\\log.txt" "-L" "--cache-tmp-wait-time" "7s" "--drive-chunk-size" "64m" "--buffer-size" "512m" "--dump" "filters" "-vv"]
2020/09/17 18:11:42 DEBUG : Using config file from "C:\\Users\\UwU\\.config\\rclone\\rclone.conf"
2020/09/17 18:11:42 DEBUG : fs cache: renaming cache item "c:\\" to be canonical "//?/c:/"
2020/09/17 18:11:44 DEBUG : fs cache: renaming cache item "Media:\\test" to be canonical "Media:/test"
2020/09/17 18:11:44 DEBUG : $Recycle.Bin: Excluded
2020/09/17 18:11:44 DEBUG : $WinREAgent: Excluded
2020/09/17 18:11:44 DEBUG : .Move To Desktop: Excluded
2020/09/17 18:11:44 DEBUG : Config.Msi: Excluded
2020/09/17 18:11:44 DEBUG : Documents and Settings: Excluded
2020/09/17 18:11:44 DEBUG : DumpStack.log.tmp: Excluded
2020/09/17 18:11:44 DEBUG : ffmpeg-4.2.1-win64-static: Excluded
2020/09/17 18:11:44 DEBUG : Games: Excluded
2020/09/17 18:11:44 DEBUG : hiberfil.sys: Excluded
2020/09/17 18:11:44 DEBUG : KeyLock: Excluded
2020/09/17 18:11:44 DEBUG : nvflash_5.620.0: Excluded
2020/09/17 18:11:44 DEBUG : pagefile.sys: Excluded
2020/09/17 18:11:44 DEBUG : PerfLogs: Excluded
2020/09/17 18:11:44 DEBUG : Program Files: Excluded
2020/09/17 18:11:44 DEBUG : Program Files (x86): Excluded
2020/09/17 18:11:44 DEBUG : ProgramData: Excluded
2020/09/17 18:11:44 DEBUG : RealBench_v2.43: Excluded
2020/09/17 18:11:44 DEBUG : Recovery: Excluded
2020/09/17 18:11:44 DEBUG : swapfile.sys: Excluded
2020/09/17 18:11:44 DEBUG : System Volume Information: Excluded
2020/09/17 18:11:44 DEBUG : Users: Excluded
2020/09/17 18:11:44 DEBUG : Windows: Excluded
2020/09/17 18:11:44 DEBUG : Program Files: Excluded
2020/09/17 18:11:44 DEBUG : $Recycle.Bin: Excluded
2020/09/17 18:11:44 DEBUG : RealBench_v2.43: Excluded
2020/09/17 18:11:44 DEBUG : ffmpeg-4.2.1-win64-static: Excluded
2020/09/17 18:11:44 DEBUG : .Move To Desktop: Excluded
2020/09/17 18:11:44 DEBUG : desktop.ini: Excluded
2020/09/17 18:11:44 DEBUG : Encrypted drive 'Media:/test': Waiting for checks to finish
2020/09/17 18:11:44 DEBUG : Encrypted drive 'Media:/test': Waiting for transfers to finish
2020/09/17 18:11:44 INFO  : There was nothing to transfer
2020/09/17 18:11:44 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2020/09/17 18:11:44 DEBUG : 6 go routines active

from: replacing filter-from to -include-from

+ "/Program Files/Rockstar Games"**
+ "/Program Files (x86)/Black Desert Online"**
+ /Users/UwU/.VirtualBox VMs**

okay so i removed the include directions in the file and it started to copy: https://i.imgur.com/RqsZghb.png

--cache-tmp-wait-time, if you are not using the cache backend, you do not need that flag.
--dump filters not sure why you need that?

only issue i think now is grabbing folders with spaces in them as it doesn't seem to pick up the "folder" and excluding the .log and desktop.ini's as it skipped program files all together and went to /Games .

as for the --dump filters was for testing to see what it was doing. for the cache-temp-wait-time, i kept it there for testing just incase it tries to cache something when i was using sync instead of copy but it seems not needed anymore.

2020/09/17 18:21:14 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Transferred:         8743 / 8743, 100%
Elapsed time:         0.0s

2020/09/17 18:21:14 DEBUG : 6 go routines active

just ran the dry run of it, it grabs the right files and it auto excludes everything else not in that file. but it doesn't seem to pick up the "folders with spaces in them" But for the majority at least its working properly now.