Noticed a lot of passer / sleeps. Along with trying to:
Kee certain directories not being deleted from Sorce as they are used and tied to the folder id for my custom python script I don't really see a way to exclude or filter-from so certian root directories aren't deleted without resorting to
2021/12/05 19:16:15 NOTICE: A-Library/Movies: Skipped remove directory as --dry-run is set 2021/12/05 19:16:15 NOTICE: A-Library: Skipped remove directory as --dry-run is set
What is your rclone version (output from rclone version)
Windows(Actual os version: Microsoft Windows 11 Pro for Workstations 2009 (64 bit)
rclone v1.57.0
os/version: Microsoft Windows 10 Pro for Workstations 2009 (64 bit)
os/kernel: 10.0.22000.348 (x86_64)
os/type: windows
os/arch: amd64
go/version: go1.17.2
go/linking: dynamic
go/tags: cmount
Which cloud storage system are you using? (eg Google Drive)
Google Drive but local also.
The command you were trying to run (eg rclone copy /tmp remote:tmp)
and placing the same file in those directories but still keeps deleting them with dry run.
I ran the script with and without the filter applied and searching for them did apear in the one without the filter. but either way it still deletes the root folders when a file is in the path even if its set to be ignored and not moved.
2021/12/05 22:23:44 NOTICE: A-Library/Shows/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: K-Library/Movies/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: #Temp/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: K-Library/Shows/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: S-Library/Shows/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: A-Library/Movies/Folder.Drives: Skipped move as --dry-run is set (size 5)
2021/12/05 22:23:44 NOTICE: S-Library/Movies/Folder.Drives: Skipped move as --dry-run is set (size 5)
with --filter in the command:
2021/12/05 22:23:44 NOTICE: A-Library/Movies: Skipped remove directory as --dry-run is set
2021/12/05 22:23:44 NOTICE: A-Library: Skipped remove directory as --dry-run is set
2021/12/05 22:23:44 NOTICE: #Temp: Skipped remove directory as --dry-run is set
I went ahead and moved it to the first command so different remotes but same process, and same file tree/files that are needed to stay as "placeholders" so the directories aren't deleted. just not a crypted mount for simplicity as those are the actual folder ids that I need to preserve.
Command: would be removing dry run after testing.
What I'm asking is run rclone ls with your filters for testing rather than move as you can see what shows up in the ls rather than trying move something or dry run it.
there not directories no. they are pointing to actual files to try and force them to not be empty so --delete-empty-src-dirs won't delete them because in theory they won't be empty if those files are excluded from being moved.
running the ls with the filter it does apply and those files no longer show up in the ls, or in a move command however when i run the actual command it ignores said filter and deletes those directories anyways.
The Folder.Filter file is only 5bytes if that matters at all.
Wouldn't be any point to sync as its just a temp directory tree that gets moved to the crypt part of the drive then moved to my drive and would delete what was moved if i take off dry run but.
LS command:
https://pastebin.com/Dm5sJ5kP
rclone sync with dry run:
C:\Users\PC>rclone sync GSDLC: GSDLM: --filter-from "C:\Users\PC\Desktop\Folder.Filter" -L -P --ignore-case-sync --fast-list --delete-empty-src-dirs --fast-list --drive-chunk-size 128m --buffer-size 16m --log-file "C:\Users\PC\Desktop\TestLog.log" -vv --dry-run
Error: unknown flag: --delete-empty-src-dirs
Usage:
rclone sync source:path dest:path [flags]
Flags:
--create-empty-src-dirs Create empty source dirs on destination after sync
-h, --help help for sync
Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.
2021/12/06 08:22:29 Fatal error: unknown flag: --delete-empty-src-dirs
C:\Users\PC>
Since you neglected the rclone.conf, I had to guess. Are you seeing API hits on it?
So let's take one filter, you have:
A-Library/Movies/Folder.Drives
So that excludes it from the move and deletion.
So if that folder is excluded and A-Library/Movies is empty (no files) and A-Library is empty (no files), it would remove it since you've excluded it and they are empty as you've moved all the files with the move command.
I think it's doing exactly what you asked, but not what you want it to do.
So I personally wouldn't do this but in reality, if you exclude a file and you try to remove a directory that rclone thinks is empty but a file is there, it'll error out:
felix@gemini:~$ rclone move /home/felix/test /home/felix/test1 -vv --exclude "A-Library/Movies/Folder.Drives"
2021/12/06 11:33:13 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/12/06 11:33:13 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test1" "-vv" "--exclude" "A-Library/Movies/Folder.Drives"]
2021/12/06 11:33:13 DEBUG : Creating backend with remote "/home/felix/test"
2021/12/06 11:33:13 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/12/06 11:33:13 DEBUG : Creating backend with remote "/home/felix/test1"
2021/12/06 11:33:13 DEBUG : A-Library/Movies/Folder.Drives: Excluded
2021/12/06 11:33:13 DEBUG : Local file system at /home/felix/test1: Waiting for checks to finish
2021/12/06 11:33:13 DEBUG : Local file system at /home/felix/test1: Waiting for transfers to finish
2021/12/06 11:33:13 INFO : There was nothing to transfer
2021/12/06 11:33:13 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Elapsed time: 0.0s
2021/12/06 11:33:13 DEBUG : 3 go routines active
felix@gemini:~$ cd test
felix@gemini:~/test$ ls
A-Library one two
felix@gemini:~/test$ cd ..
felix@gemini:~$
felix@gemini:~$ cd
felix@gemini:~$ rclone move /home/felix/test /home/felix/test1 -vv --exclude "A-Library/Movies/Folder.Drives" --delete-empty-src-dirs
2021/12/06 11:34:53 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2021/12/06 11:34:53 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test1" "-vv" "--exclude" "A-Library/Movies/Folder.Drives" "--delete-empty-src-dirs"]
2021/12/06 11:34:53 DEBUG : Creating backend with remote "/home/felix/test"
2021/12/06 11:34:53 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2021/12/06 11:34:53 DEBUG : Creating backend with remote "/home/felix/test1"
2021/12/06 11:34:53 DEBUG : A-Library/Movies/Folder.Drives: Excluded
2021/12/06 11:34:53 DEBUG : Local file system at /home/felix/test1: Waiting for checks to finish
2021/12/06 11:34:53 DEBUG : Local file system at /home/felix/test1: Waiting for transfers to finish
2021/12/06 11:34:53 INFO : two: Removing directory
2021/12/06 11:34:53 INFO : one: Removing directory
2021/12/06 11:34:53 INFO : A-Library/Movies: Removing directory
2021/12/06 11:34:53 DEBUG : A-Library/Movies: Failed to Rmdir: remove /home/felix/test/A-Library/Movies: directory not empty
2021/12/06 11:34:53 INFO : A-Library: Removing directory
2021/12/06 11:34:53 DEBUG : A-Library: Failed to Rmdir: remove /home/felix/test/A-Library: directory not empty
2021/12/06 11:34:53 DEBUG : Local file system at /home/felix/test: failed to delete 2 directories
2021/12/06 11:34:53 DEBUG : Local file system at /home/felix/test: deleted 2 directories
2021/12/06 11:34:53 INFO : There was nothing to transfer
2021/12/06 11:34:53 INFO :
Transferred: 0 B / 0 B, -, 0 B/s, ETA -
Deleted: 0 (files), 4 (dirs)
Elapsed time: 0.0s
2021/12/06 11:34:53 DEBUG : 2 go routines active
felix@gemini:~$ cd test
felix@gemini:~/test$ ls
A-Library
That's an example of not using that flag.
It generates an error, which for me isn't ideal.
The problems lies with you are excluding it so rclone based on what your ask is says the directory is now empty and tries to remove it.
That'll be the same with any exclusion you do.
I'm a bit confused why you are using that flag at all though.
Do you just have empty directories on the source that are creating an issue?
correct. I never bothered to clean the empty folders, and nor googledrive via web or onecommader will load them as its just to many files and freezes / sorting is an issue with all the empty directories as they are there for just a temp/cache. + i need to move them to #temp to rename the folders for sonarr/radarr to scan them in. then move them to the directory either:
A- anime
K- korean/other languages
S- normal language/netflix etc.
they are on a share drive because thats the only way i can clone them then moved off and re crypted to that same share drive and moved off to the main directory. i can just run the --delete-empty-src-dirs on the crypt portion as those are auto defaulted back with no merge conflicts. the Cache remote is the issue as i need to keep the folder ids the same.
It is a share drive owned by me, so I don't have the feature of just wiping it with empty trash either (unlike My Drive) if I could force it to remove everything except the root folders listed.
exactly. the reason behind clearing those empty directories is because google drive web can't walk them all.
I can make the mount walk them all but I don't mount the drives to the vps. only on Windows for playback etc and eventually Linux by using: start rclone --no-console mount --file-perms 0777 GMDmedia:\ C:\.Mounts\Media --no-modtime --rc --rc-addr=:3750 --vfs-cache-mode writes --vfs-cache-max-age 5h --vfs-write-back 5s --cache-dir x:\rclone --dir-cache-time 5m
rclone rc vfs/refresh recursive=true --rc-addr=:7530 --timeout 15m -q
Not sure if i need _async=true though to make it walk the tree and prime the cache or if without, will still work.
--delete-empty-src-dirs probably shouldn't be trying to delete non-empty folders (or raising an error when they are non-empty) in the first place.
It seems like manually deleting subfolders (with rmdirs) is your best option. I don't see how this functionality could be easily added to --delete-empty-src-dirs. Would rclone need a second layer of excludes? That seems cumbersome.
