Movie from files excluding everything

Help and Support
1

Attempting to move files which I have in a list in a txt file but for some reason everything is being excluded - it doesn't look like rclone is traversing the directories. I'm sure I'm missing something basic, guide me!

2022/11/28 19:54:07 DEBUG : Creating backend with remote "remote:"
2022/11/28 19:54:07 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2022/11/28 19:54:07 DEBUG : 5 Meyers ct: Excluded
2022/11/28 19:54:07 DEBUG : Crypto Folder: Excluded
2022/11/28 19:54:07 DEBUG : Mom: Excluded
2022/11/28 19:54:07 DEBUG : My Music: Excluded
2022/11/28 19:54:07 DEBUG : My Pictures: Excluded
2022/11/28 19:54:07 DEBUG : My Videos: Excluded
2022/11/28 19:54:07 DEBUG : pCloud Save: Excluded
2022/11/28 19:54:07 DEBUG : Personal: Excluded
2022/11/28 19:54:07 DEBUG : scripts: Excluded
2022/11/28 19:54:07 DEBUG : System Volume Information: Excluded
2022/11/28 19:54:07 DEBUG : Windows_Backups: Excluded
2022/11/28 19:54:07 DEBUG : .apdisk: Excluded
2022/11/28 19:54:07 DEBUG : .DS_Store: Excluded
2022/11/28 19:54:07 DEBUG : 1Password Emergency Kit A3-BZC9FD-my.pdf: Excluded
2022/11/28 19:54:07 DEBUG : scripts␉est.csv: Excluded
2022/11/28 19:54:07 DEBUG : 3 go routines active

Sample from my MPC.txt file

2022-03-02 08-03-23.jpeg
2022-03-02 08-13-20.png
2022-03-04 07-33-01.mov

these files are somewhere buried in the directories from root...

2

What rclone command are you using? If you are using --files-from then it needs to have the paths from the root in.

You could use --include-from and I think that will work without paths from the root.

3

oh sorry IDK why my copy left out the most important debug line!

2022/11/29 06:41:31 DEBUG : rclone: Version "v1.61.0-beta.6555.0c56c4652" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "move" "remote:" "remote:/MPC" "--files-from" "P:\\scripts\\mpc.txt" "--dry-run" "-vv"]

thats the command - I swear that --files-from traverses the remote though?

Rclone commands with a --files-from flag traverse the remote, treating the names in --files-from as a set of filters.

let me try --include-from and report back

4

Looks like I can't move without

2022/11/29 06:47:19 Failed to move: can't sync or move files on overlapping remotes (try excluding the destination with a filter rule)

so I guess I need to copy and then delete ? :frowning:

5

might be related to Overlapping moves too conservative? · Issue #1082 · rclone/rclone · GitHub ??

6

It's really tough to tell as you haven't ever listed a command you are running and what you expect to happen.

It might be related, but it might not.

You really need to include what you are running with all the flags and some output to look at.

7

I did here:

I just changed it from --files-from TO --include-from

8

Did you try adding the exclude as well?

It's telling you that your move command overlaps, which it does.

Why are you using a beta version?

Is that the whole file? Just file names over and over?

9

I don't think I need exclude?? the docs say exclude ** is implied - and yes its just a list of filenames

10

You said you've switched to files from and the error message says you should try using exclude.

I get:

felix@gemini:~/test$ rclone move /home/felix/test /home/felix/test/blah -vv
2022/11/29 16:20:09 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 16:20:09 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "-vv"]
2022/11/29 16:20:09 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 16:20:09 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 16:20:09 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 16:20:09 DEBUG : Local file system at /home/felix/test/blah: Using server-side directory move
2022/11/29 16:20:09 INFO  : Local file system at /home/felix/test/blah: Server side directory move failed - fallback to file moves: can't copy directory - destination already exists
2022/11/29 16:20:09 ERROR : Fatal error received - not attempting retries
2022/11/29 16:20:09 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)
Elapsed time:         0.0s

2022/11/29 16:20:09 DEBUG : 2 go routines active
2022/11/29 16:20:09 Failed to move: can't sync or move files on overlapping remotes (try excluding the destination with a filter rule)

and I add in an exclude

felix@gemini:~/test$ rclone move /home/felix/test /home/felix/test/blah -vv --exclude blah/**
2022/11/29 16:26:30 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 16:26:30 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "-vv" "--exclude" "blah/**"]
2022/11/29 16:26:30 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 16:26:30 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 16:26:30 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 16:26:30 DEBUG : blah: Excluded
2022/11/29 16:26:30 DEBUG : Local file system at /home/felix/test/blah: Waiting for checks to finish
2022/11/29 16:26:30 DEBUG : Local file system at /home/felix/test/blah: Waiting for transfers to finish
2022/11/29 16:26:30 INFO  : test: Moved (server-side)
2022/11/29 16:26:30 INFO  : There was nothing to transfer
2022/11/29 16:26:30 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         0.0s

2022/11/29 16:26:30 DEBUG : 4 go routines active

and

files from works for me without an exclude

felix@gemini:~/test/blah$ rclone move /home/felix/test /home/felix/test/blah --files-from /home/felix/filesfrom -vv
2022/11/29 16:28:57 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 16:28:57 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "--files-from" "/home/felix/filesfrom" "-vv"]
2022/11/29 16:28:57 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 16:28:57 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 16:28:57 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 16:28:57 DEBUG : blah: Excluded
2022/11/29 16:28:57 DEBUG : Local file system at /home/felix/test/blah: Waiting for checks to finish
2022/11/29 16:28:57 DEBUG : Local file system at /home/felix/test/blah: Waiting for transfers to finish
2022/11/29 16:28:57 INFO  : test: Moved (server-side)
2022/11/29 16:28:57 INFO  : There was nothing to transfer
2022/11/29 16:28:57 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         0.0s

2022/11/29 16:28:57 DEBUG : 2 go routines active

but if you have if buried somewhere and just have a file name

felix@gemini:~/test$ rclone move /home/felix/test /home/felix/test/blah --files-from /home/felix/filesfrom -vv
2022/11/29 16:29:52 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 16:29:52 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "--files-from" "/home/felix/filesfrom" "-vv"]
2022/11/29 16:29:52 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 16:29:52 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 16:29:52 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 16:29:52 DEBUG : onedeep: Excluded
2022/11/29 16:29:52 DEBUG : blah: Excluded
2022/11/29 16:29:52 DEBUG : Local file system at /home/felix/test/blah: Waiting for checks to finish
2022/11/29 16:29:52 DEBUG : Local file system at /home/felix/test/blah: Waiting for transfers to finish
2022/11/29 16:29:52 INFO  : There was nothing to transfer
2022/11/29 16:29:52 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         0.0s

2022/11/29 16:29:52 DEBUG : 3 go routines active

and with a files from:

felix@gemini:~/test$ ls -alR
.:
total 16
drwxrwxr-x 4 felix felix 4096 Nov 29 16:29 .
drwxr-x--- 9 felix felix 4096 Nov 29 16:32 ..
drwxrwxr-x 2 felix felix 4096 Nov 29 16:33 blah
drwxrwxr-x 2 felix felix 4096 Nov 29 16:33 onedeep

./blah:
total 8
drwxrwxr-x 2 felix felix 4096 Nov 29 16:33 .
drwxrwxr-x 4 felix felix 4096 Nov 29 16:29 ..

./onedeep:
total 8
drwxrwxr-x 2 felix felix 4096 Nov 29 16:33 .
drwxrwxr-x 4 felix felix 4096 Nov 29 16:29 ..
-rw-rw-r-- 1 felix felix    0 Nov 29 16:19 test
felix@gemini:~/test$ rclone move /home/felix/test /home/felix/test/blah --files-from /home/felix/filesfrom -vv
2022/11/29 16:33:28 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 16:33:28 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "--files-from" "/home/felix/filesfrom" "-vv"]
2022/11/29 16:33:28 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 16:33:28 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 16:33:28 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 16:33:28 DEBUG : blah: Excluded
2022/11/29 16:33:28 INFO  : onedeep/test: Moved (server-side)
2022/11/29 16:33:28 DEBUG : Local file system at /home/felix/test/blah: Waiting for checks to finish
2022/11/29 16:33:28 DEBUG : Local file system at /home/felix/test/blah: Waiting for transfers to finish
2022/11/29 16:33:28 INFO  : There was nothing to transfer
2022/11/29 16:33:28 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         0.0s

2022/11/29 16:33:28 DEBUG : 3 go routines active
felix@gemini:~/test$ cat ../filesfrom
onedeep/test

The more information you can share, the easier is for me to replicate/test/assist as I have more information to go on and I don't have to guess things.

11

this is the exact command I'm trying to use

2022/11/29 16:44:43 DEBUG : rclone: Version "v1.61.0-beta.6555.0c56c4652" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "move" "remote:" "remote:MPC" "--include-from" "P:\\scripts\\mpc.txt" "--dry-run" "-vv"]
2022/11/29 16:44:43 DEBUG : Creating backend with remote "remote:"
2022/11/29 16:44:43 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2022/11/29 16:44:43 DEBUG : Creating backend with remote "remote:MPC"
2022/11/29 16:44:44 ERROR : Fatal error received - not attempting retries
2022/11/29 16:44:44 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)
Elapsed time:         0.9s

2022/11/29 16:44:44 DEBUG : 4 go routines active
2022/11/29 16:44:44 Failed to move: can't sync or move files on overlapping remotes (try excluding the destination with a filter rule)
12

So if I understand your goal.

You have files that may or may not be in your top level and you want to move them.

I use blah as my directory to exclude with a filter and a testfile to match something to move it.

felix@gemini:~$ cat filter
- blah/**
+ **/testfile
- **
felix@gemini:~$ rclone move /home/felix/test /home/felix/test/blah -vv --filter-from /home/felix/filter
2022/11/29 17:09:20 DEBUG : Setting --config "/opt/rclone/rclone.conf" from environment variable RCLONE_CONFIG="/opt/rclone/rclone.conf"
2022/11/29 17:09:20 DEBUG : rclone: Version "v1.60.1" starting with parameters ["rclone" "move" "/home/felix/test" "/home/felix/test/blah" "-vv" "--filter-from" "/home/felix/filter"]
2022/11/29 17:09:20 DEBUG : Creating backend with remote "/home/felix/test"
2022/11/29 17:09:20 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2022/11/29 17:09:20 DEBUG : Creating backend with remote "/home/felix/test/blah"
2022/11/29 17:09:20 DEBUG : blah: Excluded
2022/11/29 17:09:20 DEBUG : Local file system at /home/felix/test/blah: Waiting for checks to finish
2022/11/29 17:09:20 DEBUG : Local file system at /home/felix/test/blah: Waiting for transfers to finish
2022/11/29 17:09:20 INFO  : onedeep/testfile: Moved (server-side)
2022/11/29 17:09:20 INFO  : There was nothing to transfer
2022/11/29 17:09:20 INFO  :
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Checks:                 1 / 1, 100%
Renamed:                1
Elapsed time:         0.0s

2022/11/29 17:09:20 DEBUG : 2 go routines active
felix@gemini:~$

I'm like 99.9% that would also exclude the file at the root but not sure if that's a problem or for you.

13

but even yours says There was nothing to transfer - meaning it didn't find the files in your --filter-from anywhere in your remote ??

14

I think you misread it.

and

It moved it to the new location server side.

15

ah man - so going back to my --filter-from example why is all my stuff excluded?

PS C:\Users\Tony> rclone move remote: remote:MCP --filter-from P:\scripts\mpc.txt --dry-run -vv
2022/11/30 16:59:35 DEBUG : rclone: Version "v1.61.0-beta.6555.0c56c4652" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "move" "remote:" "remote:MCP" "--filter-from" "P:\\scripts\\mpc.txt" "--dry-run" "-vv"]
2022/11/30 16:59:35 DEBUG : Creating backend with remote "remote:"
2022/11/30 16:59:35 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2022/11/30 16:59:35 DEBUG : Creating backend with remote "remote:MCP"
2022/11/30 16:59:36 ERROR : Fatal error received - not attempting retries
2022/11/30 16:59:36 NOTICE:
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Errors:                 1 (fatal error encountered)
Elapsed time:         0.6s

2022/11/30 16:59:36 DEBUG : 4 go routines active
2022/11/30 16:59:36 Failed to move: can't sync or move files on overlapping remotes (try excluding the destination with a filter rule)

my filter file:

+ **/iPhone-2019.01.01-09.31.41.675.jpg
+ **/iPhone-2019.01.01-11.22.26.816.jpg
+ **/iPhone-2019.01.01-18.37.12.420.jpg
+ **/iPhone-2019.01.01-18.37.18.711.jpg
+ **/iPhone-2019.01.01-19.36.54.000.png
+ **/iPhone-2019.01.02-07.55.00.922.jpg
+ **/iPhone-2019.01.03-19.11.15.000.png
+ **/iPhone-2019.01.04-18.08.47.721.jpg
+ **/iPhone-2019.01.04-18.09.16.542.jpg
+ **/iPhone-2019.01.04-18.09.36.731.jpg
+ **/iPhone-2019.01.04-18.09.42.380.jpg
+ **/iPhone-2019.01.04-18.09.52.021.jpg
+ **/iPhone-2019.01.04-18.09.59.818.jpg
+ **/iPhone-2019.01.04-18.10.12.446.jpg
+ **/iPhone-2019.01.04-18.10.18.113.jpg
+ **/iPhone-2019.01.04-20.42.19.305.jpg
+ **/iPhone-2019.01.07-18.08.45.000.png
+ **/iPhone-2019.01.07-18.09.23.000.png
- /**
16

You missed the first part of my filter which excludes the destination.

You need the first line to exclude it.

- MPC/**
17

ahh lord that was way more involved than I expected - thanks for the guidance.

18

one more follow-up --- seems like --dry-run in the debug log shows that Skipped move as --dry-run is set and nothing about the fact that this should of been skipped as it doesn't match whats in the --filter-from ?

2022/12/01 19:51:11 NOTICE: My Videos/Drone/DJI-FPV/DJI_0020.SRT: Skipped move as --dry-run is set (size 3.250Mi)
2022/12/01 19:51:11 NOTICE: My Videos/Drone/DJI-FPV/DJI_0021.MP4: Skipped move as --dry-run is set (size 1.763Gi)
2022/12/01 19:51:11 NOTICE: My Videos/Drone/DJI-FPV/DJI_0021.SRT: Skipped move as --dry-run is set (size 1.625Mi)
2022/12/01 19:51:11 NOTICE: Personal/Emilie/Emilie/ERB 2012 (3).doc: Skipped move as --dry-run is set (size 47Ki)
2022/12/01 19:51:11 NOTICE: Personal/Emilie/Emilie/August 1.doc: Skipped move as --dry-run is set (size 24Ki)

rclone move remote: remote:MPC --filter-from P:\scripts\mpc.txt --dry-run -vv
2022/12/01 19:55:45 DEBUG : rclone: Version "v1.61.0-beta.6555.0c56c4652" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "move" "remote:" "remote:MPC" "--filter-from" "P:\\scripts\\mpc.txt" "--dry-run" "-vv"]
2022/12/01 19:55:45 DEBUG : Creating backend with remote "remote:"
2022/12/01 19:55:45 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2022/12/01 19:55:45 DEBUG : Creating backend with remote "remote:MPC"
2022/12/01 19:55:45 NOTICE: .DS_Store: Skipped move as --dry-run is set (size 14.004Ki)
2022/12/01 19:55:45 NOTICE: scripts␉est.csv: Skipped move as --dry-run is set (size 4.861Mi)

here's the filter file again

- MPC**
+ **/2022-03-02 08-03-23.jpeg
+ **/2022-03-02 08-13-20.png
+ **/2022-03-04 07-33-01.mov
+ **/2022-03-16 17-37-06 (2).mov
+ **/2022-03-16 17-37-06.mov
+ **/2022-04-01 19-12-26 (2).mov
19

I really can't tell from the snippet.

Perhaps try with rclone ls and see if that works as that's easier to see what's going on.

20

Thanks - got it, more --filter-from issues but resolved them now buy adding at the end of the file - **