--include-from with direct file paths to copy

tb582 · March 19, 2021, 12:25am

I have a list of file paths sitting in a txt file - I thought I could just use a command like copy and feed the file to rclone to go copy them...

Example include-from file:

My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-51-41.jpeg
My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg

Command
rclone copy remote: --dry-run --no-traverse --include-from "P:\scripts\includeFile.txt"

Error
Command copy needs 2 arguments minimum: you provided 1 non flag arguments: ["remote:"]

not sure if I'm using the --include-from in the proper way given I'm giving it exact file paths to copy ?

asdffdsa · March 19, 2021, 12:32am

hi,

you need a source and destination.
where are you copying the files to?

tb582 · March 19, 2021, 1:39am

Desire is to copy them to the same file paths as they are in ... similar to sync so I guess for dest I just specify my root ie C:/myfolder ?

asdffdsa · March 19, 2021, 1:48am

if that is desired dest then use that

tb582 · March 19, 2021, 2:53pm

Now I'm running into this with my updated command

Error: Failed to create file system for "dest:C:\\Users\\Tony\\Pictures": didn't find section in config file

command:
rclone copy remote: dest:C:\Users\Tony\Pictures --dry-run --no-traverse --include-from "P:\scripts\includeFile.txt"

asdffdsa · March 19, 2021, 2:58pm

try
rclone copy remote: C:\Users\Tony\Pictures --dry-run --no-traverse --include-from "P:\scripts\includeFile.txt"

tb582 · March 19, 2021, 3:09pm

hmmm why does rclone need to scan through ALL the files if I'm giving it the exact remote file paths to copy ?

also don't know why it didn't find them ....

2021/03/19 11:02:55 INFO : There was nothing to transfer
2021/03/19 11:02:55 NOTICE:
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 39.2s

asdffdsa · March 19, 2021, 3:17pm

run the command with debug output, add -vv and post it.
post the include file
post rclone lsd remote:

tb582 · March 19, 2021, 4:25pm

It must be some syntax thing... my include from file:
My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg

debug log:

2021/03/19 11:32:31 DEBUG : --max-age 1.5549193842171296M to 2021-01-31 19:00:00.0009973 -0500 EST m=-4030350.992031199
2021/03/19 11:32:31 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\\Program Files (x86)\\rclone\\rclone.exe" "copy" "remote:" "C:\\Users\\Tony\\Pictures" "--dry-run" "--no-traverse" "--max-age" "2021-02-01" "--include-from" "P:\\scripts\\includeFile.txt" "--log-file=P:\\scripts\\rcloneDebug.txt" "--log-level" "DEBUG"]
2021/03/19 11:32:31 DEBUG : Using config file from "C:\\Users\\Tony\\.config\\rclone\\rclone.conf"
2021/03/19 11:32:31 DEBUG : Creating backend with remote "remote:"
2021/03/19 11:32:31 DEBUG : Creating backend with remote "C:\\Users\\Tony\\Pictures"
2021/03/19 11:32:31 DEBUG : fs cache: renaming cache item "C:\\Users\\Tony\\Pictures" to be canonical "//?/C:/Users/Tony/Pictures"
2021/03/19 11:32:45 DEBUG : My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg: Excluded
2021/03/19 11:33:00 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for checks to finish
2021/03/19 11:33:00 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for transfers to finish
2021/03/19 11:33:00 INFO  : There was nothing to transfer
2021/03/19 11:33:00 NOTICE: 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:        29.6s

2021/03/19 11:33:00 DEBUG : 3 go routines active

LSD command:

 rclone lsd remote:
          -1 2021-02-02 20:14:11        -1 My Music
          -1 2020-11-25 14:30:43        -1 My Pictures

asdffdsa · March 19, 2021, 4:29pm

i see that you added --max-age perhaps that is the reason?

tb582 · March 19, 2021, 4:32pm

Nope - that file is less than the max age but even if I remove max age I still get

2021/03/19 12:30:51 INFO : There was nothing to transfer

asdffdsa · March 19, 2021, 4:32pm

post the debug log without --max-age

tb582 · March 19, 2021, 4:35pm

2021/03/19 12:30:21 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\Program Files (x86)\rclone\rclone.exe" "copy" "remote:" "C:\Users\Tony\Pictures" "--dry-run" "--no-traverse" "--include-from" "P:\scripts\includeFile.txt" "--log-file=P:\scripts\rcloneDebug.txt" "--log-level" "DEBUG"]
2021/03/19 12:30:21 DEBUG : Using config file from "C:\Users\Tony\.config\rclone\rclone.conf"
2021/03/19 12:30:21 DEBUG : Creating backend with remote "remote:"
2021/03/19 12:30:21 DEBUG : Creating backend with remote "C:\Users\Tony\Pictures"
2021/03/19 12:30:21 DEBUG : fs cache: renaming cache item "C:\Users\Tony\Pictures" to be canonical "//?/C:/Users/Tony/Pictures"
2021/03/19 12:30:36 DEBUG : My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg: Excluded
2021/03/19 12:30:51 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for checks to finish
2021/03/19 12:30:51 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for transfers to finish
2021/03/19 12:30:51 INFO : There was nothing to transfer
2021/03/19 12:30:51 NOTICE:
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 29.7s

asdffdsa · March 19, 2021, 4:36pm

not sure,
instread of include-from, try this
https://rclone.org/filtering/#files-from-read-list-of-source-file-names

tb582 · March 19, 2021, 4:56pm

hmm I had hoped that would work but it still excludes the My Pictures folder....

2021/03/19 12:54:03 DEBUG : rclone: Version "v1.54.0" starting with parameters ["C:\Program Files (x86)\rclone\rclone.exe" "copy" "remote:" "C:\Users\Tony\Pictures" "--dry-run" "--files-from" "P:\scripts\includeFile.txt" "--log-file=P:\scripts\rcloneDebug.txt" "--log-level" "DEBUG"]
2021/03/19 12:54:03 DEBUG : Using config file from "C:\Users\Tony\.config\rclone\rclone.conf"
2021/03/19 12:54:03 DEBUG : Creating backend with remote "remote:"
2021/03/19 12:54:03 DEBUG : Creating backend with remote "C:\Users\Tony\Pictures"
2021/03/19 12:54:03 DEBUG : fs cache: renaming cache item "C:\Users\Tony\Pictures" to be canonical "//?/C:/Users/Tony/Pictures"
2021/03/19 12:54:03 DEBUG : My Pictures: Excluded
2021/03/19 12:54:03 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for checks to finish
2021/03/19 12:54:03 DEBUG : Local file system at //?/C:/Users/Tony/Pictures: Waiting for transfers to finish
2021/03/19 12:54:03 INFO : There was nothing to transfer
2021/03/19 12:54:03 NOTICE:
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time: 0.4s

asdffdsa · March 19, 2021, 5:06pm

post that .txt file

tb582 · March 19, 2021, 5:09pm

includeFile.txt (136 Bytes)

asdffdsa · March 19, 2021, 5:39pm

not sure, as it seems to work for me.
maybe a case-sensitive problem?

rclone.exe ls remote: 
        2 My Pictures/02.txt
        2 My Pictures/Kaelyn/03.txt
        2 My Pictures/Kaelyn/Automatic Upload/04.txt
        0 My Pictures/Kaelyn/Automatic Upload/iPhone/01.txt
        2 My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg

rclone.exe ls remote: --files-from=filesfrom.txt 
        2 My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg

and this is the filesfrom.txt

My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-52-02.jpeg

tb582 · March 19, 2021, 7:26pm

even using --ignore-case seems to exclude everything

tb582 · March 19, 2021, 7:44pm

Ok I was able to get this to work manually:

rclone copy remote:"My Pictures/Kaelyn/Automatic Upload/iPhone/2021-02-05 15-51-41.jpeg" "C:\Users\Tony\Pictures" --dry-run -vv

Working my way backwards - maybe a quoting thing in the from file.