Issues Working with --file-names

What is the problem you are having with rclone?

Trying to figure how to use the --file-names parameter. I have spent many hours searching through web articles and the Global Flags page for examples. When I read them, I understand what they are telling me but I have yet to get this parameter to work for me.

The problem is nothing gets copied. I've tried the files individually and they copy. I've tried different lengths of sub-directories on the command-line.

What is your rclone version (output from rclone version)

Version: v1.51.0 (based on output above)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Running rclone.exe from Win 10 Server

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

[2019]
type = swift
env_auth = false
user = 2019
key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
auth = https://xxxxxxx.xxxxxx.com/auth/v1.0
user_id = 
domain = 
tenant = 
tenant_id = 
tenant_domain = 
region = 
storage_url = 
auth_token = 
auth_version = 
endpoint_type = 

A log from the command with the -vv flag

Below is the output I get (I can run the copy with an individual file name in the --file-names list and it will work but for some reason I get the below when I try to use the --file-names list)

C:\rclone>rclone.exe copy 2019:2019\VIOL_IMAGES\LONG_TERM\ D:\SWIFTSTACK\OPENTXN_T00018 --config rclone.conf --files-from C:\rclone\BATCH_FILES\OPENTXN_2019_T00018.TXT --transfers=100 --log-level DEBUG
2020/06/12 06:59:44 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone.exe" "copy" "2019:2019\\VIOL_IMAGES\\LONG_TERM\\" "D:\\SWIFTSTACK\\OPENTXN_T00018" "--config" "rclone.conf" "--files-from" "C:\\rclone\\BATCH_FILES\\OPENTXN_2019_T00018.TXT" "--transfers=100" "--log-level" "DEBUG"]
2020/06/12 06:59:44 DEBUG : Using config file from "C:\\rclone\\rclone.conf"
2020/06/12 07:00:23 DEBUG : 201810: Excluded
2020/06/12 07:00:23 DEBUG : 201901: Excluded
2020/06/12 07:00:23 DEBUG : 201902: Excluded
2020/06/12 07:00:23 DEBUG : 201903: Excluded
2020/06/12 07:00:23 DEBUG : 201904: Excluded
2020/06/12 07:00:23 DEBUG : 201905: Excluded
2020/06/12 07:00:23 DEBUG : 201906: Excluded
2020/06/12 07:00:23 DEBUG : 201907: Excluded
2020/06/12 07:00:23 DEBUG : 201908: Excluded
2020/06/12 07:00:23 DEBUG : 201909: Excluded
2020/06/12 07:00:23 DEBUG : 201910: Excluded
2020/06/12 07:00:23 DEBUG : 201911: Excluded
2020/06/12 07:00:23 DEBUG : 201912: Excluded
2020/06/12 07:00:23 DEBUG : 202001: Excluded
2020/06/12 07:00:23 INFO  : Local file system at //?/D:/SWIFTSTACK/OPENTXN_T00018: Waiting for checks to finish
2020/06/12 07:00:23 INFO  : Local file system at //?/D:/SWIFTSTACK/OPENTXN_T00018: Waiting for transfers to finish
2020/06/12 07:00:23 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Elapsed time:         0.0s

2020/06/12 07:00:23 DEBUG : 5 go routines active
2020/06/12 07:00:23 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rclone.exe" "copy" "2019:2019\\VIOL_IMAGES\\LONG_TERM\\" "D:\\SWIFTSTACK\\OPENTXN_T00018" "--config" "rclone.conf" "--files-from" "C:\\rclone\\BATCH_FILES\\OPENTXN_2019_T00018.TXT" "--transfers=100" "--log-level" "DEBUG"]

What's in your file from file?

What does the source look like you are trying to include?

Here’s a sample of what’s contained in the file:

\201901\20190105\030707_\030707_201901052331400082614_0.jpg
\201901\20190105\030707_\030707_201901052331400082614_1.jpg
\201901\20190105\030707_\030707_201901052331400082614_A.jpg
\201901\20190105\030707_\030707_201901052331400082614_B.jpg
\201901\20190105\030707_\030707_201901052331400082614_E.jpg
\201901\20190105\030707_\030707_201901052331400082614_F.jpg
\201901\20190105\030707_\030707_201901052331400082614_G.jpg
\201901\20190105\030707_\030707_201901052331400082614_H.jpg

If I remove the --files-from parameter and related file, and put the full file name in the source position, the copy works.

Files-from is a bit confusing, if you are passing:

It's relative from that path.

So I can use:

felix@gemini:~$ cat from
dupe.mkv
felix@gemini:~$ rclone ls GD:test --files-from from
      413 dupe.mkv
felix@gemini:~$ rclone ls GD: --files-from from
felix@gemini:~$

or

felix@gemini:~$ cat from
test/dupe.mkv
felix@gemini:~$ rclone ls GD:test --files-from from
felix@gemini:~$ rclone ls GD: --files-from from
      413 test/dupe.mkv

the complete path to the files are:

2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190105\030707_\030707_201901052331400082614_0.jpg

So if you consider "2019:2019\VIOL_IMAGES\LONG_TERM" the "root" directory then all the files are below it.

Based on this example:
To take a more complicated example, let's say you had a few files you want to back up regularly with these absolute paths:

/home/user1/important
/home/user1/dir/file
/home/user2/stuff

To copy these you'd find a common subdirectory - in this case /home and put the remaining files in files-from.txt with or without leading / , eg

user1/important
user1/dir/file
user2/stuff

You could then copy these to a remote like this

rclone copy --files-from files-from.txt /home remote:backup

Yes, that should work as you described.

I would use a filter-from if I needed more regex type workloads to make things easier, but files-from would accomplish it like that list you have.

You can always test very easy with just running a rclone ls on the source of the command with files-from.

Hmmm... that's the problem. It "should" work but doesn't. I don't understand why rclone tells me

2020/06/12 07:00:23 DEBUG : 201810: Excluded
.
.
.
2020/06/12 07:00:23 DEBUG : 201912: Excluded
2020/06/12 07:00:23 DEBUG : 202001: Excluded

when no parameters are telling it to "Exclude"

I'll try the rclone ls command and see if it tells me something different. The challenge I have is that I need specific files from the directories and not all which is why I'm trying to use the --files-from option.

You'd have to share:

rclone ls command without filter
rclone ls command with the filter and the contents of the filter-from file

Best to test with a single file if that's easier.

I've not used the ls command. But this is what i ran for a single file:
rclone.exe --config rclone.conf ls 2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg --log-level DEBUG --max-depth 1

The output seems to traverse the lowest directory and gives me the following (it fills up so much that I cannot scroll back to get it all):

2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_0.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_0.sig: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_1.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_A.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_A.sig: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_B.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_E.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_F.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_H.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_I.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_S.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812190037096_T.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100.vdf: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_0.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_0.sig: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_1.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_A.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_A.sig: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_B.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_C.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_D.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_G.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_H.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_I.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_S.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812200037100_T.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 030707_201901081812270037108.vdf: Excluded from sync (and deletion)
.
.
.
2020/06/15 10:56:33 DEBUG : 030707_201901082348410040056_T.jpg: Excluded from sync (and deletion)
2020/06/15 10:56:33 DEBUG : 4 go routines active
2020/06/15 10:56:33 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone.exe" "--config" "rclone.conf" "ls" "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg" "--log-level" "DEBUG" "--max-depth" "1"]

When I run the "copy" command for the same file I get this:

C:\Swiftstack>rclone.exe --config rclone.conf copy 2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg R:\SWIFTSTACK\TEST --log-level DEBUG
2020/06/15 10:57:58 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone.exe" "--config" "rclone.conf" "copy" "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg" "R:\SWIFTSTACK\TEST" "--log-level" "DEBUG"]
2020/06/15 10:57:58 DEBUG : Using config file from "C:\Swiftstack\rclone.conf"
2020/06/15 10:57:58 DEBUG : 030707_201901081651090033258_0.jpg: Couldn't find file - need to transfer
2020/06/15 10:57:58 INFO : 030707_201901081651090033258_0.jpg: Copied (new)
2020/06/15 10:57:58 INFO :
Transferred: 2.791 kBytes (9.957 kBytes/s)
Errors: 0
Checks: 0
Transferred: 1
Elapsed time: 200ms

2020/06/15 10:57:58 DEBUG : 4 go routines active
2020/06/15 10:57:58 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone.exe" "--config" "rclone.conf" "copy" "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg" "R:\SWIFTSTACK\TEST" "--log-level" "DEBUG"]

didn't run the "ls" from file because something seems to be off with the first try with a single file from the command line.

NOTE: These were run from a different server with V1.42 - but I've tried running this on both servers with the same results.

I reran this as:
C:\Swiftstack>rclone.exe --config rclone.conf ls 2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_ --log-level DEBUG --max-depth 1 > ls.output.txt
2020/06/15 11:12:48 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone.exe" "--config" "rclone.conf" "ls" "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_" "--log-level" "DEBUG" "--max-depth" "1"]
2020/06/15 11:12:48 DEBUG : Using config file from "C:\Swiftstack\rclone.conf"
2020/06/15 11:15:31 DEBUG : 4 go routines active
2020/06/15 11:15:31 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone.exe" "--config" "rclone.conf" "ls" "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_" "--log-level" "DEBUG" "--max-depth" "1"]

Looked in the ls.output.txt file and did find the filename that I was looking for 030707_201901081651090033258_0.jpg

reran the "ls" command with the full filename and path with the --log-file parm. Output was the same as above and the file name was NOT listed in the log file.

Is that expected behavior?

is years and years old so wouldn't help to compare.

I honestly am having a hard time following as you are jumping all around so I can't make sense of your paths as some commands have output, some have files, some have debug.

Take 1 file, rclone ls that one file.
Use a files-name and validate what you want to happen.
If not happening, post the info and I can recreate what you are doing and see what the issue is.

Sorry, been working on this for so long it's all blurring for me too.

  1. I ran the "ls" command - for specific file 030707_201901081651090033258_0.jpg
    2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_\030707_201901081651090033258_0.jpg --log-level DEBUG --max-depth 1

It dumped 72K+ lines - basically every other image in the path "2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_" and the file i was looking for was not include: 030707_201901081651090033258_0.jpg

Log Listing had a line like this for every other entry in that path:
2020/06/15 11:20:04 DEBUG : 030707_201901080000540017333_A.jpg: Excluded from sync (and deletion)

  1. I ran the "ls" command - at bottom path level
    2019:2019\VIOL_IMAGES\LONG_TERM\201901\20190108\030707_ --log-level DEBUG --max-depth 1

This dumped all file names in that path to the log file, and the file I'm looking for was listed log file. Again dumped 72K+ lines into the log file.

  1. I ran the "ls" command: using --file-from parameter
    rclone.exe --config rclone.conf ls 2019:2019\VIOL_IMAGES\LONG_TERM --files-from ls-test.txt --log-level DEBUG --max-depth 1 --log-file ls.dump.ff.txt

ls-test.txt file contained:
\201901\20190108\030707_\030707_201901081651090033258_0.jpg

Results:
2020/06/15 11:52:51 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone.exe" "--config" "rclone.conf" "ls" "2019:2019\VIOL_IMAGES\LONG_TERM" "--files-from" "ls-test.txt" "--log-level" "DEBUG" "--max-depth" "1" "--log-file" "ls.dump.ff.txt"]
2020/06/15 11:52:51 DEBUG : Using config file from "C:\Swiftstack\rclone.conf"
2020/06/15 11:53:34 DEBUG : 201810: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201901: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201902: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201903: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201904: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201905: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201906: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201907: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201908: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201909: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201910: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201911: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 201912: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 202001: Excluded from sync (and deletion)
2020/06/15 11:53:34 DEBUG : 4 go routines active
2020/06/15 11:53:34 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone.exe" "--config" "rclone.conf" "ls" "2019:2019\VIOL_IMAGES\LONG_TERM" "--files-from" "ls-test.txt" "--log-level" "DEBUG" "--max-depth" "1" "--log-file" "ls.dump.ff.txt"]

Basically the same results i get with the copy command.

missed this... i'll rerun the tests on the other server if required. It will take about a half hour to move test scripts to that server.

1.42 is many years old and different from anything I'm using so trying to compare the two isn't a good use of our time. If you can upgrade to current, that would be great.

Yes moving the scripts over to the current server with the 1.51

I have rerun every thing under v1.51 and all the results are the same.

Can you just perform the test with a single file, share the contents of the files from file.

I still am not sure the contents and what you are running.