Rclone with --files-from exits without copying on windows

[elhaddi (aba)]

I run rclone as follows on windows 7:

rclone -v -v -v -v --config C:\testrclone\rclone.conf copy --no-traverse --log-file=C:\testrclone\logs\flist.log --files-from C:\testrclone\tmp\flist.txt \testrt\tawasa remote:bucket1/oct

Here is the content of flist.txt used in --from-files

c:\testrt\tawasa\dir_5876_0\msg_5876_0000000000
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000001
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000002
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000003
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000004
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000005
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000006
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000007
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000008
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000009

Here is the resulting log file:

2019/10/30 13:07:34 DEBUG : rclone: Version "v1.50.0" starting with parameters ["rclone" "-v" "-v" "-v" "-v" "--config" "C:\testrclone\rclone.conf" "copy" "--no-traverse" "--log-file=flist.log" "--files-from" "flist.txt" "\testrt\tawasa" "remote:bucket1/oct"]
2019/10/30 13:07:34 DEBUG : Using config file from "C:\testrclone\rclone.conf"
2019/10/30 13:07:34 INFO : S3 bucket bucket1 path 10: Waiting for checks to finish
2019/10/30 13:07:34 INFO : S3 bucket bucket1 path 10: Waiting for transfers to finish
2019/10/30 13:07:34 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 0s

2019/10/30 13:07:34 DEBUG : 5 go routines active
2019/10/30 13:07:34 DEBUG : rclone: Version "v1.50.0" finishing with parameters ["rclone" "-v" "-v" "-v" "-v" "--config" "C:\testrclone\rclone.conf" "copy" "--no-traverse" "--log-file=flist.log" "--files-from" "flist.txt" "\testrt\tawasa" "WASABI:bucket1/oct"]

PS: I also tried with unix file separator "/" instead of the windows file sep "\"

[thestigma]

Firstly, try running with -vv (to enable debug output)
This should tell you why exactly it is failing (such as maybe not finding the files where you though it would look). When you know why it fail then we don't have to guess what the problem is

Secondly, if you are confused about what is the correct formatting of the file, I suggest you use
rclone lsf MyRemote:
this will create output that is compatible with --files-from
And this can be quite useful to generate them generally, specially when you lsf with filters

[elhaddi (aba)]

Thanks thestigma. Will try that. The same command works fine on Linux (of course with *Nix path names). One windows the --files-from exits with exit 0 without any file checks or copy.

[thestigma]

One thing I immediately note is that rclone seems to internally use forward flashes as path delimiters /
On windows (and your file) you use
AFAIK, rclone translates this for you normally on windows , but it may do that in --files-from

[ncw (Nick Craig-Wood)]

I run rclone as follows on windows 7:

rclone -v -v -v -v --config C:\testrclone\rclone.conf copy --no-traverse --log-file=C:\testrclone\logs\flist.log --files-from C:\testrclone\tmp\flist.txt \testrt\tawasa remote:bucket1/oct

Here is the content of flist.txt used in --from-files

c:\testrt\tawasa\dir_5876_0\msg_5876_0000000000
c:\testrt\tawasa\dir_5876_0\msg_5876_0000000001

Note that the contents of --files-from needs to be relative to the root of the transfer. In this case it is \testrt\tawasa so you want

dir_5876_0/msg_5876_0000000000
dir_5876_0/msg_5876_0000000001

(I think the \ will need to be / also.)

Note that you can use rclone lsf to generate lists compatible with --files-from.

[thestigma]

Since you are on windows too, I can mention that I wrote a sorting mechanism in batch for the rclone lsl format yesterday - in case you would like to be able to control the order that such --files-from transfers are performed in.

currently has size-asc/desc , age-asc/desc
Planning to add a a age-mixed which would optimize bandwidth utilization when copying a mix of large and small files.
I still have a little testing to do, but if that's something you'd want to use in your scripts then just shoot me a PM.

[elhaddi (aba)]

testrt\tawasa

Thanks.

So here is the solution to the original problem:

rclone -vv -vv -vv -P copy --no-traverse --log-file=x1.log --files-from x1.txt / REMOTE:BUCKET/PATH

Where x1.txt is without the root drive letter:

/testrt/tawasa/dir_5876_0/msg_5876_0000000000
/testrt/tawasa/dir_5876_0/msg_5876_0000000001
/testrt/tawasa/dir_5876_0/msg_5876_0000000002
/testrt/tawasa/dir_5876_0/msg_5876_0000000003

Using c:/ as root will also work:

rclone -vv -vv -vv -P copy --no-traverse --log-file=x1.log --files-from x1.txt c:/ REMOTE:BUCKET/PATH

And x1.txt is the same as above. The above will fail if x1.txt has the drive letter like the following snippet:

c:/testrt/tawasa/dir_5876_0/msg_5876_0000000000
c:/testrt/tawasa/dir_5876_0/msg_5876_0000000001
...

However, what if the files in files-from (x1.txt above) are from multiple drives?

example:

c:/dir1/file1
d:/dir1/file2
e:/dir43/file4

I tried x1.txt as (with and without the drive letters), and it will transfer only what is in "C:"
/dir1/file1
/dir2/file2
/dir43/file4

[thestigma]

-vv -vv -vv

Just a quick unrelated note - that (above) is redundant.
All you need is either
-v
(for verbose output)
or
-vv
(for debug output)
or
nothing
(for standard "notice" level output - ie. the default)

adding the same flags several times just overrides the earlier, so it does nothing (but at least it does nothing bad either).

[elhaddi (aba)]

I thought it uses incremental trace levels. Thanks for the clarification.