Concatenate (cat) files with non-random order?

Feature
1

I have to unarchive some large archive, so I want to do something like:

rclone cat --files-from \
    <(echo path1/to/archive.tar.gz.aa; 
      echo path2/to/archive.tar.gz.ab; ... ) \
    remote: | \
    /some/decrypt/cmd --with-key /path/to/key | \
    tee >(time md5sum) >(tar -zxv) >/dev/null

… but in fact the order of cat seems somewhat random. Are there any ways to have rclone respect the order of files-from argument?

I’m using rclone v1.42.

2

Hmm, I suspect rclone is doing them in parallel.

Try --checkers 1 I think that will turn off the parallelism and get you an in order list.

3

Thanks for your reply. I tried:

rclone cat --checkers 1 --files-from <(echo path1/to/A; echo path2/to/B) remote:path

and

rclone cat --checkers 1 --files-from <(echo path2/to/B; echo path1/to/A) remote:path

but I got the same results.

( I have managed to unarchive the large archive ( unarchivecmd < <(rclone cat ...; rclone cat ...; rclone cat ...)), so please never mind about that. I still believe that ordered cat would be helpful though. )

4

I’ve just tried to validate your results, and I haven’t been able to - I find cat always outputs things in order.

Can you show me how to reproduce your result? Preferably with some CLI commands I can cut and paste!

Thanks

5

Sure. I’m using Google Drive (Team Drive), so can I invite you to our testing (sandbox) drive?

$ rclone --version
rclone v1.42-005-g56e1e820β
- os/arch: linux/amd64
- go version: go1.10.1

$ rclone cat --checkers 1 --files-from <(echo A.txt; echo B.txt) fd-public:
A
B

$ rclone cat --checkers 1 --files-from <(echo B.txt; echo A.txt) fd-public:
A
B

$ rclone cat --checkers 1 --files-from <(echo B.txt) fd-public:
B
6

Thanks for that, I see where I’ve got confused. Rclone will always output things in order, in alphabetical order, not the order you put things in the --files-from file.

7

Oh I see. Thanks again for your reply, and sorry for my bad explanation.
Now I understand that I can use alphabetically ordered paths when I use --checkers 1. It’s non-random, which is maybe not the best, but great.

1 Like