Append from remote like copy (without cat)?

Hi,
My first post here. Congrats to the developer(s) because you dont find such a useful tool that often.
I'm interested in speeding up downloads of splitted files. Currently I use "rclone copy" for the first chunk and "rclone cat" and an append redirection for the following ones. But "cat" is much slower than copy because it's monothread.
Could it be possible to create a copy variant for this purpose? Something like:

rclone copyappend -P remote:/file.dat.[0-5] /mnt/file.dat

Sorry if this has been discussed before but I couldn't find it.

Manuel F.

How would this compare to --multi-thread-cutoff and --transfers. See the docs which make it seem like this is exactly what you want, right?

Or are you downloading a split file? May not help now but could you use the chunker backend next time?

1 Like

Thanks for your answer and reminding me about the existence of "chunker". I tried it almost two years ago and was not convinced for my use case.
Yes I split the files myself and upload the pieces, no problem there.
But when I have to download them I have to concatenate them after that and it takes time and space as the contents has grown with time.
Your suggestion makes me revisit "chunker" now, but as I'm happy with my own upload, do you think I could adapt the chunk names in the chunker directory so that the download "thinks" about them as one file? This way I wouldnt have to append them.

Manuel F.

You can try rclone cat --files-from listing remote: > output.file where listing contains a list, in order, of all the files that you want concatenating.

The chunker backend has quite a flexible naming scheme for its chunks - check it out here

https://rclone.org/chunker/#chunk-names

Many thanks Nick,
I think I found a way to speed up and simplify splitted downloads by using chunker's json file and modifying the naming scheme in the config as you said. I'm testing it with both copy and cat commands and looks good. The only problem with "cat" I cannot see the progress (-P) of the download as with copy (if anybody can help me, because I dont see any conclusion to a thread about this). Anyway it's no big deal as I have other ways to check whether it's going as expected.
Thanks again
Manuel F.

1 Like

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.