How to copy files simultaneously with HTTP remote

Help and Support
1

I need to copy hundreds of thousands of files from remote HTTP server. The server doesn’t allow file listings, meaning I can access http://example.com/directory/123.jpg but not http://example.com/directory/
I have all file names in the directory.

This works fine when I use direct file path
rclone copy remote1:123.jpg remote2:savepath
where remote1 is /directory/

But how can I give it a list of all file names so they are copied simultaneously?

I tried to do it with
rclone --files-from list.txt copy remote1: remote2:savepath
where in list.txt I provide file names, but I get “error reading source directory”

Is it possible to skip file listings? As far as I understand, rclone tries to read directory, get all file names, and compare them with provided file names, so this part causes problem.

2

It isn’t possible currently to do that - copy will require a directory listing :frowning:

You could do a little bash loop over the input files something like

for i in `cat zz`; do rclone copy remote1:$i remote2:savepath ; done
3

Thank you for your response! What I’m doing now is, I opened 5 instances of cmd.exe and I enter hundreds of rclone commands, one for each file, in each cmd window. This helps to get simultaneous upload, but it’s still too slow.

If I somehow do that loop you mentioned, would it speed things up? My understanding of this is limited. I understand that copying is slow because one rclone command needs to finish before next can start. If command is executed in loop, does one iteration needs to finish before next start? I won’t be able to use --transfers flag?

I’m not a programmer :frowning: Learning how to do this would take me a considerable amount of time. I’m just wondering if it’s worth it?

4

Yes you are correct, that would just do one at a time and you can’t use --transfers…

I had a go at making rclone work using your original thought, so making --files-from not scan the directories and skip the listings. You aren’t the first person to ask for that feature!

https://beta.rclone.org/branch/v1.44-006-g10366a73-files-from-noscan-beta/ (uploaded in 15-30 mins)

Have a go with that and see if it fixes your problem? You can set --transfers up as high as you like with that.

5

Hi Nick. It worked like a charm! I can’t thank you enough dude. I checked your donation page and I’m more than eager to help you buy your wife a flower or two. I’m afraid I don’t have for a whole bucket :grinning: (I’m checking how much it costs in UK, it’s really £20-30 ? :open_mouth:) But one day when I become rich I’ll be sure to remember you :stuck_out_tongue: I don’t have paypal though, they ignored my country (Serbia) for too long. I’ll see if I can open it now so I can donate.

I have one question though. Does http remote download files to my pc first and then upload? It seems whatever I tried I can’t pass 100kb/s. Default 4 transfers work best with pcloud.com. I tried to increase transfers, run multiple instances of rclone with separate --files-from lists, etc. it always comes down to 100kb/s. I don’t know if it’s pcloud.com limit or rclone limit, but my internet upload speed is 100kb/s so I come to the conclusion that it’s limit on my part?

Regardless, being able to skip directory listing is a great feature to have and I’m sure I’ll use it many times in future. I work in ecommerce field, and suppliers often give us direct links to images but directory listing usually disabled.

6

:smiley: glad it worked! I’ll work on tidying the code up and getting it merged to master, but really glad it worked for you!

Yes all rclone transfers will come through your PC so that explains it perfectly. What people often do is if they have loads of stuff to transfer is rent a cloud server for a few hours and transfer it there.

Ah, interesting use case - thanks.