Copyurl from file (different remotes)

[lacasifeyu]

What is the problem you are having with rclone?

I would like to copy files from different http direct links to my remote folder.
So I have a list of links that are from different servers, otherwise I would have tried rclone copy --files-from list.txt source: remote: -P.
The list looks like this

https://aaa.com/aaa/aaa.txt
https://bbb.com/bbb/bbb.txt

And I would like to save them all in my remote in the same folder.

The only workaround I could think of right now is mounting my remote and use wget -i text_file.txt but this is only suboptimal.

edit 1:
I thought I coud trick it by running a local redirect server, but I guess it does not copy the file sbecause it cannot find them when scanning for them.

edit 2:
rclone copy --files-from files-from.txt --http-url http://0.0.0.0:5000/?url= :http: remote: -vv --no-traverse
and adding ?url= in front of every line in the textfile lets me download the files but the names are really ugly (it creates a new folder for every url etc.)

import os

from flask import Flask, redirect
from flask import request

app = Flask(__name__)


@app.route('/')
def forward_url():
    url = request.args.get('url', type=str, default='')
    if url != '':
        return redirect(url, code=307)
    else:
        return url


if __name__ == '__main__':
    # Bind to PORT if defined, otherwise default to 5000.
    port = int(os.environ.get('PORT', 5000))
    app.run(host='0.0.0.0', port=port)

What is your rclone version (output from rclone version)

rclone v1.51.0
- os/arch: linux/amd64
- go version: go1.13.7

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copyurl --files-from files-from.txt remote: -a -P

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

Error: need 2 arguments if not using --stdout
Usage:
  rclone copyurl https://example.com dest:path [flags]

Flags:
  -a, --auto-filename   Get the file name from the URL and use it for destination file path
  -h, --help            help for copyurl
      --stdout          Write the output to stdout rather than a file

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

[ncw (Nick Craig-Wood)]

I guess you could script copyurl like this

xargs -i rclone copyurl {} remote: -a -P <file-with-list-of-URLs

does that work for you?

The other alternative would be to make copyurl take a list of URLs in a new flag.

[lacasifeyu]

Thank you for your answer ncw. Making copyurl take a list of URLs would probably be a nicer option because then you could transfer multiple files at the same time. with the xargs option, you would have to wait for one file to finish. I have never used go before, that's why I only tried to write a simple workaround with python.

[ncw (Nick Craig-Wood)]

Do you want to make a new issue on github about the list of files for copyurl?