What is the rc copyurl syntax?

Help and Support
1

Hey there, tying to use rclone to download http(s) files and I can't seem to get it to fire correctly - I have other services set up correctly, however.

I have tried my normal wrapper function (with a http/https config already in place) to no avail.
"rclone.exe" rc operations/copyfile "srcFs=https:" "srcRemote=pbs.twimg.com/media/CPStPl1UkAEHetK.png" "dstFs=a:" "dstRemote=test.jpg"
results in
2019/10/24 09:45:51 ERROR : rc: "operations/copyfile": error: object not found

So I tried my hand at operations/copyurl. I've successfully downloaded with normal (non-rc) rclone but I can't get an appropriate handoff to the server.

What part of the syntax am I missing? Either copyfile or copyurl is fine, though I would prefer copyfile if possible.

2

How is your https remote set up? I think the problem might be there.

4

I did two configs with the interactive menu, named http and https (basically just to alias the names so I didn't have to code more checks.)

Both were made with option 1. (no login info)

The urls are a list of pictures hosted on Twitter, perfectly accessible via browser and downloadable via copy url (without rc.)

5

It would be helpful to see them. Can you do rclone config show then XXX the bits you want to keep private and post it here?

1 Like
6

Sure thing. Here's everything.
image

image

Here's the setup I went through.
image
(option 30)
image

Looking at the config I see that it actually entered example.com. I thought it would set up a generic url downloader.

7

Forgot to reply to you directly. Oops.

8

I don't think that is what he asked for (although he will likely be able to figure it out with that data too).

If you run
rclone config show
this will print the contents of your rclone.conf - which describes your setup

Or alternatively - if you find it easier, just open your rclone.conf file and copypaste the content.

This will give a config-dump that is not only much easier for you to produce - but also easier for us to look through rather than trying to "reconstruct" it from the options you chose in the menus.

(!warning!, rclone.conf can contain sensitive data such as client secrets and crypt keys so this should be redacted before posting for the sake of your privacy/security)

1 Like
9

Second picture is rclone config show (the one with the blur)

10

My apologies. I missed that due to all the other pictures :slight_smile:
yes, that should suffice I think.

1 Like
11

Good point, added a space. :stuck_out_tongue:

12

That is the problem, you've configured the remote for example.com...

If you were doing this on the command line you could do

rclone copyto :http:CPStPl1UkAEHetK.png a:test.jpg --http-url https://pbs.twimg.com/media

If you make your config look like

[pbs]
type=http
url=https://pbs.twimg.com

Then this should work

rclone rc operations/copyfile "srcFs=pbs:" "srcRemote=media/CPStPl1UkAEHetK.png" "dstFs=a:" "dstRemote=test.jpg"
1 Like
13

So i'll need to add a new config for every single domain/subdomain combination? All I'm really looking for is the ability to download from a given URL while also making use of rclone's "it doesn't exist at the specified location until it's fully ready" (that is, the temp storage) feature. Some URLs passed may point to quite large files that I don't want to accidentally corrupt or incompletely process.

I'm making use of rclone to offload the upload duties and would love to also offload the download duties (processing only when the file is properly in place.)

14

If you use the rc then yes. If you just use a command line parameter then no. Note that you can create and delete configs via the remote control interface.

Or you could use copyurl - was there some reason that you didn't want to use that?

1 Like
15

I wasn't able to get copyurl to work via rc points at topic title, lol

As for why I don't want to use copyurl:
I'm working on processing a large amount of data that's updated constantly. Concurrency is key. One of the big draws of rclone (among so many! <3) is the temp storage feature, so I can check if the file exists and process or just move on. I won't have any info about file size beforehand.

I don't mind adding configs via rc. Now that I know what the command looks like it'll be easy enough to parse out. :slight_smile:

16

Here is an example

rclone rc operations/copyurl fs=/tmp remote=file.txt url=https://google.com/
1 Like
17

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.