Hey there, tying to use rclone to download http(s) files and I can't seem to get it to fire correctly - I have other services set up correctly, however.
I have tried my normal wrapper function (with a http/https config already in place) to no avail.
"rclone.exe" rc operations/copyfile "srcFs=https:" "srcRemote=pbs.twimg.com/media/CPStPl1UkAEHetK.png" "dstFs=a:" "dstRemote=test.jpg"
2019/10/24 09:45:51 ERROR : rc: "operations/copyfile": error: object not found
So I tried my hand at operations/copyurl. I've successfully downloaded with normal (non-rc) rclone but I can't get an appropriate handoff to the server.
What part of the syntax am I missing? Either copyfile or copyurl is fine, though I would prefer copyfile if possible.
How is your
https remote set up? I think the problem might be there.
I did two configs with the interactive menu, named http and https (basically just to alias the names so I didn't have to code more checks.)
Both were made with option 1. (no login info)
The urls are a list of pictures hosted on Twitter, perfectly accessible via browser and downloadable via copy url (without rc.)
It would be helpful to see them. Can you do
rclone config show then XXX the bits you want to keep private and post it here?
Sure thing. Here's everything.
Here's the setup I went through.
Looking at the config I see that it actually entered example.com. I thought it would set up a generic url downloader.
Forgot to reply to you directly. Oops.
I don't think that is what he asked for (although he will likely be able to figure it out with that data too).
If you run
rclone config show
this will print the contents of your rclone.conf - which describes your setup
Or alternatively - if you find it easier, just open your rclone.conf file and copypaste the content.
This will give a config-dump that is not only much easier for you to produce - but also easier for us to look through rather than trying to "reconstruct" it from the options you chose in the menus.
(!warning!, rclone.conf can contain sensitive data such as client secrets and crypt keys so this should be redacted before posting for the sake of your privacy/security)
Second picture is rclone config show (the one with the blur)
My apologies. I missed that due to all the other pictures
yes, that should suffice I think.
Good point, added a space.
That is the problem, you've configured the remote for
If you were doing this on the command line you could do
rclone copyto :http:CPStPl1UkAEHetK.png a:test.jpg --http-url https://pbs.twimg.com/media
If you make your config look like
Then this should work
rclone rc operations/copyfile "srcFs=pbs:" "srcRemote=media/CPStPl1UkAEHetK.png" "dstFs=a:" "dstRemote=test.jpg"
So i'll need to add a new config for every single domain/subdomain combination? All I'm really looking for is the ability to download from a given URL while also making use of rclone's "it doesn't exist at the specified location until it's fully ready" (that is, the temp storage) feature. Some URLs passed may point to quite large files that I don't want to accidentally corrupt or incompletely process.
I'm making use of rclone to offload the upload duties and would love to also offload the download duties (processing only when the file is properly in place.)
If you use the rc then yes. If you just use a command line parameter then no. Note that you can create and delete configs via the remote control interface.
Or you could use
copyurl - was there some reason that you didn't want to use that?
I wasn't able to get copyurl to work via rc points at topic title, lol
As for why I don't want to use copyurl:
I'm working on processing a large amount of data that's updated constantly. Concurrency is key. One of the big draws of rclone (among so many! <3) is the temp storage feature, so I can check if the file exists and process or just move on. I won't have any info about file size beforehand.
I don't mind adding configs via rc. Now that I know what the command looks like it'll be easy enough to parse out.
Here is an example
rclone rc operations/copyurl fs=/tmp remote=file.txt url=https://google.com/
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.