I'm looking for a way to send a file local to the user's workstation to a Ceph server using the rclone protocol.
I am sorry for this question. I have searched many places with no luck. I guess I'm not searching by the right criteria as I must not be the first to do this.
Thank you for any help you can give me.
Have a nice day!
TH
Ceph provides the radosgw which in turn provides the s3 protocol and the swift protocol, both of which rclone supports. If you have a choice the s3 protocol probably works best. There is an s3 provider for Ceph.
Ceph does lots of other things also but I'm guessing the s3 protocol is what you want.
As my rclone configuration above shows, I'm already using the S3 protocol to send data via rclone.
What I'd like to know is how to start an upload using rclone when I click on the HTML button without having to make an initial upload to the web server (sorry, I don't know if I'm making myself clear)
What you describe is the solution I'm currently using. This solution has one major problem: before using it, you have to upload the file to the web server in the conventional way (i.e. slower than with rclone).
I'm looking to eliminate this intermediate transfer.
I am afraid it has nothing to do with rclone per se. It can work from web-server, it can work from client. What sort of help you would expect from this forum here? rclone works. Where and how you run it is entirely up to you.
Also what you call "clicking on submit button" - it is some application you are using - either stand-alone or online. You have to talk to the application author about rclone integration.
IMHO you should have a chat with somebody who designed your system. They are the right people to help in this situation.
I know how to send files conventionally in HTML, but that's not what I'm looking for. My question was how to run an rclone command from an HTML button to avoid an unnecessary transfer.
I'm the one who's setting up the website that's going to simplify the opening up of data for users in my unit, and I've got no one to refer to other than people who've already used rclone for similar purposes.
you can use rc command operations/copyfile by submitting following HTTP POST:
curl -X POST 'http://127.0.0.1:5575/operations/copyfile?srcFs=/path/to/&srcRemote=myfile.txt&dstFs=spider-ceph:&dstRemote=path/to/destination/myfile.txt’
I did several tests yesterday and this morning about the API. I seem to have started it up incorrectly because, following the construction of your link, I only get a "404 page not found" error. The same applies if I use simple APIs such as "/cache/stats", which doesn't require any parameters.
Here's how I start the rclone service on the web server :
I don't have access to the API, but I have access to the rclone web interface. Which tells me that I may have missed something in my rclone configuration
--s3-chunk-size 10M <---- this does not belong here, it is S3 remote parameter. If you need it move it to your rclone.config --rc-addr 0.0.0.0:5572 <----- not sure if it makes difference but documentation says use --rc-addr :5572 to listen on all IPs
for testing if rc is running I use simple command like list all my remotes:
curl -X POST 'http://localhost:5572/config/listremotes'
for you authenticated and TLS connection this would be curl command to test:
curl -X POST 'https://localhost:5572/config/listremotes' -u "rc-user:rc-pass"
Also I would test first without any authentication --rc-no-auth then move to add --rc-user rc-user --rc-pass rc-pass and if working add all certs. There might be challenges at every step so better to work it out from the bottom instead of fighting with everything at once IMHO.
First test I always do is to check if rc is running and port is listening. I use telnet for this
With rc running:
# telnet 127.0.0.1 5575
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close
With rc not running:
# telnet 127.0.0.1 5575
400 Bad RequestConnection closed by foreign host.
Very nice that it starts working. Thank you for sharing results.
Only question is why to listen on all IP addresses? Wouldn't limiting scope to only 127.0.0.1 make it more secure? From your drawings earlier I understand that both rclone and "website/web app" are on client machine. Of course you might have some other consideration I am not aware of:)
The project development team using rclone has chosen to create a java web application, hosted by a payara-server in a docker container. I haven't carried out any further tests for the moment, but it seemed necessary to open at 0.0.0.0:5572 ... but perhaps I was wrong.
I made several attempts with copyfile but I understood that this method was more for copying a remote file to another remote server.
I've just tried again with the uploadfile method, which seems more suitable. Unfortunately, I'm not having any more success. My best result is as follows. The frame indicates that the transfer is successful (200) but the file is not transferred in the end.
C:\Users\theirman
λ curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph&remote=/recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip
Note: Unnecessary use of -X or --request, POST is already inferred.
* Trying aa.bb.cc.dd:5572...
* Connected to aa.bb.cc.dd (aa.bb.cc.dd) port 5572 (#0)
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
* Server auth using Basic with user 'rc-user'
> POST /operations/uploadfile?fs=spider-ceph&remote=/recette HTTP/1.1
> Host: aa.bb.cc.dd:5572
> Authorization: Basic xxxxxxxxxxxxxxxxxxxx==
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Length: 13722239
> Content-Type: multipart/form-data; boundary=------------------------78462a841e51d376
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: authorization, Content-Type
< Access-Control-Allow-Origin: https://0.0.0.0:5572/
< Access-Control-Request-Method: POST, OPTIONS, GET, HEAD
< X-Rclone-Jobid: 26
< Date: Thu, 08 Jun 2023 13:23:31 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
<
{}
* Connection #0 to host aa.bb.cc.dd left intact
You are correct but local filesystem is also considered "remote". It can even be configured in rclone.config as type = local but there is no need for this as rclone treats all local paths as local remote.
And as often there is more than one way to skin a cat. operations/uploadfile or operations/copyfile both should work.
curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph&remote=/recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip
I would add ":" after remote name and remove "/" from path
so:
curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph:&remote=recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip
I have never used this command myself over rc - but this is general style of providing remote and path.