How to send a file with rclone using an HTML button?

Hello

I'm looking for a way to send a file local to the user's workstation to a Ceph server using the rclone protocol.

I am sorry for this question. I have searched many places with no luck. I guess I'm not searching by the right criteria as I must not be the first to do this.

Thank you for any help you can give me.
Have a nice day!
TH

To clarify there is nothing like rclone protocol - it is just a tool able to connect to various remotes using various protocols.

What access you have to Ceph server? it has many ways to connect. Provide all info available. Do not share passwords etc.

Ceph provides the radosgw which in turn provides the s3 protocol and the swift protocol, both of which rclone supports. If you have a choice the s3 protocol probably works best. There is an s3 provider for Ceph.

Ceph does lots of other things also but I'm guessing the s3 protocol is what you want.

Thanks for you answer @kapitainsky

Here is my rclone configuration :

[spider-ceph]
type = s3
provider = Ceph
env_auth = false
access_key_id = my-access-key
secret_access_key = my-secret-access-key
endpoint = my-endpoint
acl = private

Thanks for your answer @ncw

As my rclone configuration above shows, I'm already using the S3 protocol to send data via rclone.
What I'd like to know is how to start an upload using rclone when I click on the HTML button without having to make an initial upload to the web server (sorry, I don't know if I'm making myself clear)

Ok - it means you are using S3 and it is already configured.

To copy file from your computer to ceph run this command:

rclone copy /path/to/myfile.txt  spider-ceph:path/to/destination

I am not sure what you mean by HTML button. rclone is command line utility.

What you describe is the solution I'm currently using. This solution has one major problem: before using it, you have to upload the file to the web server in the conventional way (i.e. slower than with rclone).

I'm looking to eliminate this intermediate transfer.

I am afraid it has nothing to do with rclone per se. It can work from web-server, it can work from client. What sort of help you would expect from this forum here? rclone works. Where and how you run it is entirely up to you.

Also what you call "clicking on submit button" - it is some application you are using - either stand-alone or online. You have to talk to the application author about rclone integration.

IMHO you should have a chat with somebody who designed your system. They are the right people to help in this situation.

I know how to send files conventionally in HTML, but that's not what I'm looking for. My question was how to run an rclone command from an HTML button to avoid an unnecessary transfer.

I'm the one who's setting up the website that's going to simplify the opening up of data for users in my unit, and I've got no one to refer to other than people who've already used rclone for similar purposes.

I'm sorry to bother, here.

I'll look for another tool to do what I need. rclone didn't seem so bad to me, though !

Bonne continuation

If you want to interact with rclone from a browser I can see the following solution using rclone remote control:

First run rclone listening to remote control commands - all details here. I would run it as a service - so it is always on

rclone rcd --rc-addr 127.0.0.1:5575 --rc-no-auth

Now you can issue commands using HTTP POST (it can be from your browser page when you press your proverbial HTML button)
for example to run:

rclone copy /path/to/myfile.txt  spider-ceph:path/to/destination

you can use rc command operations/copyfile by submitting following HTTP POST:

curl -X POST 'http://127.0.0.1:5575/operations/copyfile?srcFs=/path/to/&srcRemote=myfile.txt&dstFs=spider-ceph:&dstRemote=path/to/destination/myfile.txt’

Thank you for your reply @kapitainsky

I did several tests yesterday and this morning about the API. I seem to have started it up incorrectly because, following the construction of your link, I only get a "404 page not found" error. The same applies if I use simple APIs such as "/cache/stats", which doesn't require any parameters.

Here's how I start the rclone service on the web server :

[Unit]
Description=Rclone Server
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=rclone rcd --rc-addr 0.0.0.0:5572 --rc-user rc-user --rc-pass rc-pass --s3-chunk-size 10M --rc-web-gui --rc-cert /etc/letsencrypt/live/my-server/cert.pem --rc-key /etc/letsencrypt/live/my-server/privkey.pem --config=/etc/rclone/rclone.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

I don't have access to the API, but I have access to the rclone web interface. Which tells me that I may have missed something in my rclone configuration

--s3-chunk-size 10M <---- this does not belong here, it is S3 remote parameter. If you need it move it to your rclone.config
--rc-addr 0.0.0.0:5572 <----- not sure if it makes difference but documentation says use --rc-addr :5572 to listen on all IPs

for testing if rc is running I use simple command like list all my remotes:

curl -X POST 'http://localhost:5572/config/listremotes'

for you authenticated and TLS connection this would be curl command to test:

curl -X POST 'https://localhost:5572/config/listremotes' -u "rc-user:rc-pass"

Also I would test first without any authentication --rc-no-auth then move to add --rc-user rc-user --rc-pass rc-pass and if working add all certs. There might be challenges at every step so better to work it out from the bottom instead of fighting with everything at once IMHO.

My suspicion is that you do not use POST but GET.

With GET in my test I have also 404.

As per documentation:

Rclone implements a simple HTTP based protocol.

All calls must made using POST.

BTW

--rc-addr 0.0.0.0:5572 is fine - I have just tested

# rclone rcd --rc-addr 0.0.0.0:5575 --rc-no-auth
2023/06/07 09:16:18 NOTICE: Serving remote control on http://[::]:5575

First test I always do is to check if rc is running and port is listening. I use telnet for this

With rc running:

# telnet 127.0.0.1 5575
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

HTTP/1.1 400 Bad Request
Content-Type: text/plain; charset=utf-8
Connection: close

With rc not running:

# telnet 127.0.0.1 5575
400 Bad RequestConnection closed by foreign host.

Thanks for all your help

this morning, following your advice, I tried in this order

  • [OK] Run a minimal rclone service without authentication (--rc-user, --rc-pass) and without a certificate (--rc-cert, --rc-key)
$ sudo rclone rcd --rc-addr 0.0.0.0:5572 --config=/etc/rclone/rclone.conf
2023/06/08 09:56:50 NOTICE: Serving remote control on http://0.0.0.0:5572/
  • [OK] Check that the port was listening
$ curl -v telnet://aa.bb.cc.dd:5572
*   Trying aa.bb.cc.dd:5572...
* Connected to aa.bb.cc.dd (aa.bb.cc.dd) port 5572 (#0)
  • [OK] Check the curl request in localhost on the server
$ curl -X POST "http://localhost:5572/config/listremotes"
{
        "remotes": [
                "spider-ceph"
        ]
}
  • [OK] Check curl request across network on server
$ curl -X POST "http://aa.bb.cc.dd:5572/config/listremotes"
{
        "remotes": [
                "spider-ceph"
        ]
}
  • [OK] check the curl request across the network from my PC
$ curl -X POST "http://aa.bb.cc.dd:5572/config/listremotes"
{
        "remotes": [
                "spider-ceph"
        ]
}

Then I restarted the server, adding the --rc-user and --rc-pass info. All the tests detailed above worked by adding the credentials with -u

Finally, I restarted the server, adding the --rc-cert and --rc-key info. All tests worked by replacing http with https

I'm now going to test file upload with copyfile.
I'll get back to you as soon as I can.
Thanks again @kapitainsky

Very nice that it starts working. Thank you for sharing results.

Only question is why to listen on all IP addresses? Wouldn't limiting scope to only 127.0.0.1 make it more secure? From your drawings earlier I understand that both rclone and "website/web app" are on client machine. Of course you might have some other consideration I am not aware of:)

The project development team using rclone has chosen to create a java web application, hosted by a payara-server in a docker container. I haven't carried out any further tests for the moment, but it seemed necessary to open at 0.0.0.0:5572 ... but perhaps I was wrong.

I made several attempts with copyfile but I understood that this method was more for copying a remote file to another remote server.

I've just tried again with the uploadfile method, which seems more suitable. Unfortunately, I'm not having any more success. My best result is as follows. The frame indicates that the transfer is successful (200) but the file is not transferred in the end.

C:\Users\theirman
λ curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph&remote=/recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying aa.bb.cc.dd:5572...
* Connected to aa.bb.cc.dd (aa.bb.cc.dd) port 5572 (#0)
* schannel: disabled automatic use of client certificate
* schannel: using IP address, SNI is not supported by OS.
* ALPN: offers http/1.1
* ALPN: server accepted http/1.1
* using HTTP/1.1
* Server auth using Basic with user 'rc-user'
> POST /operations/uploadfile?fs=spider-ceph&remote=/recette HTTP/1.1
> Host: aa.bb.cc.dd:5572
> Authorization: Basic xxxxxxxxxxxxxxxxxxxx==
> User-Agent: curl/8.0.1
> Accept: */*
> Content-Length: 13722239
> Content-Type: multipart/form-data; boundary=------------------------78462a841e51d376
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Access-Control-Allow-Headers: authorization, Content-Type
< Access-Control-Allow-Origin: https://0.0.0.0:5572/
< Access-Control-Request-Method: POST, OPTIONS, GET, HEAD
< X-Rclone-Jobid: 26
< Date: Thu, 08 Jun 2023 13:23:31 GMT
< Content-Length: 3
< Content-Type: text/plain; charset=utf-8
<
{}
* Connection #0 to host aa.bb.cc.dd left intact

You are correct but local filesystem is also considered "remote". It can even be configured in rclone.config as type = local but there is no need for this as rclone treats all local paths as local remote.

And as often there is more than one way to skin a cat. operations/uploadfile or operations/copyfile both should work.

curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph&remote=/recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip

I would add ":" after remote name and remove "/" from path

so:

curl -v -k -X POST "https://aa.bb.cc.dd:5572/operations/uploadfile?fs=spider-ceph:&remote=recette" -u "rc-user:rc-pass" -F file=@c:\temp\aliments.zip

I have never used this command myself over rc - but this is general style of providing remote and path.