Add rcat to rc command

Help and Support
#1

What is the problem you are having with rclone?

I'm using rcat command to stream on Azure Blob a large file while this is written by a batch process.
batch process | rclone rcat ......
I would monitor file transfer with Prometheus but I understand that Prometheus metrics are available only on rclone rcd.

Are you know another way to have Prometheus metrics or is it planned to add rcat to rc command available?

Thanks in advance
Regards

What is your rclone version (output from rclone version)

1.52.0

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian (Docker)

Which cloud storage system are you using? (eg Google Drive)

Azure Blob

The command you were trying to run (eg rclone copy /tmp remote:tmp)

Paste command here

The rclone config contents with secrets removed.

Paste config here

A log from the command with the -vv flag

Paste  log here
#2

If you add --rc you will get all the things rclone rcd exposes including prometheus metrics. Note you'll need v1.52.2 as the prometheus metrics were incorrect before then!

#3

I have a different usage scenario, but I too would like control over cat/rcat over using rcd.

For Android, it would be really usefull if I could tell rclone to pump remote file contents into a named pipe (mkfifo) or accept file uploads over an arbitrary pipe. This would help immensly to simplify the glue code between the Android native fuse loop and rclone.

#4

We do have uploadfile in the latest beta whic uploads from an http stream. The RC can also download files to an http stream.

Are those primitives useful?

An upload to/from a fifo would be easy enough I think though.

#5

Sorry for hijacking this thread.

We do have uploadfile in the latest beta whic uploads from an http stream. The RC can also download files to an http stream.

Are those primitives useful?

In a remote context probably very much, e.g.with rclone is running on a server, and clients like the Android app connecting to it.

I was mostly thinking about arbitrary pipes because HTTP requries allocating new ports and introduces overhead, and the number of standard pipes (stdin/stdout/stderr) is limited and used for other purposes. And unlike an approach with shared memory via JNI (possibly slightly more efficient), pipes are available on most platforms and languages.

#6

By a pipe do you mean a file descriptor here? So you could say

rclone rcat --fd 17 remote:path
rclone cat --fd 17 remote:path

Another interface would be to allow rclone to read/write to a fifo/socket/device which would all be pretty much the same code so

rclone rcat --dev myfifo remote:path
rclone cat --dev myfifo remote:path

These could then be made to work over the rc I guess.

#7

Yeah, pretty much. For my use case, i would create a named pipe with mkfifo (or the equivalent sycall, more precisely), and then use rclone to pump file bytes into that file descriptor or in the other direction.

I'll make a note to prepare a PR, it seems pretty doable actually.

#8

Look forward to the PR!

#9

Sorry I don't understand how i'd change command.
In this way?
rclone rcd --rc-addr=127.0.0.1:6000 --rc-enable-metrics
......
echo "Hello World!" | rclone rcat remote:path --rc-addr=127.0.0.1:6000

#10

You can't yet...

It would be possible to make rcat understand the new upload file API in the beta (or maybe not merged yet - can't remember) so that could work in the future.

#11

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.