Rclone RC cat not outputting bytes

Suspected Bug
1

What is the problem you are having with rclone?

I'm trying to use rclone's rc server to run rclone cat and output it's results.
The result should be in bytes, but rclone rc uses JSON, which encodes the bytes into a malformed string.

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

  • os/version: Microsoft Windows 11 Home 22H2 (64 bit)
  • os/kernel: 10.0.22621.1 (x86_64)
  • os/type: windows
  • os/arch: amd64
  • go/version: go1.17.9
  • go/linking: dynamic
  • go/tags: cmount

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone rc core/command command=cat -a fs:/image.png

The rclone config contents with secrets removed.

Irrelevant

A log from the command with the -vv flag

A log is irrelevant, but as an example of an expected output is:

image
image1102×410 29.2 KB

This output comes from running

rclone cat fs:/image.png

While the rc command outputs

image
image1087×496 34.2 KB

1 Like
2

The output is a valid JSON string, you need to read it with a JSON parser. I just checked the output with jq and with python's json module, and both are happy with the binary encoding rclone is using.

3

Could you please provide an example? I'm scratching my head on how to parse the JSON data into bytes.

This is what I've been testing with on Python.

import json, requests

rc_data = {
    "arg": ["td:image.png"],
    "command": "cat",
}
result = requests.post(
    "http://localhost:35530/core/command",
    data=json.dumps(rc_data),
    headers={"Content-Type": "application/json"},
).json()

with open("image.png", "wb") as f:
    f.write(result["result"].encode("utf-8"))

Thanks!

4

Looking at the data more, I think that rclone has mangled the binary data beyond repair by treating it as a utf-8 string. Python decoded that validly but it is nothing like the data that is supposed to be in there!

We could fix this by adding another parameter to core/command say -o binary which would then return the result as a base64 encoded string. Would that work for you?

If so, do you want to have a go at adding it?

5

BTW there are easier ways of just reading files than using rclone cat you can use an http GET on the file serving part of rclone. Add --rc-serve to your rclone rcd command line then use

http://localhost:5572/[/]/

or

http://localhost:5572/[td:]/
1 Like
6

Thanks for the response! -o binary would be a great solution. Honestly, I've never touched golang before, but I'm still open to contributing. Could you possibly outline which files in the rclone repo are relevant to this problem?

7

The code is defined here

Looking at it I remember now that there is a STREAM option which is probably what you want instead...

import json, requests

rc_data = {
    "arg": ["/tmp/randombytes"],
    "command": "cat",
    "returnType": "STREAM_ONLY_STDOUT",
}
result = requests.post(
    "http://localhost:5572/core/command",
    data=json.dumps(rc_data),
    headers={"Content-Type": "application/json"},
)

data = result.content
print(len(data))

with open("randombytes.copy", "wb") as f:
    f.write(data)

This works but has a small bit of random data on the end {} which is probably a bug!

So you could use that, or have a go at implementing a parameter. I'd call the parameter binary and read it with binary, _ := in.GetBool("binary") which will default to false if the parameter isn't supplied.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.