Set RCLONE logging level via RPC

Help and Support
1

I'm exploring the RC API via librclone and I wonder if there is a way to set log level and log file via RPC. My goal is to run sync/copy in DryRun mode and parse the output to see if source and destination have any differences. I know there is the [check](https://rclone.org/commands/rclone_check/) command but it don't look like it is available via rc.

Here is some code I have so far based on examples I found.

type RcloneSyncRequest struct {
	SrcFs  string `json:"srcFs"`
	DstFs  string `json:"dstFs"`
	Group  string `json:"_group"`
	Async  bool   `json:"_async"`
	Config string `json:"_config"`
}

type RcloneOptions struct {
	Main string `json:"main"`
}

options := RcloneOptions{
	Main: `{"LogLevel": "DEBUG"}`,
}

optionsJSON, err := json.Marshal(options)
if err != nil {
	fmt.Println(err)
}

out, status := librclone.RPC("options/set", string(optionsJSON))
if status != 200 {
	err := fmt.Errorf("AsyncCheck librclone.RPC error: Got status %d and output %q", status, out)
	return err
}

syncRequest := RcloneSyncRequest{
	SrcFs: srcPath,
	DstFs: dstPath,
	Async:  true,
	Config: `{"DryRun": true}`,
}

syncRequestJSON, err := json.Marshal(syncRequest)
if err != nil {
	fmt.Println(err)
}

out, status = librclone.RPC("sync/copy", string(syncRequestJSON))
if status != 200 {
err := fmt.Errorf("AsyncCheck librclone.RPC error: Got status %d and output %q", status, out)
	return err
}

However, when I run the above code I get following error:

2023/11/01 20:07:05 ERROR : rc: "options/set": error: unknown option block "main"

I saw some examples where logging was set in the following way:

rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'

My question is - is there a way to set log level (--log-level) and log file (--log-file) via librclone.RPC like it is possible via cli?

I went over this thread where core/log was discussed. Did this ever get any traction?

Also, another issue I'm having is after executing sync/copy command out is empty. I did some search and looks like this is related to returnType which should be set to "STREAM", "STREAM_ONLY_STDOUT", "STREAM_ONLY_STDERR".
Is there a way to set the returnType via librclone.RPC?

Another idea I had is to use core/command to run sync/copy/check with returnType set to STREAM but not sure if this is doable. I saw in this post @Ole created an HTTP request which has pretty much what I'm looking for. However, I don't know if the below HTTP snippet can be converted into a librclone.RPC call.

POST http://localhost:5572/core/command HTTP/1.1
content-type: application/json

{
    "command": "sync",
    "arg": [
        "/home/pi/data", 
        "aws-remote:s3.rclonebucket.testing", 
        "--dry-run", 
        "--log-level=NOTICE",
        "--use-json-log=false" 
        ],
    "returnType": "STREAM"
}

Thanks in advance!

2

Just do the API call as above (which is correct) via the API to set the --log-level.

I'm not sure setting the the log file via the API works at the moment though.

3

Running rclone with:

rclone rc options/set --json '{"main": {"LogLevel": "DEBUG"}}'

will require running HTTP server and rclone installation. I prefer to use it as a go module with librclone.

If LogLevel and LofFile options cannot be set via librclone.RPC would it possible to set the ReturnType via librclone.RPC call for sync/copy commands?
I'm not sure if ReturnType is a valid option for sync/copy commands. In docs I found it in the core/command description.

4

The librclone interface is exactly the same as the rclone rc option above.

So call something like

RcloneRPC("options/set", "{\"main\": {\"LogLevel\": \"DEBUG\"}}")

You will get an error return from the sync/copy commands which is equivalent to the return code that running the command line would do.

5

What is RcloneRPC?
Do you mean to run it like this:

libclone.RPC("options/set", "{\"main\": {\"LogLevel\": \"DEBUG\"}}")

I think I already tried that, if you look at my initial post, and was getting:

2023/11/01 20:07:05 ERROR : rc: "options/set": error: unknown option block "main"

You will get an error return from the sync/copy commands which is equivalent to the return code that running the command line would do.

I'm interested in the actual output with DryRun enabled which I could either forward into a log file or parse it on the fly to detect if there are any differences between the source and destination data (source - data in a s3 bucket. destination - data on my local pc).

In one of the posts you mentioned:

The rclone sync --dry-run will tell you files it wants to transfer and files it wants to delete on the remote. The rclone check will tell you fills that are present locally and not remotely, and files that are present remotely and not locally. rclone check has better reporting options.

Basically, how do I get that differences data so I can process it later on.

6

Yes run it like that. I forgot you were using Go and gave you a C example - sorry!

I suspect it isn't working because you haven't imported enough stuff from rclone. What does librclone.RPC options/get return?

You might be better off using rclone check or the api for it which can tell you exact differences.

This is only in the latest beta though.

operations/check: check the source and destination are the same {#operations-check}

Checks the files in the source and destination match. It compares
sizes and hashes and logs a report of files that don't
match. It doesn't alter the source or destination.

This takes the following parameters:

  • srcFs - a remote name string e.g. "drive:" for the source, "/" for local filesystem
  • dstFs - a remote name string e.g. "drive2:" for the destination, "/" for local filesystem
  • download - check by downloading rather than with hash
  • checkFileHash - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
  • checkFileFs - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
  • checkFileRemote - treat checkFileFs:checkFileRemote as a SUM file with hashes of given type
  • oneWay - check one way only, source files must exist on remote
  • combined - make a combined report of changes (default false)
  • missingOnSrc - report all files missing from the source (default true)
  • missingOnDst - report all files missing from the destination (default true)
  • match - report all matching files (default false)
  • differ - report all non-matching files (default true)
  • error - report all files with errors (hashing or reading) (default true)

If you supply the download flag, it will download the data from
both remotes and check them against each other on the fly. This can
be useful for remotes that don't support hashes or if you really want
to check all the data.

If you supply the size-only global flag, it will only compare the sizes not
the hashes as well. Use this for a quick check.

If you supply the checkFileHash option with a valid hash name, the
checkFileFs:checkFileRemote must point to a text file in the SUM
format. This treats the checksum file as the source and dstFs as the
destination. Note that srcFs is not used and should not be supplied in
this case.

Returns:

  • success - true if no error, false otherwise
  • status - textual summary of check, OK or text string
  • hashType - hash used in check, may be missing
  • combined - array of strings of combined report of changes
  • missingOnSrc - array of strings of all files missing from the source
  • missingOnDst - array of strings of all files missing from the destination
  • match - array of strings of all matching files
  • differ - array of strings of all non-matching files
  • error - array of strings of all files with errors (hashing or reading)

Authentication is required for this call.

7

I suspect it isn't working because you haven't imported enough stuff from rclone. What does librclone.RPC options/get return?

Here are my current imports:

_ "github.com/rclone/rclone/backend/local" // import only required backend
_ "github.com/rclone/rclone/backend/s3"    // import only required backend
"github.com/rclone/rclone/cmd"
"github.com/rclone/rclone/fs/cache"
"github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/sync"
"github.com/rclone/rclone/librclone/librclone"

Am I missing anything?

Not sure how to run options/get via rclone.RPC as it requires method string and input string. What would I need to set the input string in this case?

You might be better off using rclone check or the api for it which can tell you exact differences.

This is only in the latest beta though.

That's why I didn't see it in the official docs then. If the changes are not in main how would I use it via librclone.RPC? The link you provided for beta shows nothing for me.

image
image1236×242 8.77 KB

  • status - textual summary of check, OK or text string

So for opertions/check do I need to set any flags in order to get status as textual summary or text string. How do I enable/switch between other return options you listed in Returns?

8

The result of options/get will tell you.

Just to {} should do it.

I fixed that now - oops!

See also rclone selfupdate --beta

You get all those things returned together in a JSON blob with the keys as given.

9

Just to {} should do it.

I wasn't sure what that exactly means so I ran it the following way:

res, status := librclone.RPC("options/get", "")
fmt.Sprintf("Result: %s", res)

The response I got was this:

Result: {}
10

At a guess, you forgot to call librclone.Initialize()

It would be helpful if you shared an example I can try, rather than just snippets. This is a bit more work for you but if you save my time, you are more likely to get a good response :slight_smile:

11

I do call librclone.Initialize(). Here's some code I ran it with:

type RcloneOptions struct {
	Main string `json:"main"`
}

options := RcloneOptions{
	Main: `{"LogLevel": "DEBUG"}`,
}

optionsJSON, err := json.Marshal(options)
if err != nil {
	fmt.Println(err)
}

librclone.Initialize()
res, status := librclone.RPC("options/set", string(optionsJSON))
if status != 200 {
	err := fmt.Errorf("librclone.RPC error: Got status %d", status)
	return err
}

fmt.Sprintf("Result: %s", res)

Above code yields following output:

2023/11/06 10:43:25 ERROR : rc: "options/set": error: unknown option block "main"

If I run this code:

librclone.Initialize()
res, status := librclone.RPC("options/get", "")
if status != 200 {
	err := fmt.Errorf("librclone.RPC error: Got status %d", status)
	return err
}

fmt.Sprintf("Result: %s", res)

I get this:

Result: {}

Now when I run sync/copy with librclone.RPC rclone works and I can see following messages from rclone:

2023/11/06 10:51:04 NOTICE: Config file "C:\\Users\\...\\rclone\\rclone.conf" not found - using defaults
2023/11/06 10:51:04 NOTICE: test1.txt: Skipped copy as --dry-run is set (size 12)
2023/11/06 10:51:04 NOTICE: data-test1/data-test1.txt: Skipped copy as --dry-run is set (size 13)

I can share the sync/copy as well if needed.

12

If you can do that, I'll try it here and try to work out what is going on. I can't debug your code unless I can run it :slight_smile:

13

Can you try this:

import (
	"encoding/json"
	"fmt"
	"os"

	_ "github.com/rclone/rclone/backend/local" // import only required backend
	_ "github.com/rclone/rclone/backend/s3"    // import only required backend
	"github.com/rclone/rclone/cmd"
	"github.com/rclone/rclone/fs/cache"
	"github.com/rclone/rclone/fs/operations"
	"github.com/rclone/rclone/fs/sync"
	"github.com/rclone/rclone/librclone/librclone"
)


type RcloneSyncRequest struct {
	SrcFs  string `json:"srcFs"`
	DstFs  string `json:"dstFs"`
	Group  string `json:"_group"`
	Async  bool   `json:"_async"`
	Config string `json:"_config"`
}

type RcloneOptions struct {
	Main string `json:"main"`
}

func main() {
    os.Setenv("RCLONE_CONFIG_AWS-S3_TYPE", "s3")
    os.Setenv("RCLONE_CONFIG_AWS-S3_PROVIDER", "AWS")
    os.Setenv("RCLONE_CONFIG_AWS-S3_ENV_AUTH", "true")

    // Init librclone
	librclone.Initialize()
	
	// Run sync/copy
	syncRequest := RcloneSyncRequest{
		SrcFs: "your srcPath here",
		DstFs: "your dstPath here",
		Async:  true,
		Config: `{"DryRun": true}`,
	}
	
	syncRequestJSON, err := json.Marshal(syncRequest)
	if err != nil {
		fmt.Println(err)
	}
	
	out, status = librclone.RPC("sync/copy", string(syncRequestJSON))
	if status != 200 {
	err := fmt.Errorf("librclone.RPC error: Got status %d and output %q", status, out)
		return err
	}
	
	// Run options/set
	options := RcloneOptions{
		Main: `{"LogLevel": "DEBUG"}`,
	}
	
	optionsJSON, err := json.Marshal(options)
	if err != nil {
		fmt.Println(err)
	}
	
	res, status := librclone.RPC("options/set", string(optionsJSON))
	if status != 200 {
		err := fmt.Errorf("librclone.RPC error: Got status %d", status)
		return err
	}
	
	fmt.Sprintf("Result: %s", res)
	
	// Run options/get
	res, status = librclone.RPC("options/get", "")
	if status != 200 {
		err := fmt.Errorf("librclone.RPC error: Got status %d", status)
		return err
	}
	
	fmt.Sprintf("Result: %s", res)
}

sync/copy works fine. but options/set and options/get don't work for me

14

I had a look at this, and it is a bug in rclone!

The rc flags aren't being registered because these things aren't being called.

And these things need to be refactored so that they don't install command line flags.

	configflags.AddFlags(ci, pflag.CommandLine)
	filterflags.AddFlags(pflag.CommandLine)
	rcflags.AddFlags(pflag.CommandLine)
	logflags.AddFlags(pflag.CommandLine)

Can you open a new issue on Github about this with a link to this forum post and I will fix at some point. Or you could have a go - happy to give pointers.

15

Hey, thanks for looking into and validating this.
I created following issue:
Set rclone logging level via RPC · Issue #7439 · rclone/rclone (github.com)

I'll be travelling next week. Happy to contribute once I'm back.

1 Like
16

Another related question. Once I started the copy process with

out, status = librclone.RPC("sync/copy", string(syncRequestJSON))

Is there a way to kill the copy process via librclone.RPC?
I'm looking for functionality similar to CTRL-C if I was running it as CLI.

Are there any examples how to use these:

core/pid: Return PID of current process
This returns PID of current process. Useful for stopping rclone process.

core/quit: Terminates the app.
(Optional) Pass an exit code to be used for terminating the app:

exitCode - int
17

You want job/stop

This assumes you started the job in an async manner and collected the job id when you started it.

18

Yep, running the sync/copy operation with async set to true.
Per rclone docs

If _async has a true value when supplied to an rc call then it will return immediately with a job id and the task will be run in the background.

Initially, I had this wrapped in a go routine to run this process in the background. Looks like with async enabled this no longer needed.

Thanks

1 Like
20

Hey, can you give me some pointers on what to do regarding rc flags aren't being registered?

21

Can you be more specific? I'm not sure which flags you mean.