Use Rclone golang to transfer files

I created a Draft pull request - fs:Added multiple ca certificate support. by alankritkharbanda · Pull Request #6688 · rclone/rclone · GitHub

Do you think this would help in supporting multiple certificates?

I think your pull request looks perfectly reasonable :slight_smile:

I'll give it a quick review now.

Hi

I updated the pull request with the comments and tested the same on my local system as well :slight_smile:

1 Like

Thank you for merging the same !
Just wanted to check - when is the next release due ? Wanted to ensure we import the next release in our codebase.

I don't think I set a release date yet, but rclone releases usually come out with a cadence of 2 months.

Hi

I had a question. Currently I am using sync.Sync, but seems like that is not the proper way to use the library and instead RPC should be used instead.

I was trying to do the following :

rclone sync s3-like-system-1://bucket1 s3-like-system-2://bucket2 --ca-cert ca-1.crt --ca-cert ca-2.crt

For which I would have needed something like :

librclone.RPC("sync", "{srcFs : {...}, dstFs : {...}}"

However I understand that sync or operations/sync is not a directly usable option. Could you kindly help me with an example to trigger RPC sync here?

Running into a somewhat similar issue:

From my go.mod file:
require GitHub - rclone/rclone: "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files v1.61.1

From my go code:

status, out := librclone.RPC("sync/sync", syncArgs)
fmt.Println(status)
fmt.Println(out)

returns me

... ERROR : rc: "sync/sync": error: couldn't find method "sync/sync"
{
	"error": "couldn't find method \"sync/sync\"",
	"input": {
		"_async": true,
		"dstFs": "<path1>",
		"srcFs": "<path2>"
	},
	"path": "sync/sync",
	"status": 404
}

404

I've taken the method name from Remote Control / API and I've tried via the CLI which works well:

 rclone version
rclone v1.61.1

Do you know what I'm doing wrong? I can't seem to find examples online to follow, is there a call to Initialize I need? I tried with and without it and it doesn't seem to change the result.

Well spotted and reproduced with this tiny example:

package main

import (
	"fmt"
	"github.com/rclone/rclone/librclone/librclone"
)

func main() {
	librclone.Initialize()

	jsonArg := "{ \"srcFs\": \"./srcFolder/\",\"dstFs\": \"./destFolder\", \"_config\": { \"DryRun\": true } }"
	output, status := librclone.RPC("sync/sync", jsonArg)
	fmt.Println(status)
	fmt.Println(output)
}

The issue seems to be that the "sync/sync" command isn't loaded into the list of available calls being checked here:

which indicates that this peace of code isn't executed when librclone is initializing:

Not sure exactly what is wrong, my best idea is that librclone should somehow import this (and possible other missing commands) to make them be available in the RPC interface.

You need to do these imports (some or all - as many as you need) the more you import the bigger the binary will be which is why you get to choose as a user. So you could import just the drive backend and just the sync commands if you wanted.

        _ "github.com/rclone/rclone/backend/all"   // import all backends
        _ "github.com/rclone/rclone/cmd/cmount"    // import cmount
        _ "github.com/rclone/rclone/cmd/mount"     // import mount
        _ "github.com/rclone/rclone/cmd/mount2"    // import mount2
        _ "github.com/rclone/rclone/fs/operations" // import operations/* rc commands
        _ "github.com/rclone/rclone/fs/sync"       // import sync/*
        _ "github.com/rclone/rclone/lib/plugin"    // import plugins

@Ole - I don't think this is properly documented!

There should be a section here rclone/librclone at master · rclone/rclone · GitHub with Go and some examples in rclone/librclone/librclone at master · rclone/rclone · GitHub would mean they came out in the Go docs here librclone package - github.com/rclone/rclone/librclone/librclone - Go Packages

Thanks folks for your responses.

Here's a working example for others who may run into this:

package main
 
import "fmt"
import "github.com/rclone/rclone/librclone/librclone"
import _ "github.com/rclone/rclone/fs/sync"  
import _ "github.com/rclone/rclone/backend/all" 
 
const syncArgs = "{\"srcFs\": \"<path1>\",\"dstFs\":\"<path2>\"}"
 
func main() {
    fmt.Println("hello world")
    librclone.Initialize()
    out, status := librclone.RPC("sync/sync", syncArgs)
    fmt.Println(status)
    fmt.Println(out)
 
}
1 Like

Thanks Shyam for the example. I have a question on how to accomplish the same with an s3 like source.

With Sync.Sync, I could pass in the credentials directly.

Do we now need to :

  1. Create and register a source (say remote1) and destination (remote2).
  2. Initiate transfer
    (basically mimicking rclone cli behavior).

How would flags (ca-cert, no-check-buckets etc.) be passed into the above?

Super happy to update the docs with a pull request after we establish the right contract here.

You can do that, or you can use connection strings.

Yes!

You can make librclone.RPC("sync/sync",) return when the command has been placed in the execution queue, that is before the command execution has completed by adding _async=true. If doing this, you probably also want to create a _group and then track execution with core/stats. I made a small example here: https://forum.rclone.org/t/c-code-for-rclone-copy/33990/11

Using connection strings or _config similar to the way I pass --dry-run in my example:

Great!

I suggest we help each other building a small example using the most used options like _config, _filters, _group and core/stats on a sync of local folders. That will be a good starting point for both you and @shenv56 - and cover most of the requests I have seen on the forum.

That could then go in the source tree as an example which would be fantastic.

Here is a little example so far : Tested this in my local env.

input := "{\n  \"srcFs\": \"/path/to/folder\",\n  \"dstFs\": \":s3,env_auth=false,access_key_id=ACC_KEY_1,secret_access_key=SECRET_KEY_1,endpoint='https://my.custom.endppoint':bucket1/\",\n  \"_config\": {\n    \"InsecureSkipVerify\": true\n  }\n}"

Will add more with certs etc.

The only thing is that cert will be an array after the next release, so will have to account for that. Might be backward incompatible.

1 Like

Hi

Here is a more formal example :

{
  "srcFs": ":local,copy_links=true:/root/test-data",
  "dstFs": ":s3,env_auth=false,provider=Other,acl=private,access_key_id=ACC_KEY,secret_access_key=SECRET_ACC_KEY,endpoint='my.custom.endpoint':/bucket2",
  "_config": {
    "CaCert": "/tmp/temp-ca-file1735315905",
    "NoCheckDest": true
  }
}

I can merge it into the repository. Please point me to the right file where I should add it.
Will also add more examples with GCS later.

We need to point out how to use the Go library here

And the example should probably go in a go directory in here

Here's my example so far (I'll add filter, config soon) that I'd be happy to contribute but I have some questions in regards to async polling using job/status:

Suppose my polling interval is > 1 minute and from the docs "The job can be queried for up to 1 minute after it has finished.". Should I be exiting on both finished=True from the job/status call as well as a 404 on the jobStatus call and assume then the job is finished, since I can't check past a minute?

package main
 
import (
	"encoding/json"
	"fmt"
	"time"
 
	"github.com/rclone/rclone/librclone/librclone"
 
	_ "github.com/rclone/rclone/fs/sync"
 
	_ "github.com/rclone/rclone/backend/all"
)
 
type syncRequest struct {
	SrcFs string `json:"srcFs"`
	DstFs string `json:"dstFs"`
	Group string `json:"_group"`
	Async bool   `json:"_async"`
}
 
type syncResponse struct {
	JobID int64 `json:"jobid"`
}
 
type statusRequest struct {
	JobID int64 `json:"jobid"`
}
 
type statusResponse struct {
	Finished bool `json:"finished"`
	Success  bool `json:"success"`
}
 
type statsRequest struct {
	Group string `json:"group"`
}
 
type statsResponse struct {
	Bytes       int64   `json:"bytes"`
	Speed       float64 `json:"speed"`
	Transfers   int64   `json:"transfers"`
	ElapsedTime float64 `json:"elapsedTime"`
	Errors      int64   `json:"errors"`
}
 
func main() {
	librclone.Initialize()
	syncRequest := syncRequest{
		SrcFs: "<absolute_path>",
		DstFs: ":s3,env_auth=false,access_key_id=<access>,secret_access_key=<secret>,endpoint='<endpoint>':<bucket>",
		Group: "MyTransfer",
		Async: true,
	}
 
	syncRequestJSON, err := json.Marshal(syncRequest)
	if err != nil {
		fmt.Println(err)
	}
	out, status := librclone.RPC("sync/copy", string(syncRequestJSON))
	if status != 200 {
		fmt.Println("Error: Got status : %d and output %q", status, out)
	}
	var syncResponse syncResponse
	err = json.Unmarshal([]byte(out), &syncResponse)
	if err != nil {
		fmt.Println(err)
		return
	}
 
	fmt.Printf("Job Id of Async Job: %d\n", syncResponse.JobID)
 
	statusRequest := statusRequest{JobID: syncResponse.JobID}
	statusRequestJSON, err := json.Marshal(statusRequest)
	if err != nil {
		fmt.Println(err)
	}
	var statusResponse statusResponse
 
	statusTries := 0
	for !statusResponse.Finished {
		out, status := librclone.RPC("job/status", string(statusRequestJSON))
		fmt.Println(out)
		if status == 404 {
			break
		}
		err = json.Unmarshal([]byte(out), &statusResponse)
		if err != nil {
			fmt.Println(err)
			break
		}
		time.Sleep(10 * time.Second)
		statusTries++
		fmt.Printf("Polled status of job %d, %d times\n", statusRequest.JobID, statusTries)
	}
 
	if !statusResponse.Success {
		fmt.Println("Job finished but did not have status success.")
		return
	}
 
	statsRequest := statsRequest{Group: "MyTransfer"}
 
	statsRequestJSON, err := json.Marshal(statsRequest)
	if err != nil {
		fmt.Println(err)
	}
 
	out, _ = librclone.RPC("core/stats", string(statsRequestJSON))
	var stats statsResponse
 
	err = json.Unmarshal([]byte(out), &stats)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("Transferred %d bytes and %d files\n", stats.Bytes, stats.Transfers)
 
}```
1 Like

Great :slight_smile:

You can change the time the jobs hang around with

  --rc-job-expire-duration Duration   Expire finished async jobs older than this value (default 1m0s)

So I would assume that you are polling more often than --rc-job-expire-duration and the 404 case can be an error.

Added pull request for a basic example :

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.