Use Rclone golang to transfer files

You need to do these imports (some or all - as many as you need) the more you import the bigger the binary will be which is why you get to choose as a user. So you could import just the drive backend and just the sync commands if you wanted.

        _ "github.com/rclone/rclone/backend/all"   // import all backends
        _ "github.com/rclone/rclone/cmd/cmount"    // import cmount
        _ "github.com/rclone/rclone/cmd/mount"     // import mount
        _ "github.com/rclone/rclone/cmd/mount2"    // import mount2
        _ "github.com/rclone/rclone/fs/operations" // import operations/* rc commands
        _ "github.com/rclone/rclone/fs/sync"       // import sync/*
        _ "github.com/rclone/rclone/lib/plugin"    // import plugins

@Ole - I don't think this is properly documented!

There should be a section here rclone/librclone at master · rclone/rclone · GitHub with Go and some examples in rclone/librclone/librclone at master · rclone/rclone · GitHub would mean they came out in the Go docs here librclone package - github.com/rclone/rclone/librclone/librclone - Go Packages

Thanks folks for your responses.

Here's a working example for others who may run into this:

package main
 
import "fmt"
import "github.com/rclone/rclone/librclone/librclone"
import _ "github.com/rclone/rclone/fs/sync"  
import _ "github.com/rclone/rclone/backend/all" 
 
const syncArgs = "{\"srcFs\": \"<path1>\",\"dstFs\":\"<path2>\"}"
 
func main() {
    fmt.Println("hello world")
    librclone.Initialize()
    out, status := librclone.RPC("sync/sync", syncArgs)
    fmt.Println(status)
    fmt.Println(out)
 
}
1 Like

Thanks Shyam for the example. I have a question on how to accomplish the same with an s3 like source.

With Sync.Sync, I could pass in the credentials directly.

Do we now need to :

  1. Create and register a source (say remote1) and destination (remote2).
  2. Initiate transfer
    (basically mimicking rclone cli behavior).

How would flags (ca-cert, no-check-buckets etc.) be passed into the above?

Super happy to update the docs with a pull request after we establish the right contract here.

You can do that, or you can use connection strings.

Yes!

You can make librclone.RPC("sync/sync",) return when the command has been placed in the execution queue, that is before the command execution has completed by adding _async=true. If doing this, you probably also want to create a _group and then track execution with core/stats. I made a small example here: https://forum.rclone.org/t/c-code-for-rclone-copy/33990/11

Using connection strings or _config similar to the way I pass --dry-run in my example:

Great!

I suggest we help each other building a small example using the most used options like _config, _filters, _group and core/stats on a sync of local folders. That will be a good starting point for both you and @shenv56 - and cover most of the requests I have seen on the forum.

That could then go in the source tree as an example which would be fantastic.

Here is a little example so far : Tested this in my local env.

input := "{\n  \"srcFs\": \"/path/to/folder\",\n  \"dstFs\": \":s3,env_auth=false,access_key_id=ACC_KEY_1,secret_access_key=SECRET_KEY_1,endpoint='https://my.custom.endppoint':bucket1/\",\n  \"_config\": {\n    \"InsecureSkipVerify\": true\n  }\n}"

Will add more with certs etc.

The only thing is that cert will be an array after the next release, so will have to account for that. Might be backward incompatible.

1 Like

Hi

Here is a more formal example :

{
  "srcFs": ":local,copy_links=true:/root/test-data",
  "dstFs": ":s3,env_auth=false,provider=Other,acl=private,access_key_id=ACC_KEY,secret_access_key=SECRET_ACC_KEY,endpoint='my.custom.endpoint':/bucket2",
  "_config": {
    "CaCert": "/tmp/temp-ca-file1735315905",
    "NoCheckDest": true
  }
}

I can merge it into the repository. Please point me to the right file where I should add it.
Will also add more examples with GCS later.

We need to point out how to use the Go library here

And the example should probably go in a go directory in here

Here's my example so far (I'll add filter, config soon) that I'd be happy to contribute but I have some questions in regards to async polling using job/status:

Suppose my polling interval is > 1 minute and from the docs "The job can be queried for up to 1 minute after it has finished.". Should I be exiting on both finished=True from the job/status call as well as a 404 on the jobStatus call and assume then the job is finished, since I can't check past a minute?

package main
 
import (
	"encoding/json"
	"fmt"
	"time"
 
	"github.com/rclone/rclone/librclone/librclone"
 
	_ "github.com/rclone/rclone/fs/sync"
 
	_ "github.com/rclone/rclone/backend/all"
)
 
type syncRequest struct {
	SrcFs string `json:"srcFs"`
	DstFs string `json:"dstFs"`
	Group string `json:"_group"`
	Async bool   `json:"_async"`
}
 
type syncResponse struct {
	JobID int64 `json:"jobid"`
}
 
type statusRequest struct {
	JobID int64 `json:"jobid"`
}
 
type statusResponse struct {
	Finished bool `json:"finished"`
	Success  bool `json:"success"`
}
 
type statsRequest struct {
	Group string `json:"group"`
}
 
type statsResponse struct {
	Bytes       int64   `json:"bytes"`
	Speed       float64 `json:"speed"`
	Transfers   int64   `json:"transfers"`
	ElapsedTime float64 `json:"elapsedTime"`
	Errors      int64   `json:"errors"`
}
 
func main() {
	librclone.Initialize()
	syncRequest := syncRequest{
		SrcFs: "<absolute_path>",
		DstFs: ":s3,env_auth=false,access_key_id=<access>,secret_access_key=<secret>,endpoint='<endpoint>':<bucket>",
		Group: "MyTransfer",
		Async: true,
	}
 
	syncRequestJSON, err := json.Marshal(syncRequest)
	if err != nil {
		fmt.Println(err)
	}
	out, status := librclone.RPC("sync/copy", string(syncRequestJSON))
	if status != 200 {
		fmt.Println("Error: Got status : %d and output %q", status, out)
	}
	var syncResponse syncResponse
	err = json.Unmarshal([]byte(out), &syncResponse)
	if err != nil {
		fmt.Println(err)
		return
	}
 
	fmt.Printf("Job Id of Async Job: %d\n", syncResponse.JobID)
 
	statusRequest := statusRequest{JobID: syncResponse.JobID}
	statusRequestJSON, err := json.Marshal(statusRequest)
	if err != nil {
		fmt.Println(err)
	}
	var statusResponse statusResponse
 
	statusTries := 0
	for !statusResponse.Finished {
		out, status := librclone.RPC("job/status", string(statusRequestJSON))
		fmt.Println(out)
		if status == 404 {
			break
		}
		err = json.Unmarshal([]byte(out), &statusResponse)
		if err != nil {
			fmt.Println(err)
			break
		}
		time.Sleep(10 * time.Second)
		statusTries++
		fmt.Printf("Polled status of job %d, %d times\n", statusRequest.JobID, statusTries)
	}
 
	if !statusResponse.Success {
		fmt.Println("Job finished but did not have status success.")
		return
	}
 
	statsRequest := statsRequest{Group: "MyTransfer"}
 
	statsRequestJSON, err := json.Marshal(statsRequest)
	if err != nil {
		fmt.Println(err)
	}
 
	out, _ = librclone.RPC("core/stats", string(statsRequestJSON))
	var stats statsResponse
 
	err = json.Unmarshal([]byte(out), &stats)
	if err != nil {
		fmt.Println(err)
	}
	fmt.Printf("Transferred %d bytes and %d files\n", stats.Bytes, stats.Transfers)
 
}```
1 Like

Great :slight_smile:

You can change the time the jobs hang around with

  --rc-job-expire-duration Duration   Expire finished async jobs older than this value (default 1m0s)

So I would assume that you are polling more often than --rc-job-expire-duration and the 404 case can be an error.

Added pull request for a basic example :

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.