Use Rclone golang to transfer files

What is the problem you are having with rclone?

Trying to use the golang version to move files from one s3 like system to another, using custom certificate.

Run the command 'rclone version' and share the full output of the command.

No

Which cloud storage system are you using? (eg Google Drive)

Amazon S3

The command you were trying to run (eg rclone copy /tmp remote:tmp)

//Using Go Library
	err = sync.Sync(ctx, dstFs, srcFs, false)


The rclone config contents with secrets removed.

NA

A log from the command with the -vv flag

NA

Hi Alankrit,

I suggest you take a look at the golang interface provided in librclone:

and perhaps first try playing with the rclone API to get acquainted with the possibilities:

Hi Ole

Thanks for the response.

My exact aim is to accomplish a transfer between 2 s3-like systems, using a custom ca certificate. Going through the above links, I did not fully grasp how I could accomplish the same with golang. Could you help me with the same by pointing to the right way to use the library ?

Without ca-crt, i can accomplish the transfer by implementing the 'Storage' interface.

Sounds like you are relatively new to rclone, so I suggest we first try doing a simple move using the rclone command line interface. It is a lot easier to understand and explain, and once we have a working example it is a lot easier to do the same using the (golang) API.

Are you able to configure both your remotes using this S3 guide and then list the content of a bucket something like this:

rclone lsl yourS3:yourBucket

If successful, then try a simple move something like this:

rclone move yourS3source:yourSourceBucket/some/test/folder/somefile yourS3target:yourTargetBucket/some/test/folder

So the command i need is the equivalent of :

rclone sync sg://bucket1 sg://bucket2

where sg is my configured source.

However I need
I am able to do the same with a ca certificate

So I am using
rclone sync sg://bucket1 sg://bucket2 --ca-cert ca.crt

Which is working fine.

The env i need to use, doesn't really allow for command line access. I am trying to trigger the sync internally via golang.

Currently I am using `sync.Sync(ctx, dstFs, srcFs, false), which triggers Sync method in sync package. Wondering if the sync method can be triggered with a custom ca cert.

You can pass new config in the ctx passed to the sync. See the fs.AddConfig function.

You can do this via librclone too - see _config in the rc docs

Thanks. I was able to use the fs.AddConfig function to achieve the same. I had misunderstood the utility of the function earlier.

I was wondering if instead of using ca crt file, if it is possible to have the certificate used as is/as a string/as a byte array. I couldn't figure the same out from the code.

For the file, the method I am using is :

  1. os.CreateTemp to create a certificate file
  2. Use with sync.Sync
  3. defer os.Remove(file.Name())

Would you recommend any other approach? I am not sure if creating a temp file would be possible in all the env's my system is deployed in (containers).

Not currently.

It would be possible to change the certificate reading code to maybe expect the certificate instead of a file name if it started with @inline@ or something like that?

Thanks. Let me check that. Also wanted to confirm that it's not possible to have separate ca certs for source and destination right? It seems like they're using the same ca cert, even if one is defined and the other is not.

I'm not sure I understand what you mean here. Do you mean using a client side certificate? If so you should be able to have a separate file for that.

Yeah sorry for the lack of clarity. What I meant was it it possible to define the source and destination certificates separately.

Something like :

rclone sync sg://bucket1 sg://bucket2 --src-ca-cert src-ca.crt --dst-ca-cert dst-ca.crt

Is it currently possible to define something like that ?

Thanks for explaining.

That isn't possible at the moment.

Though I think if you concatenate the certs into a single file then you can put them in a single --ca-cert flag. Would that work for you?

It would also be possible to change --ca-cert into a multiple flag so it can be repeated. Currently it will just take the first or the last.

I created a Draft pull request - fs:Added multiple ca certificate support. by alankritkharbanda · Pull Request #6688 · rclone/rclone · GitHub

Do you think this would help in supporting multiple certificates?

I think your pull request looks perfectly reasonable :slight_smile:

I'll give it a quick review now.

Hi

I updated the pull request with the comments and tested the same on my local system as well :slight_smile:

1 Like

Thank you for merging the same !
Just wanted to check - when is the next release due ? Wanted to ensure we import the next release in our codebase.

I don't think I set a release date yet, but rclone releases usually come out with a cadence of 2 months.

Hi

I had a question. Currently I am using sync.Sync, but seems like that is not the proper way to use the library and instead RPC should be used instead.

I was trying to do the following :

rclone sync s3-like-system-1://bucket1 s3-like-system-2://bucket2 --ca-cert ca-1.crt --ca-cert ca-2.crt

For which I would have needed something like :

librclone.RPC("sync", "{srcFs : {...}, dstFs : {...}}"

However I understand that sync or operations/sync is not a directly usable option. Could you kindly help me with an example to trigger RPC sync here?

Running into a somewhat similar issue:

From my go.mod file:
require GitHub - rclone/rclone: "rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Yandex Files v1.61.1

From my go code:

status, out := librclone.RPC("sync/sync", syncArgs)
fmt.Println(status)
fmt.Println(out)

returns me

... ERROR : rc: "sync/sync": error: couldn't find method "sync/sync"
{
	"error": "couldn't find method \"sync/sync\"",
	"input": {
		"_async": true,
		"dstFs": "<path1>",
		"srcFs": "<path2>"
	},
	"path": "sync/sync",
	"status": 404
}

404

I've taken the method name from Remote Control / API and I've tried via the CLI which works well:

 rclone version
rclone v1.61.1

Do you know what I'm doing wrong? I can't seem to find examples online to follow, is there a call to Initialize I need? I tried with and without it and it doesn't seem to change the result.

Well spotted and reproduced with this tiny example:

package main

import (
	"fmt"
	"github.com/rclone/rclone/librclone/librclone"
)

func main() {
	librclone.Initialize()

	jsonArg := "{ \"srcFs\": \"./srcFolder/\",\"dstFs\": \"./destFolder\", \"_config\": { \"DryRun\": true } }"
	output, status := librclone.RPC("sync/sync", jsonArg)
	fmt.Println(status)
	fmt.Println(output)
}

The issue seems to be that the "sync/sync" command isn't loaded into the list of available calls being checked here:

which indicates that this peace of code isn't executed when librclone is initializing:

Not sure exactly what is wrong, my best idea is that librclone should somehow import this (and possible other missing commands) to make them be available in the RPC interface.