Need example RClone copy using golang

Dev Discussions
1

Hi,

I already read the golang docs about rclone. but still clueless about the implementation as the title said. need help the sample rclone copy using golang script.

Thanks.

2

You want to call rclone from go?

You could call it as an external command using the exec module.

That would be easy.

3

Hi Nick,

Yes sure I want to call rclone from go. Exec command is great, but it had often got permission issue.

Thanks

4

What exactly do you want to do? Just copy a file? Or control the whole of rclone?

5

The idea is that we had our own application which we need rclone to perform such as copy. but on our side need to avoid using command Exec inorder to not installing them on our server for better maintenance.

6

besides all of our application which we used rclone are dockerized. so its not a good idea to use cmd exec in golang script.

7

There are several levels you can hook into rclone

If you don't mind using the rclone config system then it is quite easy

package main

import (
        "context"
        "fmt"
        "log"

        //_ "github.com/rclone/rclone/backend/drive"
        _ "github.com/rclone/rclone/backend/all"
        "github.com/rclone/rclone/fs"
)

func main() {
        f, err := fs.NewFs("drive:")
        if err != nil {
                log.Fatal(err)
        }
        entries, err := f.List(context.Background(), "")
        if err != nil {
                log.Fatal(err)
        }
        fmt.Println(entries)
        for i, o := range entries {
                fmt.Printf("object[%i] = (%T) %v\n", i, o, o)
        }
}
8

Hi nick,

Actually what we need is only the copy script in golang which it can also include the file what we want to search as well. the rest like listing that you are pointed out is no problem from us.

Thanks

9

If you want to copy a file then you want operations.Copy. You'll need to make an Fs for the source and dest, find the object you wish to copy then call operations.Copy

10

Then how about like include that only wildcards like example "*.csv" to copy then?

11

You can set up the Filter or filter the objects yourself from the directory listings.

12

we did the code like this:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/rclone/rclone/fs"
	"github.com/rclone/rclone/fs/filter"
)

func main() {
	var DefaultOpt = filter.Opt{
		MinAge:      fs.DurationOff,
		MaxAge:      fs.DurationOff,
		MinSize:     fs.SizeSuffix(-1),
		MaxSize:     fs.SizeSuffix(-1),
		IncludeRule: []string{"*.csv"},
	}
	_, err := filter.NewFilter(&DefaultOpt)
	if err != nil {
		log.Fatal(err.Error())
	}
	fsource, err := fs.NewFs("henrio test:")
	if err != nil {
		log.Fatal(err)
	}
	entries, err := fsource.List(context.Background(), "")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(entries)
}

it turns out the message like this :

2020/09/17 22:52:13 didn't find section in config file

so how I supposed to do? we are not just starting the copy yet, but we are looking the files that are matched with included wildcards?

13

Did you run rclone config to make a remote called "henrio test" ?

That is what the error means I think.

14

actually yes. I did the config run

15

Got a new error after I change the default config to custom config file. the current config file looks like this :

[henrio test]
type = drive
scope = drive
root_folder_id = root_folder_id
token = {"access_token":"access_token","token_type":"Bearer","refresh_token":"refresh_token","expiry":"2020-09-17T12:05:22.621859978+07:00"}
client_id = clientid
client_secret = secret

this is the updated code :

package main

import (
	"log"

	"github.com/rclone/rclone/fs"
	"github.com/rclone/rclone/fs/config"
	"github.com/rclone/rclone/fs/filter"
)

func main() {
	var DefaultOpt = filter.Opt{
		MinAge:      fs.DurationOff,
		MaxAge:      fs.DurationOff,
		MinSize:     fs.SizeSuffix(-1),
		MaxSize:     fs.SizeSuffix(-1),
		IncludeRule: []string{"*.csv"},
	}
	_, err := filter.NewFilter(&DefaultOpt)
	if err != nil {
		log.Fatal(err.Error())
	}
	config.ConfigPath = "/home/henrio/.myconfig"
	res, err := fs.NewFs("henrio test:")
	if err != nil {
		log.Fatal(err)
	}
	panic(res)
	
}

the message are like this :

2020/09/18 07:55:51 didn't find backend called "drive"

16

You need to import the backends you want - see the example above. So if you just want the drive backend and the local backend

import (
        _ "github.com/rclone/rclone/backend/drive"
        _ "github.com/rclone/rclone/backend/local"
)
17

The error is not shown anymore. thank you. but when I want to include only wildcard *.xls instead of *.csv to copy, the *.csv file also got copied. what should I do next?

package main

import (
	"context"
	"fmt"
	"log"

	_ "github.com/rclone/rclone/backend/drive"
	_ "github.com/rclone/rclone/backend/local"
	"github.com/rclone/rclone/fs"
	"github.com/rclone/rclone/fs/config"
	"github.com/rclone/rclone/fs/filter"
	"github.com/rclone/rclone/fs/sync"
)

func main() {
	var DefaultOpt = filter.Opt{
		MinAge:      fs.DurationOff,
		MaxAge:      fs.DurationOff,
		MinSize:     fs.SizeSuffix(-1),
		MaxSize:     fs.SizeSuffix(-1),
		IncludeRule: []string{"*.xls"},
	}
	_, err := filter.NewFilter(&DefaultOpt)
	if err != nil {
		log.Fatal(err.Error())
	}
	config.ConfigPath = "/home/henrio/.myconfig"
	fsource, err := fs.NewFs("henrio test:")
	if err != nil {
		log.Fatal(err)
	}
	fdest, err := fs.NewFs("/home/henrio/datas")
	if err != nil {
		log.Fatal(err)
	}
	entries, err := fsource.List(context.Background(), "")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(entries)
	sync.CopyDir(context.Background(), fdest, fsource, true)
}
18

You need to set the global filter filter.Active to the result of filter.NewFilter(&DefaultOpt)

1 Like
19

okay. thank you the copy it works really well.

20

Great! Make sure you either vendor rclone or use go modules to pin the version as the backend interface can change.