Filters in golang

[eleon00]

What is the problem you are having with rclone?

Hello, I am trying to incorporate rclone in a golang application. I have two problems:

1. i cant seem to use the new filter once i start doing sync.copyDir. The copy works. but it copies the whole directory. I just need a couple of files through a wildcard ("*.csv")
2. after the initial copy operation from problem 1, I need to change the filter to separate two groups from the copied files (e.g. file.csv vs file_tmp.csv -- changing the filter to *tmp.csv )

from a previous forum post, it says i need to filter.Active after I setup my new filter. But it looks to be an old version and the Active is now globalConfig

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.1

• os/version: ubuntu 18.04 (64 bit)
• os/kernel: 5.10.16.3-microsoft-standard-WSL2 (x86_64)
• os/type: linux
• os/arch: amd64
• go/version: go1.17.9
• go/linking: static
• go/tags: none

yes

Which cloud storage system are you using? (eg Google Drive)

local to azureblob, sftp to azure blob

The command you were trying to run (eg rclone copy /tmp remote:tmp)

package main

import (
	"context"
	_ "embed"
	"log"
	"os"

	_ "github.com/rclone/rclone/backend/azureblob"
	_ "github.com/rclone/rclone/backend/local"
	_ "github.com/rclone/rclone/backend/sftp"
	"github.com/rclone/rclone/fs"

	"github.com/rclone/rclone/fs/config"
	"github.com/rclone/rclone/fs/config/configfile"
	"github.com/rclone/rclone/fs/filter"
	"github.com/rclone/rclone/fs/sync"
)

//go:embed rclone.conf
var rc []byte

func main() {
	ctx := context.Background()
	ci := fs.GetConfig(ctx)
	_ = filter.GetConfig(ctx)
	ci.AskPassword = false
	if err := os.WriteFile("rclone.conf", rc, 0666); err != nil {
		log.Printf("error os.WriteFile error: %v", err)
		log.Fatal(err)
	}

	_ = config.SetConfigPath("rclone.conf")
	configfile.Install()

	var indicatorsOpt = filter.Opt{
		MinAge:      fs.DurationOff,
		MaxAge:      fs.DurationOff,
		MinSize:     fs.SizeSuffix(-1),
		MaxSize:     fs.SizeSuffix(-1),
		IncludeRule: []string{"*.csv_indicators.csv"},
	}

	_, err := filter.NewFilter(&indicatorsOpt)
	if err != nil {
		log.Fatal(err.Error())
	}
	

	newCtx, _ := filter.AddConfig(ctx)

	//cSource, err := fs.NewFs(context.Background(), "sftp1:/home/outbox")
	cSource, err := fs.NewFs(newCtx, "/home/eleon00/test")
	if err != nil {
		log.Println("Source")
		log.Fatal(err)
	}

	cTarget, err := fs.NewFs(newCtx, "azuretest:/stage")
	if err != nil {
		log.Println("Target")
		log.Fatal(err)
	}

	sync.CopyDir(newCtx, cTarget, cSource, true)

	var csvOpt = filter.Opt{
		MinAge:      fs.DurationOff,
		MaxAge:      fs.DurationOff,
		MinSize:     fs.SizeSuffix(-1),
		MaxSize:     fs.SizeSuffix(-1),
		IncludeRule: []string{"*.csv"},
	}

	_, err = filter.NewFilter(&csvOpt)
	if err != nil {
		log.Fatal(err.Error())
	}
	newCtx2, _ := filter.AddConfig(newCtx)

	sync.CopyDir(newCtx2, cTarget, cSource, true)

	os.Remove("rclone.conf")
}

The rclone config contents with secrets removed.

[azuretest]
type = azureblob
account = account
key = <redacted>

[sftp1]
type = sftp
host = host
user = user
key_pem = <redacted>

[sftp2]
type = sftp
host = host
user = user
key_pem = <redacted>

A log from the command with the -vv flag

Paste  log here

[ncw (Nick Craig-Wood)]

You want to swap out the filter in the context like this

	fi, err := filter.NewFilter(&indicatorsOpt)
	if err != nil {
		log.Fatal(err.Error())
	}

        // Change the active filter
        newCtx := filter.ReplaceConfig(ctx, fi)

[eleon00]

THank you! works perfectly!