How to get transfer progress from `accounting.Stats`?

I'm writing a small wrapper for some rclone commands, and I'm trying to get the progress of the commands, but can't seem to get anything from accounting.Stats.

I've looked at the copy and copyto commands for inspiration - here's the code I'm running:

package main

import (
	"wrapper"

	"github.com/ncw/rclone/fs"
	"github.com/ncw/rclone/fs/accounting"
	"github.com/ncw/rclone/fs/config"
)

func main() {
	// Load the config
	config.ConfigPath = "/path/to/rclone.conf"
	config.LoadConfig()

	// Output stats on one line
	fs.Config.StatsOneLine = true
	fs.Config.Progress = true

	// Create a channel to log progress
	logChannel := make(chan bool)
	go func() {
		for {
			select {
			case <-logChannel:
				return
			default:
				// Log rclone transfer progress
				// FIXME: this is broken? just logs zeroes...
				fmt.Print(fmt.Sprintf("\r %s", strings.TrimSpace(accounting.Stats.String())))
			}
			time.Sleep(time.Second / 4)
		}
	}()

	filePath := "b2:bucket/path/to/file"
	wrapper.CopyObject(filePath, "./foo")
	
	// Close stats channel
	logChannel <- true
}

And the wrapper package is defined as follows:

package wrapper

import (
	"github.com/ncw/rclone/cmd"
	"github.com/ncw/rclone/fs"
	"github.com/ncw/rclone/fs/operations"
	"github.com/ncw/rclone/fs/sync"
)

func CopyObject(src, dst string) error {
	fsrc, srcFileName, fdst := cmd.NewFsSrcFileDst([]string{src, dst})
	if srcFileName == "" {
		createEmptySrcDirs := false
		return sync.CopyDir(fdst, fsrc, createEmptySrcDirs)
	}
	return operations.CopyFile(fdst, fsrc, srcFileName, srcFileName)
}

The only output I get from this is:

0 / 0 Bytes, -, 0 Bytes/s, ETA -

I can't seem to get output at all from calling accounting.Stats.String(), and as far as I can tell from looking through the source code I'm doing the same thing that the command rclone copyto src dst --progress --stats-one-line is doing.

I'm probably missing something obvious, or doing something incorrectly.
Does anyone know how I can get the progress of active transfers?

Bonus Points: if someone could tell me how to get the progress of a specific transfer that would be even better.

That looks like the correct output if you have set StatsOneLine.

Are you saying it doesn't change when you do transfers?

That’s right - even without StatsOneLine the output doesn’t change when I am making transfers - it’s always 0. :confused:

I moved all the code into one file, you can see it here:

package main

import (
	"fmt"
	"os"
	"strings"
	"time"

	_ "github.com/ncw/rclone/backend/all" // import all rclone backends
	"github.com/ncw/rclone/cmd"
	"github.com/ncw/rclone/fs"
	"github.com/ncw/rclone/fs/accounting"
	"github.com/ncw/rclone/fs/config"
	"github.com/ncw/rclone/fs/operations"
	"github.com/ncw/rclone/fs/sync"
)

func main() {
	// Load the config
	os.Setenv("RCLONE_CONFIG_PASS", "<CONFIG_PASSWORD>")
	config.ConfigPath = "/path/to/rclone.conf"
	config.LoadConfig()
	os.Unsetenv("RCLONE_CONFIG_PASS")

	// Output stats on one line
	fs.Config.StatsOneLine = true
	fs.Config.Progress = true

	// Create a channel to log progress
	logChannel := make(chan bool)
	go func() {
		for {
			select {
			case <-logChannel:
				fmt.Println("")
				return
			default:
				// Log rclone transfer progress
				// FIXME: this is broken? just logs zeroes...
				fmt.Println(strings.TrimSpace(accounting.Stats.String()))
			}
			time.Sleep(time.Second / 4)
		}
	}()

	fmt.Println("BEFORE COPY")

	filePath := "remote:bucket/path/to/file"
	CopyObject(filePath, "./foo")
	logChannel <- true

	fmt.Println("AFTER COPY")
}

func CopyObject(src, dst string) error {
	fsrc, srcFileName, fdst := cmd.NewFsSrcFileDst([]string{src, dst})
	if srcFileName == "" {
		createEmptySrcDirs := false
		return sync.CopyDir(fdst, fsrc, createEmptySrcDirs)
	}
	return operations.CopyFile(fdst, fsrc, srcFileName, srcFileName)
}

And when I run it, this is the output:

go run main.go
BEFORE COPY
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
0 / 0 Bytes, -, 0 Bytes/s, ETA -
AFTER COPY

Afterwards, the file is correctly copied.
Am I right in assuming that the stats should update while a transfer is in progress?

They should, yes.

There is probably some bit of initialization missing. I spent a few minutes trying to track down what exactly but I came up blank!

Rclone initialisation is done in cmd/cmd.go so it should be all in there.

So I made it continue to log after the transfer had completed as well, and we do get some output:

BEFORE COPY
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
         0 / 0 Bytes, -, 0 Bytes/s, ETA -
AFTER COPY
    4.819k / 4.819 kBytes, 100%, 1.578 kBytes/s, ETA 0s
    4.819k / 4.819 kBytes, 100%, 1.456 kBytes/s, ETA 0s
    4.819k / 4.819 kBytes, 100%, 1.353 kBytes/s, ETA 0s
    4.819k / 4.819 kBytes, 100%, 1.263 kBytes/s, ETA 0s

So it looks like the stats only update after a transfer has completed, and not during the transfer itself.
I feel like this is incorrect though, since the transfer has completed but it's still showing progress (albeit the ETA is 0).


Meanwhile, I've found that using accounting.Stats.RemoteStats(...) gives me slightly more information, so I'm going to go ahead and try to use that for now.

OK!

Just on a general point - I wonder why you are using rclone as a library? It isn't really designed to be used as a library. The supported way of doing what you want to do would be using the API.

I’m attempting to make an iOS client for rclone (which I hear is something that people want but doesn’t exist at the moment!).

So I’m wrapping the desired functionality into a single package, for which I can create bindings (and thus an iOS framework).
I have to do this since iOS apps can’t spawn or include other binaries.

If you have any suggestions on how I might do this better do let me know :blush:

I see! You approach sounds reasonable, just bear in mind that I break the internal API of rclone quite regularly!

Haha! I figured as much.

Right now I’m getting the basic operations working, and so far everything seems to be possible.

Thanks for your time. :smiley:

1 Like

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.