LibRClone Bandwidth Not Working?

Help and Support
1

Using librclone from rclone version 1.62.1

I'm running into issues respecting the bandwidth limit set on the transfer. I'm running an object storage to object storage transfer specifically on Ceph -- and I'm getting a higher speed from the stats call than what I set in the bandwidth limit.

I'm doing for example 4 files of 1 GB each file transfer from 1 bucket to another.

What the code approximately looks like:

librclone.Initialize()
defer librclone.Finalize()

out, status := librclone.RPC("core/bwlimit", "{\"rate\": \"100M\"}")

// I'm using the default number of checkers and transfers 
transferRequest :=
"
{
  "srcFs": "somesrc",
  "dstFs": "somedst",
  "_group": "transfer",
  "_async": true, // I'll ignore pasting the polling async code for brevity.
  "_config": {
    "CaCert": [
      "ca.crt",
    ]
  }
}
"
out, status := librclone.RPC("sync/copy", transferRequest)

out, status := librclone.RPC("core/stats", "{\"Group\": \"transfer\"}")
json.Unmarshal([]byte(out), &stats)

fmt.Printf("speed: %s and elapsed time: %s", humanize.Bytes(uint64(stats.Speed))+"/s", time.Duration(stats.ElapsedTime * float64(time.Second)).String())

Logs Printed:

2023/06/14 23:30:32 NOTICE: Bandwidth limit set to {100Mi 100Mi}

speed: 477 MB/s and elapsed time: 9.975018905s

The transfer is running correctly (I'm printing more info about bytes moved and files moved etc.). And it seems to be acknowledging the bandwidth limit in a log. However, the speed of 477 megabytes per second seems a lot higher than 100 megabits per second. The time * speed here seem to match up with the 4.3 GB here I'm transferring so I believe this is correctly parsing and outputting from the rclone stats call.

Am I doing something wrong here in setting the bandwidth limit?

2

It looks like you are doing the right thing...

I tried this with rclone rcd

rclone rcd --rc-no-auth -v

Then

rclone rc core/bwlimit rate=1M

And

rclone rc operations/copyfile srcFs=/tmp srcRemote=1G dstFs=/tmp dstRemote=1G.copy

And I could see from rclone rc core/stats that the transfer was proceeding at 1MiB/s.

So the basic theory looks OK, let's try from librclone

package main

import (
	"log"
	"time"

	"github.com/rclone/rclone/librclone/librclone"

	_ "github.com/rclone/rclone/backend/all"   // import all backends
	_ "github.com/rclone/rclone/fs/operations" // import operations/* rc commands
	_ "github.com/rclone/rclone/fs/sync"       // import sync/*
)

func main() {
	librclone.Initialize()
	defer librclone.Finalize()

	out, status := librclone.RPC("core/bwlimit", "{\"rate\": \"1M\"}")
	log.Println(status)
	log.Println(out)

	out, status = librclone.RPC("operations/copyfile", `{"dstFs":"/tmp","dstRemote":"1G.copy","srcFs":"/tmp","srcRemote":"1G", "_async": true}`)
	log.Println(status)
	log.Println(out)

	for {
		time.Sleep(time.Second)
		out, status := librclone.RPC("core/stats", "")
		log.Println(status)
		log.Println(out)
	}
}

That works too.

So I think this must be a problem in your setup or your code somehow but I'm not sure where. Maybe you could try my example code above and see if that works for you using your setup and remotes?

3

Have some more information: Looks like my transfer is using a server-side copy.

I didn't know about server-side copy so I didn't know specifying my remotes being the same endpoint mattered.

My understanding is that this bandwidth is only respected in non server-side copy uploads and not respected server-side (since rclone isn't actually transferring files in that case).

This is great but is there a way I can disable server-side upload (for testing purposes) if I don't have two different endpoints to test with?

4

I'd state it slightly different as it's not using any bandwidth so there isn't anything to respect as it's all done server side.

5

If you run rclone with --disable Copy it will disable server side copy. You can do this via the rc by setting the DisableFeatures option

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.