Rclone using lots of download bandwidth when uploading

Help and Support
1

I’m using something like this to backup a directory to backlaze b2:-1:

rclone -v copy /mnt/shared1/samba/photos/ GrintonPhotosCrypt:/

The filenames are not encrypted but the contents are. I’m using the latest version of rclone 1.44.

For some reason its using 5x as downloading bandwidth when all I’m doing is uploading files. Here’s some vnstat output just for today:-

     today     38.44 GiB  /    7.46 GiB

Does anyone know why it should be downloading so much when all I’m trying to do is upload?

Many thanks

2

Why do you think it’s using bandwidth when you are uploading?

3

No idea why its using 5x as much download bandwidth as upload bandwidth when I’m only uploading stuff?

4

What backend are you using?

5

I’m using backblaze b2

6

That’s a local disk it is reading from to do the upload?

Can you use something like iftop to see if it’s actually that process doing the download?

7

I’ve done that and can confirm it is.

8

Is it on the initial copy that causes the issue ? I don’t use backblaze unfortunately so I wonder if it is checksumming or something.

You could try running it with -vvv and see what the debug looks like to get an idea of what it is doing and share that.

9

This is a run after the initial copy. The initial copy was ok. I’ll give the -vvv flag a try and report back. Many thanks :+1:.

10

The -vvv flag reports back a lot more information but no errors. I think what’s happening is as it cycles through each file it downloads a certain amount of information to do some checking. If a batch of files have already been backed up its going to process them a lot more quickly and therefore download a lot more information within a given time period as a result. I think that is the cause of the problem.

11

I’m not sure how backblaze works with checking the file to see if it is the same. It should just check a hash and not the whole file but I’m not 100% sure for this backend if that’s the case.

@ncw - is that the case?

12

Absolutely. It should only be downloading directory listings and hashes from b2.

Could the download have happened earlier in the day - those stats are cumulative over the day I think?

Try (replace eno1 with the name of your network interface) to see real time bandwidth

vnstat --live -i eno1

And compare with the stats from rclone with --stats 10s

13

Apologies for the delay. If I run rclone on a folder which has already been backed up eg:-

rclone -v --stats 10s copy /mnt/shared1/samba/photos/ GrintonPhotosCrypt:/

Then I get

`2018/11/27 22:31:01 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 18902 / 18902, 100%
Transferred: 0 / 0, -
Elapsed time: 12s

2018/11/27 22:31:11 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 54417 / 54417, 100%
Transferred: 0 / 0, -
Elapsed time: 22s

2018/11/27 22:31:21 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 80786 / 80786, 100%
Transferred: 0 / 0, -
Elapsed time: 32s

2018/11/27 22:31:27 INFO : Encrypted drive ‘GrintonPhotosCrypt:/’: Waiting for checks to finish
2018/11/27 22:31:27 INFO : Encrypted drive ‘GrintonPhotosCrypt:/’: Waiting for transfers to finish
2018/11/27 22:31:27 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 104164 / 104164, 100%
Transferred: 0 / 0, -
Elapsed time: 38.4s`

If I run:-

vnstat --live -i eno2

at the same time as rclone is running I get :-

rx: 13.25 Mbit/s 1192 p/s tx: 862 kbit/s 1120 p/s

after rclone has run vnstat goes back to normal:-

rx: 45 kbit/s 24 p/s tx: 41 kbit/s 25 p/s

14

That is rclone reading directory listings from the remote. You’ve got 100k files so the directory listings for those will be quite big.

15

I suspected as much. Many thanks for that.

1 Like