Rclone upload exceeding bwlimit

What is the problem you are having with rclone?

Rclone upload actual bandwidth usage nearly double bwlimit

What is your rclone version (output from rclone version)

root@Highlander:~# rclone version
rclone v1.55.1

  • os/type: linux
  • os/arch: amd64
  • go/version: go1.16.3
  • go/linking: static
  • go/tags: none

Which OS you are using and how many bits (eg Windows 7, 64 bit)

UnRaid 6.92

Which cloud storage system are you using? (eg Google Drive)

GD

The command you were trying to run (eg rclone copy /tmp remote:tmp)

I use a script to control a lot of the variables - for speed I've only entered the upload ones

rclone $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount \
	--user-agent="$RcloneUploadRemoteName" \
	-vv \
	--buffer-size 512M \
	--drive-chunk-size 512M \
	--tpslimit 8 \
	--checkers 8 \
	--transfers 4 \
	--order-by modtime,$ModSort \
	--min-age $MinimumAge \
	--exclude *fuse_hidden* \
	--exclude *_HIDDEN \
	--exclude .recycle** \
	--exclude .Recycle.Bin/** \
	--exclude *.backup~* \
	--exclude *.partial~* \
	--drive-stop-on-upload-limit \
	--bwlimit 6M \
	--bind=192.168.1.78 $DeleteEmpty

The rclone config contents with secrets removed.

[tdrive]
type = drive
scope = drive
service_account_file = /mnt/user/appdata/other/rclone/service_accounts/sa_tdrive_new.json
team_drive = xxxxxxxxxxxxxxxxx
server_side_across_configs = true

[tdrive_vfs]
type = crypt
remote = tdrive:crypt
filename_encryption = standard
directory_name_encryption = true
password = xxxxxxxxxxxxx
password2 = xxxxxxxxxxxxxxxxx

A log from the command with the -vv flag

2021/06/16 08:12:59 INFO :
Transferred: 118.307G / 12.229 TBytes, 1%, 4.269 MBytes/s, ETA 4w6d10h28m2s
Checks: 168 / 172, 98%
Deleted: 84 (files), 0 (dirs)
Renamed: 84
Transferred: 84 / 3937, 2%
Elapsed time: 7h53m0.7s
Checking:

Transferring:
* unfiled/movies/lists/T…] xxxxxxxx.mkv: 94% /6.058G, 1.295M/s, 4m22s
* unfiled/movies/lists/T…xxxxxxxxxxxmkv: 12% /17.816G, 1.115M/s, 3h58m3s
* unfiled/movies/lists/L… xxxxxxxxxx.mkv: 57% /3.430G, 1.073M/s, 23m14s
* unfiled/movies/lists/T…] xxxxxxxxxxx.mkv: 53% /3.547G, 1.070M/s, 26m19s

In my logs, I can see rclone upload adhering to my bwlimt schedule e.g. as I post the limit is "6M" and the upload is staying roughly within this. However, when I check my router, this particular instance (192.168.1.78) is using between 95-105Mbps or 11-13MB/s, which is almost my whole upload speed. This is causing issues for the rest of my network traffic.

I'm pretty sure this is a recent thing as when I last checked my router, rclone uploads bwlimit applied to not just the upload component but the "overhead" as well.

Has something changed, or is there a way for the bwlimit to apply to the whole connection, not just the bwlimit? I can do this at my router end and put a hard cap on, but I'd rather do it within rclone as it's more flexible/easier to do.

Thanks in advance

I'm not aware offhand of anything that changed. I thought the bwlimit did pretty well overall, but I did think there are situations were it can exceed based on directory listings and some other things going on while it levels out.

Is that one IP just rclone traffic and nothing else is using it? While the upload is going on, is there any o the rclone traffic or other applications on that same IP that could make up the difference?

I tend to traffic shape on pfSense as it tends to work better.

The traffic on that IP is just an rclone upload job - my rclone mounts, other dockers, VMs etc have other IP addresses.

How do you traffic shape the rclone upload traffic on pfSense? I tried applying some limiters/schedules this morning to the upload job, but because it's a virtual IP address it didn't work.

Been a bit since I've used pfSense, but if I recall properly, traffic shaping was done by setting up floating rules and on those you can match the source ip's and tag them properly.

pfSense didn't well for me so I've been using OPNsense as the traffic shaping is a bit easier and just works for me.

With recent rclone's (like 1.55.1) the bwlimit applies to the whole connection.

Could you have multiple copies of rclone running? The bwlimits are independent between them.

I've got multiple mounts and upload scripts, but I've bound them all to unique IP addresses.

My bwlimit switched to 3MB/s at 2pm and things have improved with the "overhead" now just under 3MB/s with the total upload bandwidth used running at 45-50Mbps. That's still not good though as I really want to cap the total transfer to 3MB/s.

Could the problem be because my the mount is fairly big and there's 12TB to be uploaded, so there's a lot of checking going on?

root@Highlander: rclone size tdrive_vfs:
Total objects: 38268
Total size: 102.337 TBytes (112520440542627 Bytes)

In recent rclone's the overhead should be limited with the uploads to a total of 3MB/s. Maybe that "edge" limiting isn't working for some reason. It sounds like the core limiting the uploads to 3MB/s is working.

Maybe you could use tcpdump/wireshark to get the IP addresses being communicated with - that might give us some idea of what the traffic is.

I did a quick packetcapture:

16:02:12.948962 IP 142.250.179.234.443 > 192.168.1.78.54823: tcp 0
16:02:12.949001 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.949034 IP 216.58.212.202.443 > 192.168.1.78.35941: tcp 0
16:02:12.949059 IP 142.250.179.234.443 > 192.168.1.78.54823: tcp 0
16:02:12.949125 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 1418
16:02:12.949171 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 1418
16:02:12.949195 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 1418
16:02:12.949211 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 1418
16:02:12.949318 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.949978 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 1418
16:02:12.950283 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.950307 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.950612 IP 142.250.179.234.443 > 192.168.1.78.54823: tcp 0
16:02:12.950802 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 1418
16:02:12.950832 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 1418
16:02:12.950856 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 1418
16:02:12.950872 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 1418
16:02:12.950882 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.950893 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.951329 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.951663 IP 192.168.1.78.44145 > 216.58.212.234.443: tcp 808
16:02:12.951700 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.951721 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.952011 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.952035 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.952359 IP 142.250.179.234.443 > 192.168.1.78.54823: tcp 0
16:02:12.952366 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.952397 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.952421 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 1418
16:02:12.952436 IP 192.168.1.78.52973 > 216.58.212.202.443: tcp 808
16:02:12.952521 IP 142.250.179.234.443 > 192.168.1.78.54823: tcp 0
16:02:12.952657 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.952749 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.952830 IP 216.58.212.234.443 > 192.168.1.78.44145: tcp 0
16:02:12.952864 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.952902 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.953148 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.953175 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.953710 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.953740 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.954291 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.954313 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.954582 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.954605 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.954852 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.955040 IP 216.58.212.234.443 > 192.168.1.78.44145: tcp 0
16:02:12.955159 IP 216.58.212.234.443 > 192.168.1.78.44145: tcp 0
16:02:12.955487 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 1418
16:02:12.955515 IP 192.168.1.78.34013 > 142.250.200.10.443: tcp 808
16:02:12.955533 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.955546 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.955639 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.956246 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.956414 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 1418
16:02:12.956420 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.956467 IP 192.168.1.78.47713 > 216.58.212.234.443: tcp 808
16:02:12.956521 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.956771 IP 216.58.212.234.443 > 192.168.1.78.44145: tcp 0
16:02:12.956913 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.956952 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.957492 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.957518 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.957836 IP 216.58.212.234.443 > 192.168.1.78.44145: tcp 0
16:02:12.957873 IP 216.58.212.202.443 > 192.168.1.78.52973: tcp 0
16:02:12.957887 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.957944 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.957982 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.958275 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.958281 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.958307 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.958749 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.958834 IP 216.58.212.234.443 > 192.168.1.78.47713: tcp 0
16:02:12.958841 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.958888 IP 216.58.212.234.443 > 192.168.1.78.47713: tcp 0
16:02:12.959195 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.959552 IP 216.58.212.234.443 > 192.168.1.78.47713: tcp 0
16:02:12.959664 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.959688 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.959772 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.959784 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.960270 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.960571 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 1418
16:02:12.960609 IP 192.168.1.78.54127 > 216.58.213.10.443: tcp 808
16:02:12.961084 IP 216.58.212.234.443 > 192.168.1.78.47713: tcp 0
16:02:12.961381 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.961411 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.961544 IP 192.168.1.78.53341 > 142.250.180.10.443: tcp 1418
16:02:12.961569 IP 192.168.1.78.53341 > 142.250.180.10.443: tcp 1418
16:02:12.962008 IP 216.58.213.10.443 > 192.168.1.78.54127: tcp 0
16:02:12.962044 IP 142.250.200.10.443 > 192.168.1.78.34013: tcp 0
16:02:12.962465 IP 216.58.213.10.443 > 192.168.1.78.54127: tcp 0
16:02:12.962618 IP 192.168.1.78.53341 > 142.250.180.10.443: tcp 1418
16:02:12.962649 IP 192.168.1.78.53341 > 142.250.180.10.443: tcp 1418
16:02:12.962668 IP 192.168.1.78.54823 > 142.250.179.234.443: tcp 1418
16:02:12.962684 IP 192.168.1.78.54823 > 142.250.179.234.443: tcp 1418
16:02:12.962861 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.962899 IP 192.168.1.78.35941 > 216.58.212.202.443: tcp 1418
16:02:12.963015 IP 192.168.1.78.54823 > 142.250.179.234.443: tcp 1418
16:02:12.963044 IP 192.168.1.78.54823 > 142.250.179.234.443: tcp 1418

Did the IP addresses help?

The IPs in use are these, all on port 443 from 192.168.1.78

142.250.179.234
142.250.200.10
216.58.212.202
216.58.212.234
216.58.213.10

These are all google endpoints and that all looks OK to me.

Could you try reducing --drive-chunk-size 512M and see if that makes a difference? I wonder if there is a bug in handling that buffer.

Thanks - I've dropped it to 256M. I will report back

For what it's worth, I use --drive-chunk-size 512M all the time along with --bwlimit 95M. No issues.