Rclone saturating network bandwidth

Help and Support
1

#### What is the problem you are having with rclone?
When running rclone, the entire network is brought to a complete crawl including the device that rclone is on. The WiFi network and connected clients are most affected.

There are no other applications running performing uploads at the same time. The total available upload speed is 400Mbps.

The currently solution has been to reduce the bandwidth consumption to ~6 or 7MB which is equivalent to circa 50Mbps.

#### What is your rclone version (output from rclone version)
rclone v1.53.1

#### Which OS you are using and how many bits (eg Windows 7, 64 bit)
Microsoft Windows 10 Professional version 1909 Build 18636.1110

#### Which cloud storage system are you using? (eg Google Drive)
Microsoft OneDrive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone.exe --stats 10s --progress --bwlimit 7M --checksum --log-file=rclone_log.txt --log-level DEBUG -copy /path/to/files/on/local/hard/drive remote:/ --backup-dir=remote:/path

2

I'm not sure what the question is? You can limit the bandwidth if that's the problem to reduce saturating your network.

rclone works very well :slight_smile:

3

@Animosity022, yes the current bandwidth for rclone is limited to 6MB. However at 10MB for example, it consumes the entire bandwidth even though the upload is 400Mbps.

4

Can you post a log and share some screenshots so we can look at the issue?

5

@Animosity022, sure if you can let me know the screenshots that would be helpful. What log information is needed as it's limited to events such as multipart start, SHA validation, etc?

There is no network related information at this point unless I am missing a command flag.

6

Run the command with debug log.
Share whatever tool you are using to measure bandwidth so the log can be correlated to the log.

7

@Animosity022, with or without the flag --bwlimit? If without, it'll have to be tested late at night as during the daytime, the network is actively used.

8

Without the bwlimit flag, it'll consume what it can.

For during day, I'd surmise you want to use bwlimit.

bwlimit only works for actual file copy operations and such.

Few issues back to finer grain controls:

9

Thanks @Animosity022. When you say that bwlimit only works for actual file copy operations and such, do yo mean copy, sync or move (bidirectional) and not checksum?

10

I'm assuming we are talking about your command.

The checksum is calculated locally on your file.
The remote (cloud provider) calculates on the checksum on their side.

There isn't any data transferred to calculate the checksum as the data is transferred when moving/copying the file.

11

If the checksums are calculated locally (source and destination), how are these compared? Is there no data transferred e.g. listing all directories and files?

12

Take one file as an example.

felix@guardian:~$ rclone copy /etc/hosts GD: -vv --checksum
2020/10/13 15:34:12 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv" "--checksum"]
2020/10/13 15:34:12 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/10/13 15:34:12 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2020/10/13 15:34:12 DEBUG : hosts: Need to transfer - File not found at Destination
2020/10/13 15:34:13 DEBUG : hosts: MD5 = 3e3007aa5490459a1658d5d31be3a594 OK
2020/10/13 15:34:13 INFO  : hosts: Copied (new)
2020/10/13 15:34:13 INFO  :
Transferred:           455 / 455 Bytes, 100%, 338 Bytes/s, ETA 0s
Transferred:            1 / 1, 100%
Elapsed time:         1.3s

2020/10/13 15:34:13 DEBUG : 4 go routines active

What happens is the file has a checksum calculated locally. The file is copied. The cloud calculates the checksum on the file. Rclone has both values and compares and they are the same. Once the command is done, you'd repeat as checksums are not stored locally so if you checked again, it would calculate locally and get the sum from the provider since it's part of the metadata.

felix@guardian:~$ rclone copy /etc/hosts GD: -vv --checksum
2020/10/13 15:35:52 DEBUG : rclone: Version "v1.52.3" starting with parameters ["rclone" "copy" "/etc/hosts" "GD:" "-vv" "--checksum"]
2020/10/13 15:35:52 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2020/10/13 15:35:52 DEBUG : fs cache: adding new entry for parent of "/etc/hosts", "/etc"
2020/10/13 15:35:52 DEBUG : hosts: MD5 = 3e3007aa5490459a1658d5d31be3a594 OK
2020/10/13 15:35:52 DEBUG : hosts: Size and MD5 of src and dst objects identical
2020/10/13 15:35:52 DEBUG : hosts: Unchanged skipping
2020/10/13 15:35:52 INFO  :
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
Checks:                 1 / 1, 100%
Elapsed time:         0.0s

2020/10/13 15:35:52 DEBUG : 4 go routines active
13

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.