Slow upload and download of individual files via rclone CLI?

Help and Support
1

This is a continuation of the discussion that developed in another topic. I thought that it would be best to create a new topic, so that the information might be easier to find by other people that might be interested in this.

Setup: Windows 7 x64; rclone v1.51.0 (os/arch: windows/amd6; go version: go1.13.7); Samsung 850 EVO and Patriot Burst SSD drives (used in the tests); Marvell Yukon 88E8001/8003/8010 PCI Gigabit Ethernet Controller; ISP plan: 500 Mbps download, 100 Mbps upload (speedtest.pro confirms those numbers; ping = 12ms); I have already created an individual client ID for rclone and I can connect to gdrive without problems;

Issue: I can only get about 20% of my upload speed (i.e., 2.3-2.5MB/s) when uploading or downloading a single file via rclone (see Stats 1). This also happens with FileZilla Pro, Mountain Duck or Air Explorer, although with Air Explorer I can get up to 4.7MB/s per single file. I can successfully max out my upload (about 12MB/s) when uploading or downloading multiple files (see Stats 2). I can also max out my upload (about 11-12MB/s) when uploading a single or multiple files via Chrome or Opera.

Stats 1

C:\Users\an531\Desktop\Rclone\rclone-v1.51.0-windows-amd64>rclone copy "B:\file" gdrive:\ -P --drive-chunk-size 128M
Transferred:      118.684M / 2.261 GBytes, 5%, 2.498 MBytes/s, ETA 14m39s
Transferred:            0 / 1, 0%
Elapsed time:        47.5s
Transferring:
 *                                    file.7z:  5% /2.261G, 2.517M/s, 14m32s

Stats 2

C:\Users\an531\Desktop\Rclone\rclone-v1.51.0-windows-amd64>rclone copy "C:\Users\an531\Desktop\test3" gdrive:\ -P --drive-chunk-size 128M --transfers 5
    Transferred:      176.199M / 396.643 MBytes, 44%, 12.022 MBytes/s, ETA 18s
    Transferred:            0 / 5, 0%
    Elapsed time:        14.6s
    Transferring:
     *                                     test1.rar: 42% /79.329M, 2.323M/s, 19s
     *                                     test2.rar: 45% /79.329M, 2.462M/s, 17s
     *                                     test3.rar: 45% /79.329M, 2.455M/s, 17s
     *                                     test4.rar: 44% /79.329M, 2.386M/s, 18s
     *                                     test5.rar: 44% /79.329M, 2.453M/s, 18s

@thestigma suggested to contact the ISP once we know how to get the IP for the gdrive server to which I am connecting to via rclone. However, I honestly doubt that they will be of any help. Most likely, I will get reconnected 5-10 times to different people just to finally hear that they'll call me back, which will never happen. I'm still waiting for some of those calls since the 20th century...

Therefore, I would really appreciate any information that might help me solve this issue. It might have something to do with router or port settings.

Thank you for your time and help in advance.

2

I suggest you try the latest beta as I fixed multithread download being slow on windows. Hopefully that will fix it :slight_smile:

3

I tried it on the latest rclone (version below) and, unfortunately, the results are exactly the same.

rclone v1.51.0-129-gd9c8c47e-beta
os/arch: windows/amd64
go version: go1.14.1
4

Try running with -vv and see whether the multithread download is active. Assuming it is, experiment with

  --multi-thread-cutoff SizeSuffix   Use multi-thread downloads for files above this size. (default 250M)
  --multi-thread-streams int         Max number of streams to use for multi-thread downloads. (default 4)

And see if that helps. Try increasing and decreasing the number of streams.

5

I can't tell from the information below if the multithread download is active. Perhaps you mean if the chunks are being send simultaneously... If so, it would seems that no, I get the Sending chunk messages sequentially.

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv
2020/03/25 18:41:32 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv"]
2020/03/25 18:41:32 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 18:41:33 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 18:41:33 DEBUG : 101.7z.006: Sending chunk 0 length 8388608
2020-03-25 18:41:36 DEBUG : 101.7z.006: Sending chunk 8388608 length 8388608
2020-03-25 18:41:40 DEBUG : 101.7z.006: Sending chunk 16777216 length 8388608
2020-03-25 18:41:43 DEBUG : 101.7z.006: Sending chunk 25165824 length 8388608
2020-03-25 18:41:46 DEBUG : 101.7z.006: Sending chunk 33554432 length 8388608
2020-03-25 18:41:49 DEBUG : 101.7z.006: Sending chunk 41943040 length 8388608
2020-03-25 18:41:53 DEBUG : 101.7z.006: Sending chunk 50331648 length 8388608
2020-03-25 18:41:56 DEBUG : 101.7z.006: Sending chunk 58720256 length 8388608
2020-03-25 18:41:59 DEBUG : 101.7z.006: Sending chunk 67108864 length 8388608
2020-03-25 18:42:02 DEBUG : 101.7z.006: Sending chunk 75497472 length 8388608
2020-03-25 18:42:06 DEBUG : 101.7z.006: Sending chunk 83886080 length 8388608
2020-03-25 18:42:09 DEBUG : 101.7z.006: Sending chunk 92274688 length 8388608
Transferred: 89.184M / 2.261 GBytes, 4%, 2.415 MBytes/s, ETA 15m21s
Transferred: 0 / 1, 0%
Elapsed time: 36.9s
Transferring:
101.7z.006: 3% /2.261G, 2.434M/s, 15m14s

I have a Q9550, so 4 cores and 4 threads. I get the following information when I try to run rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --multi-thread-streams 8:

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --multi-thread-streams 8
2020/03/25 18:48:09 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv" "--multi-thread-streams" "8"]
2020/03/25 18:48:09 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 18:48:09 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020-03-25 18:48:10 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-03-25 18:48:10 DEBUG : gdrive: Saved new token in config file
2020-03-25 18:48:10 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 18:48:11 DEBUG : 101.7z.006: Sending chunk 0 length 8388608
2020-03-25 18:48:14 DEBUG : 101.7z.006: Sending chunk 8388608 length 8388608
2020-03-25 18:48:17 DEBUG : 101.7z.006: Sending chunk 16777216 length 8388608
2020-03-25 18:48:20 DEBUG : 101.7z.006: Sending chunk 25165824 length 8388608
2020-03-25 18:48:24 DEBUG : 101.7z.006: Sending chunk 33554432 length 8388608
2020-03-25 18:48:27 DEBUG : 101.7z.006: Sending chunk 41943040 length 8388608
2020-03-25 18:48:30 DEBUG : 101.7z.006: Sending chunk 50331648 length 8388608
2020-03-25 18:48:34 DEBUG : 101.7z.006: Sending chunk 58720256 length 8388608
2020-03-25 18:48:37 DEBUG : 101.7z.006: Sending chunk 67108864 length 8388608
2020-03-25 18:48:40 DEBUG : 101.7z.006: Sending chunk 75497472 length 8388608
2020-03-25 18:48:43 DEBUG : 101.7z.006: Sending chunk 83886080 length 8388608
2020-03-25 18:48:47 DEBUG : 101.7z.006: Sending chunk 92274688 length 8388608
2020-03-25 18:48:50 DEBUG : 101.7z.006: Sending chunk 100663296 length 8388608
2020-03-25 18:48:53 DEBUG : 101.7z.006: Sending chunk 109051904 length 8388608
2020-03-25 18:48:57 DEBUG : 101.7z.006: Sending chunk 117440512 length 8388608
2020-03-25 18:49:00 DEBUG : 101.7z.006: Sending chunk 125829120 length 8388608
Transferred: 120.812M / 2.261 GBytes, 5%, 2.399 MBytes/s, ETA 15m14s
Transferred: 0 / 1, 0%
Elapsed time: 50.3s
Transferring:
101.7z.006: 5% /2.261G, 2.419M/s, 15m7s

It seems to make no difference...

6

Multi stream will only help you for downloading.

For uploading increasing --drive-chunk-size usually helps quite a bit at the expense of using more memory.

7

I get the same results with rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --drive-chunk-size 256M as shown below.

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --drive-chunk-size 256M
2020/03/25 19:01:23 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv" "--drive-chunk-size" "256M"]
2020/03/25 19:01:23 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 19:01:23 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 19:01:24 DEBUG : 101.7z.006: Sending chunk 0 length 268435456
Transferred: 53.652M / 2.261 GBytes, 2%, 2.438 MBytes/s, ETA 15m27s
Transferred: 0 / 1, 0%
Elapsed time: 22.0s
Transferring:
101.7z.006: 2% /2.261G, 2.443M/s, 15m25s

Is it possible to send multiple chunks simultaneously?

8

Google drive does not support async chunk upload, all upload must be done in sync. :confused:

See: https://stackoverflow.com/questions/56076208/can-you-upload-a-file-in-multiple-simultaneous-chunks-to-google-drive-via-the-re

9

The conclusion I'm drawing from these tests is that you somehow have poor connectivity, or throttled connectivity to the drive API.

Note that the drive web app uses a different API, not the published API that rclone uses.

It could conceivably be your ISP limiting the speed, or it might be google.

One thing people have had some success with is using a VPN - you could try that.

10

I have finally managed to figure out what was causing the problem. While I tweaked several adapter settings as well, things only started to work properly after I enabled Windows AutoTuning, which had been disabled before.

According to my ISP plan, my upload speed should be 100Mbps or 12.5MB/s. In a speed test, I can usually do a little better, but it varies throughout the day. With AutoTuning disabled, I could upload an individual file through rclone at 2.5MB/s max. With AutoTuning enabled, I can now reach about 11.5MB/s for that same file. This isn't 12.5MB/s, but it's close enough. Besides, there are other network adapter and Windows settings that could probably be optimised better for a fiber connection. I read several guides about this before (for both Windows 7 and Windows 10), but unfortunately virtually all of them give conflicting information on what the best settings are... Any suggestions in that area are very welcome.

Anyway, to check the status of Auto-Tuning feature on your system, in an elevated command prompt windows, type the following and hit Enter:

netsh interface tcp show global

To enable Windows AutoTuning, run the following command:

netsh int tcp set global autotuninglevel=normal

Maybe this is of some use to someone in the future.

Finally, I would like to thank again all the people who tried to help me with this issue in this and another topic, where it was also discussed.

UPDATE: Apparently, it is necessary to disable Internet Protocol Version 6 (TCP/IPv6) in addition to the above, because it won't work otherwise. This can be found on Windows 7 (which is the OS that I am currently using) in Control Panel -> Network and Internet -> Network Connections, then select the relevant network interface controller/network adapter and go into Properties. Goddamn Windows and goddamn computer stuff...

11

Have you tried to disable that feature and set the max MTU and RWIN manually instead?

12

Which feature, Windows AutoTuning or Internet Protocol Version 6 (TCP/IPv6)? You probably mean the former, but I just want to make sure.

I understand that MTU is the same as Jumbo Packet and can be checked/set either in adapter settings (Control Panel) or checked/set via cmd (netsh interface ipv4 show subinterface). If that's the case, the current setting for Jumbo Packet in adapter settings is 9MB, which is the highest value in the drop-down menu.

As for RWIN, I assume you mean TCP Auto-Tuning or Receive Window Auto-Tuning level, which can only be checked/set via cmd (netsh int tcp show global) as far as I know. It only takes predefined values, which are as follows:

disabled : uses a fixed value for the tcp receive window. Limits it to 64KB (limited at 65535).
highlyrestricted : allows the receive window to grow beyond its default value, very conservatively
restricted : somewhat restricted growth of the tcp receive window beyond its default value
normal : default value, allows the receive window to grow to accommodate most conditions
experimental : allows the receive window to grow to accommodate extreme scenarios (not recommended, it can degrade performance in common scenarios, only intended for research purposes. It enables RWIN values of over 16 MB)

I'm not sure how to set the Receive Window Auto-Tuning level manually or how to determine the max. The current setting is 'normal'.

1 Like
13

How did this get disabled? My understanding is that enabled is the default?

Glad you figured it out :slight_smile:

14

I'm not sure why it was disabled, but unfortunately it was. I know that Microsoft changed the default adapter settings from W7 to W10, like the congestion provider, but I'm not sure if that was also the case with Windows AutoTuning.

In addition, Internet Protocol Version 6 (TCP/IPv6) was enabled and with it enabled I still wasn't able to get my max upload speed. However, I'm not sure if it was the setting itself that kept my upload down or perhaps the fact that, for example, I didn't restart my PC after toggling it on/off. It's my understanding that changing some adapter settings requires a reboot for them to work properly.

15

Interesting!

Was your computer using IPv6? I've found that quite often IPv6 isn't as fast as IPv4 as the ISPs still pay more attention to routing IPv4 well than IPv6.

16

I used this website to check my IPv6 connectivity and at first checking/unchecking IPv6 in adapter settings did have the expected effect, the website did/didn't see my IPv6 address. However, for whatever reason, after a while I couldn't get IPv6 connectivity at all, regardless of adapter settings. I tried to restart the adapter a couple of times, but cannot get it to work again. Currently, both the website and adapter properties show that there is no IPv6 internet access, despite it being enabled in adapter settings. Only IPv4 still works.

So, perhaps the problem was only to do with Windows AutoTuning after all. I will try to test it more in the next couple of days. Fortunately, my one file transfers are still good.

17

Windows AutoTuning fits the symptoms perfectly, so I'd guess it is that if I had to guess!

1 Like
18

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.