Slow upload and download of individual files via rclone CLI?

This is a continuation of the discussion that developed in another topic. I thought that it would be best to create a new topic, so that the information might be easier to find by other people that might be interested in this.

Setup: Windows 7 x64; rclone v1.51.0 (os/arch: windows/amd6; go version: go1.13.7); Samsung 850 EVO and Patriot Burst SSD drives (used in the tests); Marvell Yukon 88E8001/8003/8010 PCI Gigabit Ethernet Controller; ISP plan: 500 Mbps download, 100 Mbps upload (speedtest.pro confirms those numbers; ping = 12ms); I have already created an individual client ID for rclone and I can connect to gdrive without problems;

Issue: I can only get about 20% of my upload speed (i.e., 2.3-2.5MB/s) when uploading or downloading a single file via rclone (see Stats 1). This also happens with FileZilla Pro, Mountain Duck or Air Explorer, although with Air Explorer I can get up to 4.7MB/s per single file. I can successfully max out my upload (about 12MB/s) when uploading or downloading multiple files (see Stats 2). I can also max out my upload (about 11-12MB/s) when uploading a single or multiple files via Chrome or Opera.

Stats 1

C:\Users\an531\Desktop\Rclone\rclone-v1.51.0-windows-amd64>rclone copy "B:\file" gdrive:\ -P --drive-chunk-size 128M
Transferred:      118.684M / 2.261 GBytes, 5%, 2.498 MBytes/s, ETA 14m39s
Transferred:            0 / 1, 0%
Elapsed time:        47.5s
Transferring:
 *                                    file.7z:  5% /2.261G, 2.517M/s, 14m32s

Stats 2

C:\Users\an531\Desktop\Rclone\rclone-v1.51.0-windows-amd64>rclone copy "C:\Users\an531\Desktop\test3" gdrive:\ -P --drive-chunk-size 128M --transfers 5
    Transferred:      176.199M / 396.643 MBytes, 44%, 12.022 MBytes/s, ETA 18s
    Transferred:            0 / 5, 0%
    Elapsed time:        14.6s
    Transferring:
     *                                     test1.rar: 42% /79.329M, 2.323M/s, 19s
     *                                     test2.rar: 45% /79.329M, 2.462M/s, 17s
     *                                     test3.rar: 45% /79.329M, 2.455M/s, 17s
     *                                     test4.rar: 44% /79.329M, 2.386M/s, 18s
     *                                     test5.rar: 44% /79.329M, 2.453M/s, 18s

@thestigma suggested to contact the ISP once we know how to get the IP for the gdrive server to which I am connecting to via rclone. However, I honestly doubt that they will be of any help. Most likely, I will get reconnected 5-10 times to different people just to finally hear that they'll call me back, which will never happen. I'm still waiting for some of those calls since the 20th century...

Therefore, I would really appreciate any information that might help me solve this issue. It might have something to do with router or port settings.

Thank you for your time and help in advance.

I suggest you try the latest beta as I fixed multithread download being slow on windows. Hopefully that will fix it :slight_smile:

I tried it on the latest rclone (version below) and, unfortunately, the results are exactly the same.

rclone v1.51.0-129-gd9c8c47e-beta
os/arch: windows/amd64
go version: go1.14.1

Try running with -vv and see whether the multithread download is active. Assuming it is, experiment with

  --multi-thread-cutoff SizeSuffix   Use multi-thread downloads for files above this size. (default 250M)
  --multi-thread-streams int         Max number of streams to use for multi-thread downloads. (default 4)

And see if that helps. Try increasing and decreasing the number of streams.

I can't tell from the information below if the multithread download is active. Perhaps you mean if the chunks are being send simultaneously... If so, it would seems that no, I get the Sending chunk messages sequentially.

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv
2020/03/25 18:41:32 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv"]
2020/03/25 18:41:32 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 18:41:33 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 18:41:33 DEBUG : 101.7z.006: Sending chunk 0 length 8388608
2020-03-25 18:41:36 DEBUG : 101.7z.006: Sending chunk 8388608 length 8388608
2020-03-25 18:41:40 DEBUG : 101.7z.006: Sending chunk 16777216 length 8388608
2020-03-25 18:41:43 DEBUG : 101.7z.006: Sending chunk 25165824 length 8388608
2020-03-25 18:41:46 DEBUG : 101.7z.006: Sending chunk 33554432 length 8388608
2020-03-25 18:41:49 DEBUG : 101.7z.006: Sending chunk 41943040 length 8388608
2020-03-25 18:41:53 DEBUG : 101.7z.006: Sending chunk 50331648 length 8388608
2020-03-25 18:41:56 DEBUG : 101.7z.006: Sending chunk 58720256 length 8388608
2020-03-25 18:41:59 DEBUG : 101.7z.006: Sending chunk 67108864 length 8388608
2020-03-25 18:42:02 DEBUG : 101.7z.006: Sending chunk 75497472 length 8388608
2020-03-25 18:42:06 DEBUG : 101.7z.006: Sending chunk 83886080 length 8388608
2020-03-25 18:42:09 DEBUG : 101.7z.006: Sending chunk 92274688 length 8388608
Transferred: 89.184M / 2.261 GBytes, 4%, 2.415 MBytes/s, ETA 15m21s
Transferred: 0 / 1, 0%
Elapsed time: 36.9s
Transferring:
101.7z.006: 3% /2.261G, 2.434M/s, 15m14s

I have a Q9550, so 4 cores and 4 threads. I get the following information when I try to run rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --multi-thread-streams 8:

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --multi-thread-streams 8
2020/03/25 18:48:09 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv" "--multi-thread-streams" "8"]
2020/03/25 18:48:09 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 18:48:09 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020-03-25 18:48:10 DEBUG : Keeping previous permissions for config file: -rw-rw-rw-
2020-03-25 18:48:10 DEBUG : gdrive: Saved new token in config file
2020-03-25 18:48:10 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 18:48:11 DEBUG : 101.7z.006: Sending chunk 0 length 8388608
2020-03-25 18:48:14 DEBUG : 101.7z.006: Sending chunk 8388608 length 8388608
2020-03-25 18:48:17 DEBUG : 101.7z.006: Sending chunk 16777216 length 8388608
2020-03-25 18:48:20 DEBUG : 101.7z.006: Sending chunk 25165824 length 8388608
2020-03-25 18:48:24 DEBUG : 101.7z.006: Sending chunk 33554432 length 8388608
2020-03-25 18:48:27 DEBUG : 101.7z.006: Sending chunk 41943040 length 8388608
2020-03-25 18:48:30 DEBUG : 101.7z.006: Sending chunk 50331648 length 8388608
2020-03-25 18:48:34 DEBUG : 101.7z.006: Sending chunk 58720256 length 8388608
2020-03-25 18:48:37 DEBUG : 101.7z.006: Sending chunk 67108864 length 8388608
2020-03-25 18:48:40 DEBUG : 101.7z.006: Sending chunk 75497472 length 8388608
2020-03-25 18:48:43 DEBUG : 101.7z.006: Sending chunk 83886080 length 8388608
2020-03-25 18:48:47 DEBUG : 101.7z.006: Sending chunk 92274688 length 8388608
2020-03-25 18:48:50 DEBUG : 101.7z.006: Sending chunk 100663296 length 8388608
2020-03-25 18:48:53 DEBUG : 101.7z.006: Sending chunk 109051904 length 8388608
2020-03-25 18:48:57 DEBUG : 101.7z.006: Sending chunk 117440512 length 8388608
2020-03-25 18:49:00 DEBUG : 101.7z.006: Sending chunk 125829120 length 8388608
Transferred: 120.812M / 2.261 GBytes, 5%, 2.399 MBytes/s, ETA 15m14s
Transferred: 0 / 1, 0%
Elapsed time: 50.3s
Transferring:
101.7z.006: 5% /2.261G, 2.419M/s, 15m7s

It seems to make no difference...

Multi stream will only help you for downloading.

For uploading increasing --drive-chunk-size usually helps quite a bit at the expense of using more memory.

I get the same results with rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --drive-chunk-size 256M as shown below.

C:\Users\an531\Desktop\rclone-beta-latest-windows-amd64\rclone-v1.51.0-129-gd9c8c47e-beta-windows-amd64>rclone copy "B:\101\101.7z.006" gdrive:\ -P -vv --drive-chunk-size 256M
2020/03/25 19:01:23 DEBUG : rclone: Version "v1.51.0-129-gd9c8c47e-beta" starting with parameters ["rclone" "copy" "B:\101\101.7z.006" "gdrive:\" "-P" "-vv" "--drive-chunk-size" "256M"]
2020/03/25 19:01:23 DEBUG : Using config file from "C:\Users\an531\.config\rclone\rclone.conf"
2020-03-25 19:01:23 DEBUG : 101.7z.006: Need to transfer - File not found at Destination
2020-03-25 19:01:24 DEBUG : 101.7z.006: Sending chunk 0 length 268435456
Transferred: 53.652M / 2.261 GBytes, 2%, 2.438 MBytes/s, ETA 15m27s
Transferred: 0 / 1, 0%
Elapsed time: 22.0s
Transferring:
101.7z.006: 2% /2.261G, 2.443M/s, 15m25s

Is it possible to send multiple chunks simultaneously?

Google drive does not support async chunk upload, all upload must be done in sync. :confused:

See: https://stackoverflow.com/questions/56076208/can-you-upload-a-file-in-multiple-simultaneous-chunks-to-google-drive-via-the-re

The conclusion I'm drawing from these tests is that you somehow have poor connectivity, or throttled connectivity to the drive API.

Note that the drive web app uses a different API, not the published API that rclone uses.

It could conceivably be your ISP limiting the speed, or it might be google.

One thing people have had some success with is using a VPN - you could try that.