Rclone GDrive Upload Speeds VS GDrive Web UI Upload Speeds

Help and Support
1

I have searched around and seen this brought up in another post on here, but is it possible to get speeds as fast as the WebUI Uploader when using Rclone?

I max out my upload speed in the WebUI using my 1Gbit upload on my server but when using Rclone, I average 30MB/s to 40MB/s. I have messed around with all the drive chunk sizes + everything else that is recommended and 30 to 40MB/s is the best result I eventually came up with.

In another post, it was mentioned that the WebUI uses a different endpoint than Rclone uses. Is it possible to have Rclone use the same endpoint the WebUI uses?

I love Rclone and want to continue using it, but if I can't max out my speed then I have no other choice to use the WebUI uploader if I want to max out.

Thanks

2

No, one uses the Drive API, the other does not.

Better to figure out what command you are running, version, etc.

4

After a bunch of trial and error, I figured out how to get my speed really fast. I'm not sure what's going on because this is a very weird issue...

So originally in my above post, I was transferring some movies to my movie folder. That folder has about 3,000 movies and folders in it. I was getting speeds of 30MB/s to 40MB/s max... I tried everything and it wouldn't go higher...

Now here's the weird part... I tried to initiate this same transfer, except I started the transfer to a brand new empty folder on the root of the Drive without anything in it. Now I'm getting speeds of 80MB/s to 90MB/s!! I'm not sure what's going on. I'm getting faster speeds when I transfer to the empty new folder. I kept testing it out, switching back and forth from my 3,000 movie folder and this new folder on the drive root and the results were very consistent. 30-40MB/s on the movie folder, and 80-90MB/s on the new folder.

While I transferring to the movie folder, it takes about 30 to 60 seconds for the speed to slowly rise to the 30's. But on the empty folder, it jumped to the 80's in less than 10 seconds.

Maybe someone else should test this and see if they get similar results. Transfer to a folder with thousands of folders\files, then transfer to a new folder on the root and see if you see any speed differences.

5

You still haven't shared what version you are running, what command, some logging information , etc.

My average uploads on a daily basis:

Transferred:   	  117.158G / 117.158 GBytes, 100%, 55.950 MBytes/s, ETA 0s
Transferred:   	  709.774G / 709.774 GBytes, 100%, 90.389 MBytes/s, ETA 0s
Transferred:   	  714.480G / 717.775 GBytes, 100%, 90.315 MBytes/s, ETA 37s
Transferred:   	  716.855G / 717.775 GBytes, 100%, 89.949 MBytes/s, ETA 10s
Transferred:   	  718.383G / 718.383 GBytes, 100%, 89.596 MBytes/s, ETA 0s
Transferred:   	  155.781G / 155.781 GBytes, 100%, 70.944 MBytes/s, ETA 0s
Transferred:   	   87.026G / 87.026 GBytes, 100%, 72.934 MBytes/s, ETA 0s
Transferred:   	   95.129G / 95.129 GBytes, 100%, 86.883 MBytes/s, ETA 0s
Transferred:   	   40.317G / 40.317 GBytes, 100%, 76.382 MBytes/s, ETA 0s
Transferred:   	   93.209G / 93.372 GBytes, 100%, 68.946 MBytes/s, ETA 2s
Transferred:   	   93.381G / 93.381 GBytes, 100%, 68.776 MBytes/s, ETA 0s
Transferred:   	   19.870G / 19.870 GBytes, 100%, 62.648 MBytes/s, ETA 0s
Transferred:   	   19.614G / 19.614 GBytes, 100%, 78.524 MBytes/s, ETA 0s
Transferred:   	   41.434G / 41.434 GBytes, 100%, 88.688 MBytes/s, ETA 0s
Transferred:   	   25.469G / 25.469 GBytes, 100%, 43.844 MBytes/s, ETA 0s
Transferred:   	   36.100G / 36.100 GBytes, 100%, 63.044 MBytes/s, ETA 0s
Transferred:   	  214.053G / 214.312 GBytes, 100%, 84.928 MBytes/s, ETA 3s
Transferred:   	  214.316G / 214.316 GBytes, 100%, 84.695 MBytes/s, ETA 0s
Transferred:   	   26.200G / 26.200 GBytes, 100%, 64.196 MBytes/s, ETA 0s
Transferred:   	  137.919G / 137.919 GBytes, 100%, 58.015 MBytes/s, ETA 0s
Transferred:   	   86.395G / 86.653 GBytes, 100%, 64.064 MBytes/s, ETA 4s
Transferred:   	   86.660G / 86.660 GBytes, 100%, 63.813 MBytes/s, ETA 0s
Transferred:   	   25.542G / 25.542 GBytes, 100%, 81.002 MBytes/s, ETA 0s
Transferred:   	  110.392G / 110.392 GBytes, 100%, 84.165 MBytes/s, ETA 0s
Transferred:   	   80.740G / 80.740 GBytes, 100%, 78.072 MBytes/s, ETA 0s
Transferred:   	   71.950G / 71.950 GBytes, 100%, 85.437 MBytes/s, ETA 0s
Transferred:   	  192.316G / 192.316 GBytes, 100%, 89.733 MBytes/s, ETA 0s
Transferred:   	  259.947G / 259.947 GBytes, 100%, 89.436 MBytes/s, ETA 0s

Have a few things in flight as sometimes I am uploading other things as well, but I'm pretty constant at being able to push my Gigabit almost all the time. I pretty much use defaults:

/usr/bin/rclone move /data/local/ gcrypt: --log-file /opt/rclone/logs/upload.log -v --drive-chunk-size 32M --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs

Minus the chunk size as a little bigger helps.

6

I'm using the latest version. I'm using the same command in both runs but just different folders. Why would it be faster in 1 folder and slower in the other?

Untitled-1
Untitled-1.png1187×881 38.5 KB

7

tpslimit 3 slows things down. No reason for that so I'd recommend removing that.

You are running both at the same time and the question is why is one window running faster than the other?

8

Those are 2 separate screenshots. I am not running them at the same time.

9

Run the transfer with -vv and grab the output and we can see by the logs usually what's going on if it's a rclone related item.

10

This is the slow one. It's the exact same as the other one, except the other one is faster.

2019/05/31 10:48:41 DEBUG : rclone: Version "v1.47.0" starting with parameters ["rclone" "copy" "G:\\Finished\\Movies" "Drive:Movies" "--transfers" "4" "--stats=1s" "--drive-chunk-size" "64M" "--checkers" "6" "--tpslimit" "3" "--max-transfer" "100G" " "-vv" "--log-file" "backup.log"]
2019/05/31 10:48:41 DEBUG : Using config file from "C:\\Users\\Administrator\\.config\\rclone\\rclone.conf"
2019/05/31 10:48:41 INFO  : Starting HTTP transaction limiter: max 3 transactions/s with burst 1
2019/05/31 10:48:43 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        1.6s

2019/05/31 10:48:44 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        2.6s

2019/05/31 10:48:45 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        3.6s

2019/05/31 10:48:46 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        4.6s

2019/05/31 10:48:47 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        5.6s

2019/05/31 10:48:48 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        6.6s

2019/05/31 10:48:49 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        7.6s

2019/05/31 10:48:50 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:        8.6s

2019/05/31 10:48:51 INFO  : 
Transferred:   	         0 / 2.795 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 2, 0%
Elapsed time:        9.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -


2019/05/31 10:48:52 INFO  : 
Transferred:   	         0 / 2.795 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 2, 0%
Elapsed time:       10.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -


2019/05/31 10:48:53 INFO  : 
Transferred:   	         0 / 6.860 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 4, 0%
Elapsed time:       11.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -


2019/05/31 10:48:54 INFO  : 
Transferred:   	         0 / 8.481 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 5, 0%
Elapsed time:       12.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -


2019/05/31 10:48:55 INFO  : 
Transferred:   	         0 / 12.696 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 7, 0%
Elapsed time:       13.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:48:55 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 0 length 67108864
2019/05/31 10:48:56 INFO  : 
Transferred:   	         0 / 14.090 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 8, 0%
Elapsed time:       14.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:48:57 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 0 length 67108864
2019/05/31 10:48:57 INFO  : 
Transferred:   	         0 / 16.048 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 9, 0%
Elapsed time:       15.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:48:58 INFO  : 
Transferred:   	         0 / 20.349 GBytes, 0%, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 13, 0%
Elapsed time:       16.6s
Transferring:
 * _________________________________:  0% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:48:58 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 0 length 67108864
2019/05/31 10:48:59 INFO  : 
Transferred:   	   37.910M / 21.861 GBytes, 0%, 2.153 MBytes/s, ETA 2h53m0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 15, 0%
Elapsed time:       17.6s
Transferring:
 * _________________________________:  2% /1.672G, 0/s, -
 * _________________________________:  0% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:48:59 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 0 length 67108864
2019/05/31 10:48:59 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 67108864 length 67108864
2019/05/31 10:49:00 INFO  : 
Transferred:   	   85.102M / 25.834 GBytes, 0%, 4.573 MBytes/s, ETA 1h36m6s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 23, 0%
Elapsed time:       18.6s
Transferring:
 * _________________________________:  3% /1.672G, 5.001M/s, 5m29s
 * _________________________________:  1% /1.123G, 0/s, -
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -


2019/05/31 10:49:01 INFO  : 
Transferred:   	      128M / 27.370 GBytes, 0%, 6.527 MBytes/s, ETA 1h11m14s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 26, 0%
Elapsed time:       19.6s
Transferring:
 * _________________________________:  3% /1.672G, 6.399M/s, 4m17s
 * _________________________________:  5% /1.123G, 5.940M/s, 3m2s
 * _________________________________:  0% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:49:01 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 67108864 length 67108864
2019/05/31 10:49:02 INFO  : 
Transferred:   	  165.477M / 31.063 GBytes, 1%, 8.029 MBytes/s, ETA 1h5m41s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 32, 0%
Elapsed time:       20.6s
Transferring:
 * _________________________________:  3% /1.672G, 5.817M/s, 4m43s
 * _________________________________:  5% /1.123G, 5.832M/s, 3m6s
 * _________________________________:  2% /1.657G, 0/s, -
 * _________________________________:  0% /2.408G, 0/s, -

2019/05/31 10:49:02 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 67108864 length 67108864
2019/05/31 10:49:03 INFO  : 
Transferred:   	  275.082M / 32.427 GBytes, 1%, 12.730 MBytes/s, ETA 43m6s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 33, 0%
Elapsed time:       21.6s
Transferring:
 * _________________________________:  5% /1.672G, 5.332M/s, 5m4s
 * _________________________________:  5% /1.123G, 5.346M/s, 3m23s
 * _________________________________:  3% /1.657G, 4.809M/s, 5m39s
 * _________________________________:  2% /2.408G, 4.456M/s, 8m59s

2019/05/31 10:49:03 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 67108864 length 67108864
2019/05/31 10:49:04 INFO  : 
Transferred:   	  318.184M / 34.344 GBytes, 1%, 14.073 MBytes/s, ETA 41m16s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 34, 0%
Elapsed time:       22.6s
Transferring:
 * _________________________________:  7% /1.672G, 6.835M/s, 3m52s
 * _________________________________:  5% /1.123G, 4.935M/s, 3m39s
 * _________________________________:  3% /1.657G, 5.880M/s, 4m37s
 * _________________________________:  2% /2.408G, 5.816M/s, 6m53s


2019/05/31 10:49:04 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 134217728 length 67108864
2019/05/31 10:49:05 INFO  : 
Transferred:   	  370.742M / 37.898 GBytes, 1%, 15.703 MBytes/s, ETA 40m47s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 39, 0%
Elapsed time:       23.6s
Transferring:
 * _________________________________:  7% /1.672G, 9.143M/s, 2m53s
 * _________________________________:  9% /1.123G, 7.113M/s, 2m25s
 * _________________________________:  3% /1.657G, 5.390M/s, 5m2s
 * _________________________________:  2% /2.408G, 5.332M/s, 7m30s

2019/05/31 10:49:05 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 134217728 length 67108864
2019/05/31 10:49:06 INFO  : 
Transferred:   	  460.273M / 42.112 GBytes, 1%, 18.703 MBytes/s, ETA 38m0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 41, 0%
Elapsed time:       24.6s
Transferring:
 * _________________________________:  7% /1.672G, 8.534M/s, 3m5s
 * _________________________________: 11% /1.123G, 8.544M/s, 1m59s
 * _________________________________:  7% /1.657G, 4.975M/s, 5m16s
 * _________________________________:  3% /2.408G, 4.921M/s, 8m4s

2019/05/31 10:49:06 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 134217728 length 67108864
2019/05/31 10:49:07 INFO  : 
Transferred:   	      512M / 43.687 GBytes, 1%, 19.982 MBytes/s, ETA 36m53s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 43, 0%
Elapsed time:       25.6s
Transferring:
 * _________________________________:  7% /1.672G, 8.000M/s, 3m18s
 * _________________________________: 11% /1.123G, 8.010M/s, 2m7s
 * _________________________________:  7% /1.657G, 9.190M/s, 2m50s
 * _________________________________:  5% /2.408G, 8.610M/s, 4m31s

2019/05/31 10:49:07 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 134217728 length 67108864
2019/05/31 10:49:08 INFO  : 
Transferred:   	  570.953M / 47.509 GBytes, 1%, 21.456 MBytes/s, ETA 37m20s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 49, 0%
Elapsed time:       26.6s
Transferring:
 * _________________________________: 10% /1.672G, 7.500M/s, 3m23s
 * _________________________________: 11% /1.123G, 7.509M/s, 2m16s
 * _________________________________:  7% /1.657G, 8.577M/s, 3m2s
 * _________________________________:  5% /2.408G, 8.530M/s, 4m34s

2019/05/31 10:49:08 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 201326592 length 67108864
2019/05/31 10:49:09 INFO  : 
Transferred:   	  645.293M / 51.136 GBytes, 1%, 23.372 MBytes/s, ETA 36m52s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 51, 0%
Elapsed time:       27.6s
Transferring:
 * _________________________________: 11% /1.672G, 10.943M/s, 2m18s
 * _________________________________: 15% /1.123G, 9.185M/s, 1m45s
 * _________________________________:  8% /1.657G, 8.041M/s, 3m12s
 * _________________________________:  5% /2.408G, 7.997M/s, 4m52s

2019/05/31 10:49:09 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 201326592 length 67108864
2019/05/31 10:49:10 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 201326592 length 67108864
2019/05/31 10:49:10 INFO  : 
Transferred:   	      704M / 54.611 GBytes, 1%, 24.607 MBytes/s, ETA 37m23s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 56, 0%
Elapsed time:       28.6s
Transferring:
 * _________________________________: 11% /1.672G, 10.350M/s, 2m26s
 * _________________________________: 16% /1.123G, 10.467M/s, 1m31s
 * _________________________________: 11% /1.657G, 9.373M/s, 2m40s
 * _________________________________:  5% /2.408G, 7.497M/s, 5m11s

2019/05/31 10:49:11 INFO  : 
Transferred:   	  755.586M / 58.020 GBytes, 1%, 25.518 MBytes/s, ETA 38m18s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 58, 0%
Elapsed time:       29.6s
Transferring:
 * _________________________________: 11% /1.672G, 9.703M/s, 2m36s
 * _________________________________: 16% /1.123G, 9.812M/s, 1m37s
 * _________________________________: 11% /1.657G, 10.936M/s, 2m17s
 * _________________________________:  7% /2.408G, 9.444M/s, 4m2s

2019/05/31 10:49:11 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 201326592 length 67108864
2019/05/31 10:49:12 INFO  : 
Transferred:   	  811.328M / 60.307 GBytes, 1%, 26.505 MBytes/s, ETA 38m19s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 61, 0%
Elapsed time:       30.6s
Transferring:
 * _________________________________: 13% /1.672G, 9.097M/s, 2m42s
 * _________________________________: 16% /1.123G, 9.199M/s, 1m44s
 * _________________________________: 11% /1.657G, 10.253M/s, 2m26s
 * _________________________________:  7% /2.408G, 10.437M/s, 3m37s

2019/05/31 10:49:12 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 268435456 length 67108864
2019/05/31 10:49:13 INFO  : 
Transferred:   	  908.605M / 64.056 GBytes, 1%, 28.744 MBytes/s, ETA 37m30s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 63, 0%
Elapsed time:       31.6s
Transferring:
 * _________________________________: 14% /1.672G, 11.554M/s, 2m6s
 * _________________________________: 21% /1.123G, 11.167M/s, 1m20s
 * _________________________________: 12% /1.657G, 9.612M/s, 2m34s
 * _________________________________:  7% /2.408G, 9.785M/s, 3m52s

2019/05/31 10:49:13 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 268435456 length 67108864
2019/05/31 10:49:14 INFO  : 
Transferred:   	  953.188M / 66.432 GBytes, 1%, 29.231 MBytes/s, ETA 38m14s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 65, 0%
Elapsed time:       32.6s
Transferring:
 * _________________________________: 14% /1.672G, 11.806M/s, 2m3s
 * _________________________________: 22% /1.123G, 11.926M/s, 1m14s
 * _________________________________: 14% /1.657G, 10.649M/s, 2m15s
 * _________________________________:  7% /2.408G, 9.173M/s, 4m7s

2019/05/31 10:49:14 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 268435456 length 67108864
2019/05/31 10:49:15 INFO  : 
Transferred:   	  984.492M / 70.252 GBytes, 1%, 29.292 MBytes/s, ETA 40m22s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 67, 0%
Elapsed time:       33.6s
Transferring:
 * _________________________________: 14% /1.672G, 11.068M/s, 2m11s
 * _________________________________: 22% /1.123G, 11.180M/s, 1m19s
 * _________________________________: 15% /1.657G, 12.297M/s, 1m57s
 * _________________________________:  8% /2.408G, 9.816M/s, 3m49s

2019/05/31 10:49:16 INFO  : 
Transferred:   	    1.066G / 72.150 GBytes, 1%, 31.531 MBytes/s, ETA 38m28s
Errors:                 0
Checks:                 1 / 1, 100%
Transferred:            0 / 68, 0%
Elapsed time:       34.6s
Transferring:
 * _________________________________: 17% /1.672G, 10.376M/s, 2m15s
 * _________________________________: 23% /1.123G, 10.482M/s, 1m23s
 * _________________________________: 15% /1.657G, 11.578M/s, 2m4s
 * _________________________________: 10% /2.408G, 11.326M/s, 3m15s

2019/05/31 10:49:16 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 268435456 length 67108864
2019/05/31 10:49:16 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 335544320 length 67108864
2019/05/31 10:49:17 INFO  : 
Transferred:   	    1.125G / 75.599 GBytes, 1%, 32.348 MBytes/s, ETA 39m17s
Errors:                 0
Checks:                 2 / 2, 100%
Transferred:            0 / 71, 0%
Elapsed time:       35.6s
Transferring:
 * _________________________________: 18% /1.672G, 13.320M/s, 1m44s
 * _________________________________: 27% /1.123G, 13.550M/s, 1m1s
 * _________________________________: 15% /1.657G, 10.854M/s, 2m12s
 * _________________________________: 10% /2.408G, 11.279M/s, 3m15s

2019/05/31 10:49:17 DEBUG : _________________________________/_________________________________.mp4: Sending chunk 335544320 length 67108864
11

You'd want to remove the tpslimit 3 as I had suggested before as that slow things down.

You really have no rclone items in the log causing anything to be slow.

Are you able to validate CPU/Memory/Disk IO/networking/virus checkers/etc?

12

Into a new folder:
Transferred: 5.982G / 5.982 GBytes, 100%, 38.081 MBytes/s, ETA 0s

Into an existing large folder:
Transferred: 5.983G / 5.983 GBytes, 100%, 37.686 MBytes/s, ETA 0s

retested with a larger chunk size. :slight_smile:

New folder:
Transferred: 4.637G / 5.982 GBytes, 78%, 78.408 MBytes/s, ETA 17s

Existing folder:
Transferred: 5.983G / 5.983 GBytes, 100%, 77.318 MBytes/s, ETA 0s

1 Like
13

Xeon E31230 16GB 2x2TB HD's. Fresh install of Windows Server.

Not sure how that would factor into doing 2 exact same transfers.

The only thing that is changed between them is the destination folder. Like I said, it only goes fast when it's an empty folder but when transferring to the folder that contains 3,000+ Folders\Files, then it seems to be throttled below 40MB/s.

14

Ehhh, thanks for tryin! Have no idea what could be going on with mine. :neutral_face:

15

I tested by copying into a folder with 1.9k files as that's the best I have to test with.

You can try --fast-list and see if that helps:

image
image.png804×269 62.8 KB

16

you didn't post your whole log. But what I am going to assume is that you're seeing this happen if a folder is massive:

root@:~# rclone copy test robgs:xxxx/20190519/ --transfers 6 --drive-chunk-size 64M -vv --stats=30s
2019/05/31 13:19:59 DEBUG : rclone: Version "v1.47.0-067-g9c6f3ae8-beta" starting with parameters ["rclone" "copy" "test" "robgs:xxxx/20190519/" "--transfers" "6" "--drive-chunk-size" "64M" "-vv" "--stats=30s"]
2019/05/31 13:19:59 DEBUG : Using config file from "/root/.rclone.conf"

2019/05/31 13:20:30 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:       30.7s

2019/05/31 13:21:00 INFO  : 
Transferred:   	         0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 0, -
Elapsed time:      1m0.7s
2019/05/31 13:22:04 DEBUG : test5: Sending chunk 0 length 67108864
2019/05/31 13:22:04 DEBUG : test1: Sending chunk 0 length 67108864
2019/05/31 13:22:04 DEBUG : test3: Sending chunk 0 length 67108864
2019/05/31 13:22:04 DEBUG : test6: Sending chunk 0 length 67108864
2019/05/31 13:22:04 DEBUG : test2: Sending chunk 0 length 67108864
2019/05/31 13:22:04 DEBUG : test4: Sending chunk 0 length 67108864
2019/05/31 13:22:10 DEBUG : test2: Sending chunk 67108864 length 67108864
2019/05/31 13:22:10 DEBUG : test3: Sending chunk 67108864 length 67108864
2019/05/31 13:22:10 DEBUG : test5: Sending chunk 67108864 length 67108864
2019/05/31 13:22:10 DEBUG : test4: Sending chunk 67108864 length 67108864
2019/05/31 13:22:11 DEBUG : test1: Sending chunk 67108864 length 67108864
2019/05/31 13:22:12 DEBUG : test6: Sending chunk 67108864 length 67108864
2019/05/31 13:22:14 DEBUG : test2: Sending chunk 134217728 length 67108864
2019/05/31 13:22:14 DEBUG : test3: Sending chunk 134217728 length 67108864
2019/05/31 13:22:15 DEBUG : test5: Sending chunk 134217728 length 67108864
2019/05/31 13:22:15 DEBUG : test4: Sending chunk 134217728 length 67108864
2019/05/31 13:22:18 DEBUG : test2: Sending chunk 201326592 length 67108864
2019/05/31 13:22:18 DEBUG : test1: Sending chunk 134217728 length 67108864
2019/05/31 13:22:18 DEBUG : test6: Sending chunk 134217728 length 67108864
2019/05/31 13:22:19 DEBUG : test3: Sending chunk 201326592 length 67108864
2019/05/31 13:22:20 DEBUG : test5: Sending chunk 201326592 length 67108864
2019/05/31 13:22:20 DEBUG : test4: Sending chunk 201326592 length 67108864
2019/05/31 13:22:21 DEBUG : test2: Sending chunk 268435456 length 67108864
2019/05/31 13:22:23 DEBUG : test3: Sending chunk 268435456 length 67108864
2019/05/31 13:22:24 DEBUG : test2: Sending chunk 335544320 length 67108864
2019/05/31 13:22:24 DEBUG : test4: Sending chunk 268435456 length 67108864
2019/05/31 13:22:24 DEBUG : test5: Sending chunk 268435456 length 67108864
2019/05/31 13:22:24 DEBUG : test1: Sending chunk 201326592 length 67108864
2019/05/31 13:22:25 DEBUG : test6: Sending chunk 201326592 length 67108864
2019/05/31 13:22:27 DEBUG : test3: Sending chunk 335544320 length 67108864
2019/05/31 13:22:28 DEBUG : test2: Sending chunk 402653184 length 67108864
2019/05/31 13:22:28 DEBUG : test4: Sending chunk 335544320 length 67108864
2019/05/31 13:22:29 DEBUG : test5: Sending chunk 335544320 length 67108864
2019/05/31 13:22:29 DEBUG : test1: Sending chunk 268435456 length 67108864
2019/05/31 13:22:29 DEBUG : test6: Sending chunk 268435456 length 67108864
2019/05/31 13:22:30 INFO  : 
Transferred:   	    1.971G / 5.982 GBytes, 33%, 13.393 MBytes/s, ETA 5m6s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 6, 0%
Elapsed time:     2m30.7s
Transferring:
 *                                         test1: 25% /1020.899M, 10.355M/s, 1m13s
 *                                         test2: 40% /1020.899M, 15.998M/s, 37s
 *                                         test3: 36% /1020.899M, 14.392M/s, 45s
 *                                         test4: 34% /1020.899M, 13.682M/s, 48s
 *                                         test5: 34% /1020.899M, 13.599M/s, 49s
 *                                         test6: 26% /1020.899M, 10.709M/s, 1m10s

You'll see that there was a stall in the beginning while rclone tried to list the massive directory to see if it needed to transfer the files (i.e. checking if they exist). That time will be added to the overall time and then the total looks like this for me:

2019/05/31 13:23:58 INFO  : 
Transferred:   	    5.982G / 5.982 GBytes, 100%, 25.667 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            6 / 6, 100%
Elapsed time:     3m58.6s

But that isn't a fair answer since 50% of the time it was simply trying to list a massive directory.

17

--fast-list would help with that :slight_smile:

18

It didn't really help in my case but I left it out to show the issue with how it was tested. Rerunning with fast list was a little better but will still stalled for 1.30 minutes and still give a lower overall picture of throughput because the delay to start is factored in.

my test was against an extreme case destination folder though with 22,850 files in it without any further subdirectories. :slight_smile:

root@:/# rclone lsl --fast-list robgs:xxxx/20190519 --checkers 35 | wc -l
22850

19

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.