Jammie
March 10, 2019, 4:46pm
1
I’ve got a synchronous gigabit connection, but I’m not able to upload to Google Drive at faster than about 5MB/s
$ rclone version
rclone v1.46
- os/arch: linux/amd64
- go version: go1.11.5
$ rclone --config=/etc/rclone/rclone.conf -P copy test.bin gdrive:
Transferred: 456M / 1 GBytes, 45%, 4.427 MBytes/s, ETA 2m8s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 1m43s
Transferring:
I’m able to max out my connection, when using the web interface, and I’ve got no issues with download speed using rclone. Is there something I’m missing?
Assuming you are using your own API Key/Client ID, add drive-chunk-size and make it 32M or 64M and try that out.
Run with -vv and see if any errors are there in the output as well.
rclone copy -vv -P test.mkv gcrypt: --drive-chunk-size 32M
Jammie
March 10, 2019, 8:50pm
3
Yeah, I created my own client id. Here’s the debug output, using a larger chunk size. It seems to help (10MB/s vs 5MB/s), but I’m still far from speed I would expect.
$ rclone --config=/etc/rclone/rclone.conf --drive-chunk-size 32M -vv -P copy test.bin gdrive:
2019/03/10 21:47:54 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "--config=/etc/rclone/rclone.conf" "--drive-chunk-size" "32M" "-vv" "-P" "copy" "test.bin" "gdrive:"]
2019/03/10 21:47:54 DEBUG : Using config file from "/etc/rclone/rclone.conf"
2019-03-10 21:47:55 DEBUG : test.bin: Couldn't find file - need to transfer
2019-03-10 21:47:55 DEBUG : test.bin: Sending chunk 0 length 33554432
2019-03-10 21:47:57 DEBUG : test.bin: Sending chunk 33554432 length 33554432
2019-03-10 21:48:00 DEBUG : test.bin: Sending chunk 67108864 length 33554432
2019-03-10 21:48:03 DEBUG : test.bin: Sending chunk 100663296 length 33554432
2019-03-10 21:48:06 DEBUG : test.bin: Sending chunk 134217728 length 33554432
2019-03-10 21:48:09 DEBUG : test.bin: Sending chunk 167772160 length 33554432
2019-03-10 21:48:12 DEBUG : test.bin: Sending chunk 201326592 length 33554432
2019-03-10 21:48:16 DEBUG : test.bin: Sending chunk 234881024 length 33554432
2019-03-10 21:48:19 DEBUG : test.bin: Sending chunk 268435456 length 33554432
2019-03-10 21:48:22 DEBUG : test.bin: Sending chunk 301989888 length 33554432
2019-03-10 21:48:25 DEBUG : test.bin: Sending chunk 335544320 length 33554432
2019-03-10 21:48:28 DEBUG : test.bin: Sending chunk 369098752 length 33554432
2019-03-10 21:48:31 DEBUG : test.bin: Sending chunk 402653184 length 33554432
2019-03-10 21:48:34 DEBUG : test.bin: Sending chunk 436207616 length 33554432
2019-03-10 21:48:37 DEBUG : test.bin: Sending chunk 469762048 length 33554432
2019-03-10 21:48:40 DEBUG : test.bin: Sending chunk 503316480 length 33554432
2019-03-10 21:48:44 DEBUG : test.bin: Sending chunk 536870912 length 33554432
2019-03-10 21:48:48 DEBUG : test.bin: Sending chunk 570425344 length 33554432
2019-03-10 21:48:52 DEBUG : test.bin: Sending chunk 603979776 length 33554432
2019-03-10 21:48:55 DEBUG : test.bin: Sending chunk 637534208 length 33554432
2019-03-10 21:48:58 DEBUG : test.bin: Sending chunk 671088640 length 33554432
2019-03-10 21:49:01 DEBUG : test.bin: Sending chunk 704643072 length 33554432
2019-03-10 21:49:04 DEBUG : test.bin: Sending chunk 738197504 length 33554432
2019-03-10 21:49:07 DEBUG : test.bin: Sending chunk 771751936 length 33554432
2019-03-10 21:49:11 DEBUG : test.bin: Sending chunk 805306368 length 33554432
2019-03-10 21:49:14 DEBUG : test.bin: Sending chunk 838860800 length 33554432
2019-03-10 21:49:17 DEBUG : test.bin: Sending chunk 872415232 length 33554432
2019-03-10 21:49:21 DEBUG : test.bin: Sending chunk 905969664 length 33554432
2019-03-10 21:49:24 DEBUG : test.bin: Sending chunk 939524096 length 33554432
2019-03-10 21:49:27 DEBUG : test.bin: Sending chunk 973078528 length 33554432
2019-03-10 21:49:30 DEBUG : test.bin: Sending chunk 1006632960 length 33554432
2019-03-10 21:49:34 DEBUG : test.bin: Sending chunk 1040187392 length 33554432
2019-03-10 21:49:39 INFO : test.bin: Copied (new)
Transferred: 1G / 1 GBytes, 100%, 9.756 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 1m44.9s
2019/03/10 21:49:39 INFO :
Transferred: 1G / 1 GBytes, 100%, 9.756 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 1m44.9s
2019/03/10 21:49:39 DEBUG : 4 go routines active
2019/03/10 21:49:39 DEBUG : rclone: Version "v1.46" finishing with parameters ["rclone" "--config=/etc/rclone/rclone.conf" "--drive-chunk-size" "32M" "-vv" "-P" "copy" "test.bin" "gdrive:"]
calisro
March 10, 2019, 9:36pm
4
Multiple files would likely be faster. What would you expect as far as upload for a single file?
I normally can get a pretty nice speed up with a single file.
rclone copy 1G.file gcrypt:deleteme.file -P --drive-chunk-size 32M
Transferred: 1000.244M / 1000.244 MBytes, 100%, 30.288 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 33s
I’m also gigabit Verizon.
calisro
March 10, 2019, 11:32pm
6
More than I get but my pipe is smaller. I do max out my 100 megabit (around 12 Megabytes). The op has a gigabit so I’m surprised it’s considerably less than yours.
alfred
March 11, 2019, 9:10pm
7
I have the same problem, only upload at 6 MB/s per file.
Jammie
March 11, 2019, 9:22pm
8
Yeah, I would expect something more than 10MB/s. Is there something else I can do to determine my bottleneck?
seuffert
March 11, 2019, 11:42pm
10
Ipv4 vs Ipv6 might play a role in this. Are you sure both tests (webui & rclone) use the same?
calisro
March 11, 2019, 11:46pm
11
latency may also play a role. You could be connecting to different end points with the gui vs rclone. What is your ping to the Google end points? Also verify you’re not having any retries/dropped packets.
The Web downloads and v2 / v3 api definitely use different end points as I’ve checked that before myself as well.
The web downloads are always much faster from what I’ve seen as well.
The last issue I saw with a Linux agent was related back to a kernel bug.
Jammie
March 12, 2019, 8:33am
13
@Animosity022 I’m running Arch Linux. My kernel is Linux alfred 5.0.0-arch1-1-ARCH #1 SMP PREEMPT Mon Mar 4 14:11:43 UTC 2019 x86_64 GNU/Linux
@seuffert I only have ipv4 enabled on my network.
@calisro Which endpoints in particular? Can you give me some IPs or URLs?
Endpoint is just what you connect to. I use something like ntopng and you can see my rclone API hit:
compared to my Web Download
Showing the end points are slightly different as I’m guessing one is for API traffic and one is for the web drive.
The only other thing I can think of is testing out on a different kernel and see your results. If I get some time today, I’ll spin up an arch VM and see if I can test as well.
I’m on a Fedora 29 spin atm.
[felix@gemini ~]$ uname -a
Linux gemini 4.20.14-200.fc29.x86_64 #1 SMP Tue Mar 5 19:55:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
and prior to that a Debian 9.5 install.
Jammie
March 13, 2019, 6:54pm
15
Yeah, I’m seeing the same thing on my end, www.googleapis.com and googleapis.l.google.com from rclone, and a different one from the web client.
I can’t imagine the kernel would have such a large affect, but maybe I’ll try one of the other kernels (linux-zen?)
You’d think that but the kernel bug was in 4 though and you are running 5.
I would find that pretty odd in the later one.
Sudden drop in download bandwidth during transfer was the old post related back to that.
I was going to try arch and see if I can reproduce it.
i got the same issue gbit speed up and down and only get 3-4 mb/sec upload and 50 mb/sec download so raring up files would make it much faster but is there any way we can un archive from google drive?
I’d recommend to start a new thread and list out your version / logs / etc.
