Gdrive faster transfer speed?

Help and Support
1

What is the problem you are having with rclone?

I am getting very slow transfer speeds..around 2mb/s..

Some things about my setup:

  • I am on a cable (no wifi)
  • Using Gsuite Business
  • Download speed is around 55 mb/s
  • Upload speed is around 19 mb/s
  • Have setup google client and secret to get my own connection to google
    Since I am uploading to gdrive i should be able to have faster speeds than 2 since I have 19..
    I have been checking around the forum for similar cases and tried a few things.. but I hope that I am overlooking something..

What is your rclone version (output from rclone version)

v1.50.2

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac OSX Mojave 10.14.6

Which cloud storage system are you using? (eg Google Drive)

Google Drive

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone copy /Volumes/AkitioRAID/VIDEO_Auto_Raid2/ gdrive:/VIDEO_AutoG/ --fast-list --transfers=10 --checkers=10 --tpslimit=10 --drive-chunk-size=128M --log-file=/Users/ken/Documents/rclone/2019-12-14-4.log -P -vv

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2019/12/15 00:00:59 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "copy" "/Volumes/AkitioRAID/VIDEO_Auto_Raid2/" "gdrive:/VIDEO_AutoG/" "--fast-list" "--transfers=10" "--checkers=10" "--tpslimit=10" "--drive-chunk-size=128M" "--log-file=/Users/ken/Documents/rclone/2019-12-14-4.log" "-P" "-vv"]
2019/12/15 00:00:59 DEBUG : Using config file from "/Users/ken/.config/rclone/rclone.conf"
2019/12/15 00:00:59 INFO : Starting HTTP transaction limiter: max 10 transactions/s with burst 1
2019/12/15 00:01:01 INFO : Google drive root 'VIDEO_AutoG': Waiting for checks to finish
2019/12/15 00:01:01 DEBUG : 18-Div/20190122_1529_video_43571676911.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : .DS_Store: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : .DS_Store: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 18-Div/20190122_1529_video_43571676911.mov: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 18-Div/video_43070040145.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : 190709-LTYF-Ian-Ken-Talk/.DS_Store: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : 190709-LTYF-Ian-Ken-Talk/.DS_Store: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 18-Div/video_43070040145.mov: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 18-Div/video_29882753418.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : 18-Div/video_29882753418.mov: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 190806-Jini-victim-interview-and-Jen-Ken/.DS_Store: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : 190806-Jini-victim-interview-and-Jen-Ken/.DS_Store: Unchanged skipping
2019/12/15 00:01:01 DEBUG : 18-Div/video_43975543651.mov: Size and modification time the same (differ by 0s, within tolerance 1ms)
2019/12/15 00:01:01 DEBUG : 18-Div/video_43975543651.mov: Unchanged skipping
2019/12/15 00:01:01 INFO : Google drive root 'VIDEO_AutoG': Waiting for transfers to finish
2019/12/15 00:01:03 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1430_P1322079.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:05 DEBUG : 190623-UK-Misc/20190622_2031_P1011664.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190706-JinisHorses-Canada/20190706_2059_P1011678.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1625_P1322080.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1630_P1322081.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190706-JinisHorses-Canada/20190706_2059_P1011677.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1634_P1322082.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1900_P1322085.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1728_P1322084.MOV: Sending chunk 0 length 134217728
2019/12/15 00:01:07 DEBUG : 190819-NewEden-Relationship-Week-Holland/20190819_1726_P1322083.MOV: Sending chunk 0 length 134217728

2

19 Mbit/s is approximately 2 Mbyte/s. When you write mb/s, do you mean Mbit or Mbyte? rclone will display byte unless you specify otherwise using --stats-unit.

1 Like
3

I understand now, so I got stuck in a conversion thing here.. 19 Mbit/s is around 2 Mbyte/s... so that is the limit then... my upload speed.. :neutral_face: bugger..

1 Like
4

Yes, exactly. You can use --stats-unit bits and it'll show you 19 Mbit/s as your upload speed instead.

1 Like
5

To convert

  • divide Mbits by 8.
  • Then subtract roughly 12-13% (this is TCP transfer overhead aproximately)

So for example 20Mbit/sec would be 20/8 * 0.88 = 2.5 * 0.88 = 2.2Mbytes/sec.
This is what you can realistically expect to see in rclone under optimal conditions (if you had 20Mbit/sec). All measurements and settings in rclone generally use Mbytes as a unit rather than Mbits.
Note: In some countries with strong consumer protections, speeds are advertised including overhead (ie. practical maximum). In other (most) countries they advertise the technical gross speed (which is unachievable in practice due to overhead). As an example, I have a 150Mbit fiber here in Norway, but in reality it's something like 165Mbit.

You already use 128M chunks, which is close to the ideal for uploading (only).
Downloads already use 128M segments, so there's not much to optimize there.
Note that this only helps on larger files anyway (larger than 8MB compared to the defaults).
Also note that very many very small files will always transfer fairly slowly on a Gdrive. This has little to do with bandwidth, but is because Gdrive will only allow about 2-3 new transfers to start each second - so on very small files this will bottleneck you a bit. (the same reason that more than 4-5 transfers isn't really viable). Usually you need much more expensive premium services to get higher allowances in this area.

Gdrive can generally upload/download around 40-45MBytes/sec pr transfer on as many transfers as you can keep going, so bandwidth isn't a limiting factor unless you start to talk about well in excess of 1Gbit/sec speeds (at which point it's really a luxury problem hehe).

2 Likes
6

Nice to hear from a fellow Norwegian :grinning:
Internet speeds there are much better than in the UK that's for sure, I think I need to move again!
Thanks for the added details :slight_smile:

God jul :evergreen_tree: (Merry Christmas in Norwegian)

2 Likes