Very slow speed for google drive

Hi,

I just configured a new google drive remote and added client_id and client_secret.

I then tested by copying a large (about 750MB) file like so:

rclone copy -vvv georgien_konzert.mp4 gdrive: --drive-chunk-size 64M

I get this output:

2020/04/19 14:12:12 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "copy" "-vvv" "georgien_konzert.mp4" "gdrive:" "--drive-chunk-size" "64M"]
2020/04/19 14:12:12 DEBUG : Using config file from "/home/mh/.config/rclone/rclone.conf"
2020/04/19 14:12:12 DEBUG : georgien_konzert.mp4: Need to transfer - File not found at Destination
2020/04/19 14:12:13 DEBUG : georgien_konzert.mp4: Sending chunk 0 length 67108864
2020/04/19 14:13:12 INFO :
Transferred: 20.652M / 755.382 MBytes, 3%, 354.902 kBytes/s, ETA 35m19s
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 1, 0%
Elapsed time: 59.5s
Transferring:

Is this the normal speed you can expect?

What could I try to make it better?
What I ultimately want is to use this as a backend to restic but with speeds like this there is no point.

I am running on Debian testing, kernel 5.5.0 and rclone 1.50.2.

Many thanks!

No. That's not a normal upload speed. You have either a bottle neck somewhere or a poor pairing with Google. Or something odd like a ip6 issue or network errors.

It's best to use the question template for any questions.

You are using a bleeding edge kernel so I'd avoid that and use something stable.
You'd want to grab the latest version and use that.
You'd want to post a full transfer log and see what's going on.
If you are mainly transferring large files, you can increase that to 128/256/512/1024M depending on your system specs.

I've tried several chunk-size settings, however the speed does not seem to change...

What would be the point of a "full transfer log"?

All you get is the same stanza for every chunk transmitted no?

What can I do to investigate this?

The point is that it helps trouble shoot the issue.

Producing a full log would take an hour.
Is there any information printed at the end that would be useful?
Because it seems to me it's just the same output for each chunk transmitted, so I would let it run for an hour for no extra information...

It all depends on how much you want to solve the issue.

The more information you can provide, the more help folks can give.

Without seeing the log, it's just a guessing game.

Use a stable kernel was my suggestion earlier.

I've created a full log now (for a slightly smaller file), as I expected there is nothing there that could not be gleaned from just looking at the first stanza.

Here the beginning and the end:

2020/04/20 08:17:25 DEBUG : rclone: Version "v1.50.2" starting with parameters ["rclone" "copy" "-vvv" "testfile.mp4" "gdrive:" "--log-file=out" "--drive-chunk-size" "128M"]
2020/04/20 08:17:25 DEBUG : Using config file from "/home/mh/.config/rclone/rclone.conf"
2020/04/20 08:17:25 DEBUG : gdrive: Loaded invalid token from config file - ignoring
2020/04/20 08:17:26 DEBUG : gdrive: Saved new token in config file
2020/04/20 08:17:26 DEBUG : testfile.mp4: Need to transfer - File not found at Destination
2020/04/20 08:17:26 DEBUG : testfile.mp4: Sending chunk 0 length 134217728
2020/04/20 08:18:25 INFO  :
Transferred:       21.152M / 218.811 MBytes, 10%, 364.767 kBytes/s, ETA 9m14s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 1, 0%
Elapsed time:       59.3s
Transferring:
 *                                  testfile.mp4:  9% /218.811M, 359.316k/s, 9m23s

2020/04/20 08:19:25 INFO  :
Transferred:       41.684M / 218.811 MBytes, 19%, 357.546 kBytes/s, ETA 8m27s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 1, 0%
Elapsed time:     1m59.3s
Transferring:
 *                                  testfile.mp4: 19% /218.811M, 330.007k/s, 9m9s


(and so on)


2020/04/20 08:27:25 INFO  :
Transferred:      208.590M / 218.811 MBytes, 95%, 356.360 kBytes/s, ETA 29s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            0 / 1, 0%
Elapsed time:     9m59.3s
Transferring:
 *                                  testfile.mp4: 95% /218.811M, 359.878k/s, 29s

2020/04/20 08:27:57 DEBUG : testfile.mp4: MD5 = 183f78cb6831a02c7b2daac6a027c625 OK
2020/04/20 08:27:57 INFO  : testfile.mp4: Copied (new)
2020/04/20 08:27:57 INFO  :
Transferred:      218.811M / 218.811 MBytes, 100%, 355.008 kBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:    10m31.1s

2020/04/20 08:27:57 DEBUG : 5 go routines active
2020/04/20 08:27:57 DEBUG : rclone: Version "v1.50.2" finishing with parameters ["rclone" "copy" "-vvv" "testfile.mp4" "gdrive:" "--log-file=out" "--drive-chunk-size" "128M"]

So all I can try is a different kernel? C'mon...

That seems to be a snippet of the log and not the full log.

A general tip as when you have someone using their time to help you out with a problem, why fight them?

Can you just post the full log?

I asked for the kernel because if you search the forums, there are numerous issues with the newer kernels due to async reads in the new kernels. Those tend to show up in the full logs and if not, it is another thing that can be ruled out.

I'm assuming when the transfer is happening, you have sufficient bandwidth on the system and you've checked for CPU/Disk/Memory items already as well.

My general speeds are maxing out my gigabit using rclone/Google drive when I uploaded:

Transferred:   	   54.635G / 54.635 GBytes, 100%, 86.967 MBytes/s, ETA 0s
Transferred:   	  246.518G / 246.518 GBytes, 100%, 88.866 MBytes/s, ETA 0s
Transferred:   	  228.247G / 228.247 GBytes, 100%, 91.020 MBytes/s, ETA 0s

Those are my last 3 days as an example. I use primarily defaults minus a very large drive-chunk-size of 1024M since I have a machine with 32G of memory on it and no constraints.

You are also using 1.50.2 and you can upgrade to 1.51 and produce a log.

If you want a list of "things you can look at".

Did you check network errors? traceroute? See if you have intermittent issues along the network route? Are you hosting on a VPS? Perhaps you're being throttled by your or your VPS provider? Perhaps try a VPN? Hardcode some IPs mapping too google servers and try different ones. Use a different DNS provider (like google or cloudflare) to see if you get different results? Are you okay on memory? Disk i/o? CPU?

Ok. That's the kitchen sink - and it's not helping.

We've evidently reached the stage of blah-blah.

I am not on a vpn, I am already using 8.8.8.8 for DNS, I have run "top" while it was running and did not see any bottlenecks.

So that's it then for me. I like rclone to access my google drive now and then. But for higher throughput it seems to be completely unpredictable.

For some people it seems to work, for me it does not.

Whatever.

I don't mean to sound insensitive. But if you ask for help, you kind of need to be ready to provide information necessary for people to help you. The things I mentioned do help in figuring out what is wrong and where the bottle-neck is. No one knows what configuration you have or what you're running. If you'd like help, then most folks on here would be glad to help.

I was asking you to actually 'try' a vpn.

That isn't neccessarily a good or bad thing. The question is around peering here.

Top gives you CPU resources. It doesn't tell you a lot of information on much else like if your disk is busy. Perhaps you have a I/O bottleneck. We don't know.

In my observation its pretty predictable. Its the OS/network/Google API that can be unpredictable.

I don't see any detail you've actually given anyone to help. All we know is you run a command and you're not happy and you're running version 1.50.2 without a vpn and with google DNS.

2 Likes

I have tried using a vpn now but the rates don't change.

Could you please explain why you considered that worth to try?

As far as the peering is concerned - I suppose there is not much I can do?

And I had a look at iotop while it was running - the rclone process shows 0.02% in the IO-column.

So if my kernel is the problem should I see problems will all backends?

If I get a B2 account and see the same poor perfomance there would that be an indication that indeed my kernel is the problem?

And what kernel version would be worth trying?

I am no longer considering using rclone with gdrive as a backend for restic but I would be curious as to what the problem is here...

Update to 1.51
Create a full debug log.
Share the log.

Without seeing logs, it's hard to say 'all' as we aren't sure the root cause is.

Me too as that's why I'd like to see a full log.

Bypassing traffic shaping. QOS. If it was targeting https or rclone.

Not a lot. You can change your dns and see if you get different ip targets for the API or potentially hardcode them in your hosts file.

Can you run traceroutes to the Google api ip and see if you're seeing network issues?

Are you on a vps? Or on your home network? If your on your own network is there another computer (perhaps even different enough OS) you can use to see if you see the same low performance?

Do you see higher speeds using other programs like curl? You can try:

 curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.