log with --drive-upload-cutoff 100G
seems to be slow than previous test, only 30mbits, so 3 or 4MB/s
Am I right in assuming that this has to fit the entire uploaded file into memory? Or is that only when chunking?
Because if the chunking is mainly there to prevent you from having to upload tens of GB in case of a transmit error and you don't need to fit the whole thing into memory then I will definitely play around with that. My fiber connection is pretty stable so I think that would be a worthwhile tradeoff up to a significant size.
I've though about this before but never ran a test on it...
That graph shows a very typical TCP-rampup. TCP starts slow then accelerates up until it hits you max.
However, usually this should happen within a few seconds at most.
Your sawtooth pattern look very drawn out (unless this is very zoomed in?)
This may indicate a high latency to the server. Higher roundtrip latency will make the TCP ramp up speed slower.
Larger chunks or no chunks at all should help remedy this, but it's weird you are not getting good results from that.
Does anyone know if there is a way to get an estimate of the ping-time to the google server your drive is talking to? This could potentially be a case where you live in an area of the world where the nearest google datacenter is far away, the countrys infrastructure is not so good, and/or the drive you are accessing is just not meant to be really used from your area. I'm not really sure how Google deals with regionality for Gdrive and if they get hosted in a spesific region depending on the info of the person registering. I know Google cloud has many settings spesifically to configure this.
i'm in italy, my fiber is vodafone ftth 1000/200 and via browser i have full speed,so i guess is something in rclone that needed to be setup
Unfortunately, there's nothing in rclone that fixes Internet routing / latency. You have to address that with a VPN/ changing data centers / etc.
What chunk size did you use when you took the screenshot of that graph?
If it is a large latency issue we may not be able to fix that, but using as large chunks as possible (or even better - no chunks) will hide the problem as much as possible.
with vpn change nothing, is worste than without vpn. Maybe because routing or there are a lot of people that using vpn to download and upload, so are very full load.
Is is my isp, via browser should be slow upload, but it isn't, is high, only with rclone is slow.
About googleapis.com i don't understand what i must do, i open it and blank page says, "not found"
@thestigma abiut chunk size, i use --drive-chunk-size" "512M"
Yea that is very abnormal. It should be staying at the top of the curve about 95% of the time then, but you barely hit 200Mbit before it drops down again (next chunk), indicating a very slow ramp-up. It is possible that a "--dump headers" on a log would be able to tell us the approximate response time on requests even if we can not ping the server directly.
I can try analyzing it for you if you want to provide one.
log with --dump headers part 1
log with --dump headers part 2
log with --dump headers part 3
I believe in you 
thanks a lot for the support and patience, @thestigma @Animosity022 @ncw
note. i guess to deleted all sensible data in this logs. feel free of course to notice me, if i forgot something.
Hmm, I don't see anything too obviously wrong. I forgot we don't have sub-second timestamps in the log so I can only really say that the latency is less than a second - which is not all that helpful 
I think I may be a little stuck for ideas at this point, sorry - but I am inclined to believe that the problem you are experiencing is not coming from the rclone software itself.
something in my router ?
fritz box 7490 ?
That would one possibility, yes. Load balancing, filtering and other functions like this could under certain circumstances cause weird behaviors - especially is not configured properly.
Is this your own router you bought? Did you get a standard one for your ISP company also? If so I would recommend trying the standard one with default setting just temporarily (even better if you can test directly via cable) and seeing if the result is the same. If the result is the same then you can at least eliminate the router as a source of problems. This is basically the only part of the routing you can do anything about anyway.
my isp use another router, i'will try with it, but fritz box 7490 is more powerful and better.
Yea I don't doubt that (I heard it's a good router) - but this is just to eliminate a problem. Not being powerful enough is not the issue here.
If you find that the problem goes away with the stock router then that just means that you have to look at the configuration you are using on the Fritzbox.
Hi have tried with isp router but change nothing.
And is very very strange because are two weeks, that speed via browser is slow and have the same issue like rclone ( for me ), so the network graph, is the same, peak and down, and of course the speed is slow.
- I have tried to change ip, restart my fritz box, no luck.
- I have tried speed test and i have full bandwidth.
- But only if i try to upload with rclone or for example another software ( if i can say the name, i i'll post here ) a lot of files, like 5, or 8 in the same time, i can get my full upload speed.
I have tried also with NordVPN, but in down, i can get with some certains servers, 500Mbits, so good, but in upload, i have tried a lot of country and seems that vpn, to google or also with speed test, are very slow, like 20Mbits, so isn't impossible, know the true.
Seems that isn't my isp, because with speed test, seems that is all ok, and i can get my full upload speed with a lot of files ( big files of course), so my isp doesn't filtrate bandwidth to google.
Maybe is google ? I have tried with another google account and same story.
Or seems that Single Threat upload bandwidth, have some trouble ? possible fix it ?
The sawtooth graph is unavoidable if the file are small, as each new file will have to start a new connection - however, using many concurrent uploads may help offset this somewhat.
If the --drive-chunk-size is small it will happen on large files too, but if it is set large (like 128M for example) it should normally be minimal.
TLDR: Best to test with relatively large files (50-100MB or more) as that is where performance SHOULD be good. Small files performance on Gdrive is known to not have great performance.
My best guess is that you have a lot of latency to the google server, and thus your TCP speed ramps up very slow - leading to much worse sawtoothing than normal. The best I can suggest to alleviate this is to use a reasonably large size drive chunk size and --transfers 8. This won't fix the underlying problem for you (which I don't know if we can fix honestly) but it will at least help alleviate some of the worst sawtoothing and help you maximize your bandwidth utilization.
I wish there was some way of pinging the actual server to confirm the underlying issue - but I am not aware of a way to do so...
with--drive-chunk-size 1G --drive-v2-download-min-size 200M and 12 files in upload, every files is 9GB, i upload only large files.
The issue is when finish some files and there are only a few files to complete, in this case, is slow 
My ping from every server isn't less than 25ms, my ftth anyway is in island, so maybe the cable under the sea, have more hop. But is strange that are two weeks that also via browser i have this issue.
Last mouth, via browser worked like a charm, now... not i'm very unlucky
You should also make some speedtests outside your country. Test a country in Europe (France, Spain and Germany) and to the US. That way you can check if there is a bottleneck between the network of your ISP and the peer networks of your provider, as Speedtest will always choose the nearest server too you (and you won't probably have that exact speed to servers far away from you)
At least seems to reasonably max out with many concurrent files, so that's something.
@ferferga Yes, this is something along the lines of what I suspect. Some areas of the world have really poor connectivity to other regions (Australia was plagued by this for a long time). It is entirely plausible that google might have a website access point inside the network, but that the closest direct API connection is outside - thus leading to poor performance. in that regard.
A ping and speed test to a server in the nearest large city on the continental mainland will probably give a good indication of any such problems. Especially any large mainland city in the US or Europe.
1 Like