That looks like just part of the log and it’s mainly just checking files that are already there.
Small files are particular slow as you can only create 3 per second.
Best to get a full log of what you are trying to do as that has the starting command and the final output.
I did let it run for over an hour… it’s just so damn slow that it didn’t accomplish much. If I gave you the whole log for the whole upload, it would take 600 years before it’s done. Any suggestions on how to get you some useful information so you can see if I’m doing something dumb?
As far as sizes of files go… they aren’t very small. Most are about 800mb, some are 25mb, a few are smaller in the 1-2mb range.
You can run a smaller set of data through and produce a log.
You can test a 100MB file or something:
dd if=/dev/urandom of=file.txt bs=1048576 count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.559296 s, 187 MB/s
rclone copy file.txt --drive-chunk-size 32M GD: -P
Transferred: 100M / 100 MBytes, 100%, 22.145 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 4.5s
Thanks again for your response. I made 2 tests using your method… one going to an encrypted remote and the other using a non-encrypted remote. They were both very similar (kind of what I expected) but I just wanted to make sure.
Here is the encrypted remote’s log:
Here is the non-encrypted remote’s log:
Yeah, normally sending one file, you’d expect to see it max out at 20-25MB/s.
rclone copy file.txt --drive-chunk-size 32M GD: -P -vv
2019/04/07 11:12:32 DEBUG : rclone: Version "v1.46" starting with parameters ["rclone" "copy" "file.txt" "--drive-chunk-size" "32M" "GD:" "-P" "-vv"]
2019/04/07 11:12:32 DEBUG : Using config file from "/opt/rclone/rclone.conf"
2019-04-07 11:12:33 DEBUG : file.txt: Couldn't find file - need to transfer
2019-04-07 11:12:33 DEBUG : file.txt: Sending chunk 0 length 33554432
2019-04-07 11:12:34 DEBUG : file.txt: Sending chunk 33554432 length 33554432
2019-04-07 11:12:35 DEBUG : file.txt: Sending chunk 67108864 length 33554432
2019-04-07 11:12:36 DEBUG : file.txt: Sending chunk 100663296 length 4194304
2019-04-07 11:12:37 INFO : file.txt: Copied (new)
Transferred: 100M / 100 MBytes, 100%, 21.470 MBytes/s, ETA 0s
Errors: 0
Checks: 0 / 0, -
Transferred: 1 / 1, 100%
Elapsed time: 4.6s
I’d start peeling the onion back and looking from your server out and checking your internet speeds and whatnot. You are sticking within the same region as well? Copying US server to US GD? Peering is always tough to figure out what’s going on as well.
Thanks for confirming. I have no idea where to start with peering issues, I’ll have to dig into that tomorrow or later… one thing that really sucks is that I know it’s google, or the path there… when I run speedtest-cli from the SAME server, same network, same path… I get about 12MB upload.
Peering just means how your server/location connects to Google. You can’t do a thing about it really.
If you have a quality VPN, try it on it.
Well, this sucks. With privateinternetaccess, I get about 100k max. I tried several hop off sites… 100k was the max, most were around there but some were as bad as 50k.
Then you may be able to rule out isp. I use PIA and can get just as good speed across it as I can direct.
So weird… I assumed google flagged my account for actually trying to use my storage space. I just signed up for a business gsuite account last week, on a new email address, new everything… and it’s slow on that and my regular google drive that I’ve been using for ages. So, if it is an account flag, why would it be like that from the start with my new account? That’s a rhetorical question, I don’t expect you to know… but as a paying customer, there should be a way to get assistance. I contacted the gsuite helpdesk. What a joke. They were nice but weren’t able to give me ANY assistance except for “rclone is a 3rd party application” and “please use chrome to upload from incognito mode.” Of course, neither of those are acceptable.
If anyone has any additional ideas, please let me know.
Did you try directly from your modem/router with a computer wired in?
I have not tried bypassing my router. I’ll look at doing that some evening this week. HOWEVER, I don’t know why I didn’t think of this before… but I tried with a remote at onedrive and another at opendrive. They are both the same speed as Google. Same with over a VPN. These pieces are not fitting together. If I had to guess with the evidence I have now, it’s my router or my ISP and more heavily leaning on the router… but it doesn’t make any sense to me since my upload speeds everywhere else are fine (unless my ISP is screwing with my speed tests).
That’s pretty goofy in general.
The v3 API is just standard HTTPS traffic that goes out so it would be hard to screw with just that albeit, it is possible.
What’s the ISP?
I have comcast. No other options around here. However, I just noticed that rclone is using megaBYTES for data transfer when the industry standard is megaBITS. So, when it says I’m getting 1.5 megaBYTES, it equals to 12 megaBITs. Speedtest.net shows me maxing out at 12.5 megaBITs.
I think I may be chasing fake speeds that aren’t possible. I should probably open a feature request or bug report to ask for megaBITs in the output display of rclone.
Oh man.
The difference of reading MB/S and Mb/s.
Yes, that would do it to. Both are standard to report as it just depends if it’s a big B or a little b.
yeah, suck. Typically, when you are moving data, you do it in megabits, if you’re storing it, it’s in megabytes. it gets confusing when you are moving stored data.
It just depends as drive transfer speeds are normally MB/s and if this is more like a drive.
I can see both sides but happy you have the issue identified!
Hi,
I just encountered this very similar problem today, when I wanted to backup
photos/videos to my google cloud storage (not gDrive).
I am located in France, with a very good connection (optic fiber)
PING 1 ms DL 889.86 Mbps UL 604.27 Mbps
Yesterday around 10-11 PM, I wanted to backup a few GBytes of data on my GCS, which I remembered to be quite fast
But I could not go past a few kB/s. Here are a few attempts, that I aborted.
Transferred: 54.095M / 6.377 GBytes, 1%, 429.644 kBytes/s, ETA 4h17m15s
Transferred: 26.117M / 6.356 GBytes, 0%, 405.568 kBytes/s, ETA 4h32m46s
Transferred: 26.622M / 6.030 GBytes, 0%, 190.115 kBytes/s, ETA 9h11m56s
I tought my client Id was the problem, so I did setup one, but performance almost the same.
I decided to try the next morning, and I was surprised that everything was back to normal:
Transferred: 5.870G / 5.870 GBytes, 100%, 67.773 MBytes/s, ETA 0s
Transferred: 3.327G / 3.327 GBytes, 100%, 66.591 MBytes/s, ETA 0s
Transferred: 2.324G / 2.324 GBytes, 100%, 63.791 MBytes/s, ETA 0s
Maybe throttling from my ISP?
Best to open a new topic and use the question template if you are having an issue.