Can you upload to an rclone mount without caching?

You can use that flag i mentioned, but if you want it to be a permanent default it's best to set it in the config.

To know where you config is for the current user - run in CMD
rclone config file
This file can be opened in any plan text editor like notepad ect. (do not recommend Word and such due to potential formatting troubles)

As I said, that needs to be 128M
(meaning 128MegaBytes)
if you use no postfix it assumes you are talking about kilobytes, which is to low to even be accepted as a value.

Please note, with 128M chunks you can use up up 128MB RAM pr transfer stream (4 by default).
So 4x128 = 512MB max for example.
Adjust as needed if you have low ram. 64MB is also pretty close in performance but at half the cost.

Yeah, sorry about that. I corrected it and nothing changes as far as upload speed is concerned.

Can you do a copypaste of that transfer for me please? (you can abort it with CTRL+C after a minute or so)
(to copy from CMD, mark the text with the mouse and either right-click mouse or CTRL+C)

Perhaps I can spot something obvious .

with s3 and wasabi, i do
--s3-chunk-size=64M
and
--s3-upload-concurrency=10

perhaps you can convert that to gdrive flags

On Gdrive you'd have basically the same with chunk-size 64M and --transfers 10 (although for Gdrive going above 5 is rather pointless due to backend hardlimits).

I think something else fundamental is the problem here has usually Gdrive will do way more on a Gigabit connection. For it's limitations, one thing Gdrive does not lack is bandwidth capacity.

"My upload, as measured by a reliable speedtest is 15MB/s"

does he has a gigabit connection or me?

C:\Users\an531\Desktop\Rclone\rclone-v1.51.0-windows-amd64>rclone copy "B:\file 1" gdrive:\ -P --drive-chunk-size 128M
Transferred:      118.684M / 2.261 GBytes, 5%, 2.498 MBytes/s, ETA 14m39s
Transferred:            0 / 1, 0%
Elapsed time:        47.5s
Transferring:
 *                                    101.7z.006:  5% /2.261G, 2.517M/s, 14m32s

My connection is FTTH (I get fiber to an optical terminal in my house from my ISP and from there ethernet cable to the router), DOWNLOAD = 500 megabits per second and UPLOAD = 100 megabits per second. The connection is asymmetric.

Oh ops! I guess I skimmed too fast.
Well if he can do 15MB/sec on a speedtest (assuming he didn't mean 15Mbit/sec of course) then he can do 15MB/sec in Gdrive easily a long as it's not bunches of tiny files.

EDIT: although from his speedtest results, that is a 100Mbit connection, to the theoretical max on that would 12,5MB/sec - probably a bit less in practice from various overhead, but in that general ballpark.

What is the B: source in this case?

I ask because B: is not a typical letter for physical harddrive on Windows, so I just want to make sure it's not something else, like another mount or something along those lines.

B:/ is another internal SATA II SSD, which is currently filled with files only. I changed the drive letter to B:/ in the beginning, because I wanted to install W10 there. I have W7 x64 installed on my C:/, which is what I'm using now. I get the same results from C:/.

what about the router?

i have found that not all routers do well with fiber.
in fact, recently, i uses a netgate sg-1100 router, running pfsense, and it could not handle my internet connection.
yet when i replaced that an old used computer running pfsense and a dual nic card, that could push my fiber connection to it limits.

Very unlikely to be an issue here as long as he can get his speed on speedtest.
Running 5 concurrent connections isn't going to break his router :slight_smile:

If the router was the limiting factor, then I don't think that I would be getting 11MB/s when uploading through the browser (Chrome). According to speedtest, my upload is about 15MB/s, but I managed to measure only 11.7MB/s when uploading via Chrome or (multiple files) Filezilla... still this beats 2.3MB/s that I'm getting with rclone now.

Have you created your own client-ID for Gdrive?
Or have you used the default for now?

I'd also like to understand better your location. Eastern europe or western? Some eastern europe locations I know from feedback have sometimes had issues with bad routing (ie. they can achieve great speeds if routed via a VPN, but connection is high-latency directly).

China and some other Asian locations seems even worse in this aspect.

And as I said - the web-interface is a completely different system, so the routing to it can be completely different. This is not a diagnosis I'd want to jump to unless we have eliminated all other reasonable causes because it is hard to solve properly yourself (outside of routing traffic via VPN).

On a 100Mbit upload like it looks like you have, your theoretical max (gross) is 12,5MB/sec, so a bit under this is what you could expect to get - but yes, you are obviously not getting this in rclone right now. I have no problem saturating by full connection (about 33MB/sec) from Norway just to give you an idea, and I've seen Gdrive handle a full gagabit many times before under ideal circumstances.

ok, ok , so it might not be the router...

how about this?

Google Application Client Id
Setting your own is recommended.
See https://rclone.org/drive/#making-your-own-client-id for how to create your own.
If you leave this blank, it will use an internal key which is low performance.

True - it's something I would always recommend.
But it should not affect bandwidth at all - only API. Once the transfers are going it would have to be extremely overcrowded to affect the resulting speed.
Possible - but very unlikely.

ok. good to know

what about this? can i use default?

Google Application Client Secret
Setting your own is recommended.
Enter a string value. Press Enter for the default ("").
client_secret>

@an531 could you fo another test for me where you transfer at least 5 files 64MB or larger?
also add the --transfers 5 flag.

The results of that test (ie. if it is better or worse) will help indicate what part of the system is bottlenecking.

1 Like

If you use default you get rclone's default API client - which Nick admins.
This is much much larger capacity than a normal user's (though with the same max pr user) - but it is also shared between all users who use the default. Nick can monitor this via google metrics to ensure he can request more quota from Google as needed.