Increase single upload speed of 20gbps seedbox ssd to Google Drive

Currently getting speed's of 60MB/s using this command " rclone move ~downloads/completed/ gdrive: -v --drive-chunk-size=512M

However my deluge torrent upload seed is nearly 380MB/'s . Is there a command you can post here . So i can use to increase single file upload speed, I was expecting 100MB/s+ with rclone . My box is with feralhosting

Thanks all

You are kind of comparing an apple to orange though.

Seeding a torrent is going to multiple people at the same time, while you are uploading to a single spot via a single process.

The main bottleneck seems to be what Google provides via the API. There has been multi part downloads implemented, but I'm not sure if Google supports multi part uploads.

If you are getting 60MB for a file, that's a pretty solid speed from what I've seen for Google API. You can do more files to get better bandwidth but that doesn't meet your question though.

@ncw - Does Google Drive support multi part uploads?

I'm pretty sure multi-part uploads aren't supported on the cloud-side. Mostly because you can only act on files - and in order to do a multipart transfer you either need to transfer the pieces into a per-allocated space (which requires drive-level access below the file level) or at least combine the files together at the end (which would require computation). I don't think this can be done until such a time that the API provides new features along that line - and I wouldn't expect that to happen anytime soon.
The reason we can do multi-part downloads is because we have much more control over the local filesystem.

The only workaround I could see would be to have the software that orders the move to happen to support multiple consecutive transfers itself - thus increasing bandwidth (hopefully) with more transfers going at once. If the software you use doesn't already have an options like that you might go ask for it is it is open software that takes requests.

And of course you generally have to be aware that speed will be limited by file-sizes, or rather many small files (of trivial transfer time) will cause a significant slowdown due to the rate-limiting on amount of files you can update (about 2/sec) - plus of course the general latency involved in starting each transfer. In the case of torrents that in practical tems that anything packed in multipart archives will tend to be slower than for example a movie that is not packed.

Sure they are. A number of them do multi part uploads. I wasn't sure if Google Drive did though

OP's question was increasing speed on one transfer as well.

I think you want to request multi part upload as it does exist for Google Drive, but I do not think it's implemented yet.

https://developers.google.com/drive/api/v3/manage-uploads#multipart

I was speaking within the context of Gdrive - you need the API to support it, or else you won't have the tools you'd need to implement a function like that yourself. Good you clarified though.
I haven't seen any option to facilitate multi-part uploads on Grive yet (why I am the most familiar with).

The link is very interesting. If the API has that function then it seems like it would solve the issue if implemented into the Gdrive backend. @lightning25 this is probably worth making an issue about on github, and include that link. I'm pretty sure this does not exist currently but I will scan the docs to make sure.

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.