# Multi-threaded uploads **URL:** \ **Category:** Help and Support\ **Created:** [November 30, 2019, 9:06pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166 "2019-11-30T21:06:42Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [November 30, 2019, 9:06pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/1 "2019-11-30T21:06:42Z")
Hello, I've researched how/if multi-threaded uploads are done, but haven't been able to fully understand what the current state is. Specifically, I'm using Google Drive and getting a max of 100-150mbps upload for a single file. I'm wondering how I could achieve more, perhaps by uploading multiple chunks simultaneously and combining them together at the remote somehow (what I mean by "multi-threaded uploads"). I've seen discussions about flags that could possibly control this, but none seem to have made it to production as of yet. Thanks in advance for information. ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [November 30, 2019, 9:10pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/2 "2019-11-30T21:10:23Z")
I can usually max out my gigabit with a few transfers. I use: ```auto /usr/bin/rclone move /local/ gcrypt: --log-file /opt/rclone/logs/upload.log -v --exclude-from /opt/rclone/scripts/excludes --delete-empty-src-dirs --fast-list --max-transfer 700G --drive-chunk-size=128M ``` I use the default checkers/transfers and the chunk size of 128M helps quite a bit. Here are my last few days of uploads: ```auto Transferred: 130.327G / 130.327 GBytes, 100%, 63.916 MBytes/s, ETA 0s Transferred: 149.769G / 149.769 GBytes, 100%, 92.211 MBytes/s, ETA 0s Transferred: 207.164G / 207.769 GBytes, 100%, 65.285 MBytes/s, ETA 9s Transferred: 207.783G / 207.783 GBytes, 100%, 65.107 MBytes/s, ETA 0s Transferred: 66.877G / 67.073 GBytes, 100%, 80.328 MBytes/s, ETA 2s Transferred: 67.075G / 67.075 GBytes, 100%, 79.994 MBytes/s, ETA 0s Transferred: 134.878G / 134.878 GBytes, 100%, 85.297 MBytes/s, ETA 0s Transferred: 93.174G / 93.174 GBytes, 100%, 80.573 MBytes/s, ETA 0s ``` ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [November 30, 2019, 9:11pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/3 "2019-11-30T21:11:08Z")
Sorry, I just updated my post to clarify that I'm referring to a single large file ---
**Author:** ![Harry](https://forum.rclone.org/letter_avatar_proxy/v4/letter/h/f6c823/32.png) [@Harry](https://forum.rclone.org/u/Harry)\ **Post date:** [November 30, 2019, 11:05pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/4 "2019-11-30T23:05:22Z")
I would also like to know about it. Sometimes I have to upload 80-90 GB single large file to Gdrive. ---
**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\ **Post date:** [December 1, 2019, 11:00am UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/5 "2019-12-01T11:00:18Z")
Multi threaded uploads depend on the cloud backend. Google drive doesn't support them unfortunately - you can only upload one chunk at a time as far as I know. Note the `--drive-chunk-size 128M` in animosities command line - that speeds things up a lot at the expense of memory. ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [December 1, 2019, 1:11pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/6 "2019-12-01T13:11:39Z")
I see, that's unfortunate. I've using `--drive-chunk-size 1024M` and get "only" 100-150mbps. ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [December 1, 2019, 1:26pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/7 "2019-12-01T13:26:12Z")
What about wrapping it with a chunker remote? Does that automatically allow multi-thread uploads? ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [December 1, 2019, 1:34pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/8 "2019-12-01T13:34:06Z")
Single file, 1G if you have the memory gives me: ![image](https://forum.rclone.org/uploads/default/original/2X/e/e12edad38ba3dbe23c9ab502f96433cb36c21d15.png) So about 380Mb/s. ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [December 1, 2019, 1:35pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/9 "2019-12-01T13:35:59Z")
Interesting, what's the exact command there? ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [December 1, 2019, 1:37pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/10 "2019-12-01T13:37:07Z")
You can actually see it all actually. I left everything at default and just added -P --drive-chunk-size 1G I used a copyto to test a single file. ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [December 1, 2019, 4:00pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/11 "2019-12-01T16:00:42Z")
After playing around for a bit it looks like encryption adds some overhead when it comes to upload speed. ---
**Author:** ![thestigma](https://forum.rclone.org/user_avatar/forum.rclone.org/thestigma/32/3414_2.png) [@thestigma](https://forum.rclone.org/u/thestigma)\ **Post date:** [December 1, 2019, 4:00pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/12 "2019-12-01T16:00:44Z")
> [@ctlaltdefeat](#): > > I see, that's unfortunate. > I've using `--drive-chunk-size 1024M` and get "only" 100-150mbps. While I haven't tested this extensively, I suspect that there might be some kind of maximum limit for chunk-size at 128M or 256M. At least looking at my bandwidth graph I can't seem to get the spacing between actual chunks transferred (judging by the graph at least) to be any wider at these high levels). I would suggest doing some testing on this yourself. Too high chunksize may have no effect or even a counterproductive effect. As I said - not certain about this due to lack of true in-depth testing - more research needed. Based on what I have seen from GCP VMs and feedback from others with much higher bandwidth than myself, there seems to be some sort of cap on the backend for individual transfers, but this should be closer to 40MB/sec pr transfer. And of course, since you can use many parallel transfers this is rarely much of an issue. Since I can't even cap out a single tranfer with my 160Mbit/sec at home, I've not taken the time to dig into this in-depth. @ncw A related curiosity however is that if you set the upload-threshold very high (to avoid chunking and thus spending no memory) the average pr-transfer speed goes way down - to more in the range of 40-80Mbit/sec. It is still something you can solve by paralell transfers of course, but I find this very curious why non-chunked uploads would behave so drastically different. Do you have any idea on what might cause this NCW? This is more of a point of curiosity than an rclone complaint. ---
**Author:** ![thestigma](https://forum.rclone.org/user_avatar/forum.rclone.org/thestigma/32/3414_2.png) [@thestigma](https://forum.rclone.org/u/thestigma)\ **Post date:** [December 1, 2019, 4:04pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/13 "2019-12-01T16:04:30Z")
It shouldn't - not unless you have a very slow CPU like a Pi or something. If so then just check the hardware status and see if it gets near to maxing out or temperature-throttling during upload. On half-decent desktop CPU from the last decade it should only take a few % of your CPU cycles to encrypt-as-you-go. (not remotely a problem even on my ancient 2500K) Once the CPU has encrypted the bits rclone or google doesn't care if the data is encrypted or not, so there should be no reason why encrypted data should be transferred any slower than anything else. ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [December 1, 2019, 4:38pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/14 "2019-12-01T16:38:13Z")
That's really on CPU overhead and very minimal. I'm tested copying to an encrypted mount. ---
**Author:** ![ctlaltdefeat](https://forum.rclone.org/letter_avatar_proxy/v4/letter/c/96bed5/32.png) [@ctlaltdefeat](https://forum.rclone.org/u/ctlaltdefeat)\ **Post date:** [December 1, 2019, 7:25pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/15 "2019-12-01T19:25:05Z")
So having done testing on the chain of my setup (gdrive -\> cache -\> encryption -\> mount), it turns out that it basically all rests upon the `vfs-cache-mode` flag of the mount. Having it set to `writes` and above yields expected performance, but setting it to `minimal` or `off` causes almost exactly a 2x time increase for copying to the mount. My test is as follows: `rclone mount --config ~/.config/rclone/rclone.conf --drive-chunk-size 256M --vfs-cache-mode writes --cache-chunk-no-memory gdrive_encrypted:/ mount_dir/ ` (Although `gdrive_encrypted:/` point to a cached remote as mentioned, I suspect it doesn't matter in this case) `head -c 1GB /dev/zero > test.bin` `time cp test.bin mount_dir/` This takes around 20-25 seconds, which corresponds to the ~40MB/s (~350mbps) figure you guys are getting. On the other hand, running the above mount command with `--vfs-cache-mode off` results in the copy taking 45-55 seconds. Can you reproduce this? Is there a way to fix this? I wouldn't want to incur the overhead of essentially doubling the amount of storage I need to account for large files at peak, as I'm working with limited storage. ---
**Author:** ![Animosity022](https://forum.rclone.org/user_avatar/forum.rclone.org/animosity022/32/451_2.png) [@Animosity022](https://forum.rclone.org/u/Animosity022)\ **Post date:** [December 1, 2019, 7:45pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/16 "2019-12-01T19:45:59Z")
I don't use my mount for writing as I use rclone move instead. I don't use the mount to write as it's just unneeded overhead in my setup. ---
**Author:** ![Harry](https://forum.rclone.org/letter_avatar_proxy/v4/letter/h/f6c823/32.png) [@Harry](https://forum.rclone.org/u/Harry)\ **Post date:** [December 2, 2019, 8:32am UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/17 "2019-12-02T08:32:31Z")
> [@ncw](#): > > Note the `--drive-chunk-size 128M` in animosities command line - that speeds things up a lot at the expense of memory. Does this command comes in effect if I am uploading to Google Drive via Crypt remote? ---
**Author:** ![ncw](https://forum.rclone.org/user_avatar/forum.rclone.org/ncw/32/18_2.png) [@ncw](https://forum.rclone.org/u/ncw)\ **Post date:** [December 2, 2019, 11:51am UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/18 "2019-12-02T11:51:23Z")
> [@thestigma](#): > > A related curiosity however is that if you set the upload-threshold very high (to avoid chunking and thus spending no memory) the average pr-transfer speed goes way down - to more in the range of 40-80Mbit/sec. It is still something you can solve by paralell transfers of course, but I find this very curious why non-chunked uploads would behave so drastically different. Do you have any idea on what might cause this NCW? This is more of a point of curiosity than an rclone complaint. I don't think it is anything rclone is doing - I suspect it must be rate limiting at the google end. ---
**Author:** ![thestigma](https://forum.rclone.org/user_avatar/forum.rclone.org/thestigma/32/3414_2.png) [@thestigma](https://forum.rclone.org/u/thestigma)\ **Post date:** [December 2, 2019, 6:52pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/19 "2019-12-02T18:52:14Z")
Oh I expect it is something on Google's end too. I just wanted to check if you knew anything about the "why" of this because it is such a curious result. I don't see any obvious reason for why chunks would be treated differently than a continuous stream. My best (totally baseless) speculation is that maybe direct streams were just not intended to very large files and thus maybe they go direct into final storage while chunks are cached somewhere server-side on faster disks. Who knows 🙂 It is nice to be aware of though - especially when dealing with low-RAM systems. ---
**Author:** ![system](https://forum.rclone.org/uploads/default/original/2X/3/39ad3235f6ffec76ccd0c81616bd55697b4c300d.png) [@system](https://forum.rclone.org/u/system)\ **Post date:** [March 1, 2020, 6:52pm UTC](https://forum.rclone.org/t/multi-threaded-uploads/13166/20 "2020-03-01T18:52:15Z")
Closed on March 1, 2020, 6:52pm UTC This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.