Ways how to improve upload speed of large single file with sync/copy

Help and Support
1

What is the problem you are having with rclone?

I've noticed that upload copy/sync of one single large file of size eg. 16GB is much slower than eg. Directory of size 16GB with 16 files. I would like to check if there is any way how to improve it. Myabe some additional flags/tweaks etc.

I'm using below flags:

--transfers 16 
--multi-thread-streams=16 
--multi-thread-cutoff=64M

"--transfers 16" nicely covers scenario of uploading/download huge dir with small and medium size files. All good.

"--multi-thread-streams and it's brother" nicely covers download of one large single file. All good as well.

However the scenario of single large file upload is kinda missing. I see there is single slow connection for rclone, no multi-threading. Are there any flags/tweaks that can be used here? Make it paralell? I'm using Koofr on Ubuntu 22.04

Run the command 'rclone version' and share the full output of the command.

rclone v1.62.2

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-75-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.20.2
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

Koofr

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone -P \
--transfers 16 \
--multi-thread-streams=16  \
--multi-thread-cutoff=64M \
--disable-http2 \
sync ./MY_DIR_WITH_ONE_LARGE_FILE  remote:SOME_DIR

The rclone config contents with secrets removed.

type = koofr
provider = koofr
user = *** redacted ***
password = *** ENCRYPTED ***

A log from the command with the -vv flag

2

hi,
without a debug log, hard to know what th exact issue.

looks like no multi-part upload and chunk size for koofr.
as compared to https://rclone.org/s3/#multipart-uploads

and based on this topic, might try --disable-http2
https://forum.rclone.org/t/multithread-upload-speed-slowed-down-to-single-thread-speed-for-koofr/35427

3

--disable-http2 is important to circumvent a bug in Go (which rclone uses) but AFAIR is only relevant to parallel transfers

The speed of a single thread mainly depends on your latency to Koofr servers*

	Ping (ms) - MBps
	 15       - 4.395  <-- Germany VPS
	 40       - 1.020  <-- VDSL
	 63       - 0.894  <-- Turkey VPS
	140       - 0.307  <-- VDSL (VMware VM, added 100 ms latency)
	240       - 0.241  <-- VDSL (VMware VM, added 200 ms latency)
	440       - 0.152  <-- VDSL (VMware VM, added 400 ms latency)

AFAIK there's not much else you can do than to reduce latency.
I never tried to split files though, maybe this produces several upload threads from a single file (in case you need no web access to said huge file).
.

.

.

* AFAIR IPs are
5.9.21.137
88.198.70.54
116.202.118.178
116.202.160.164
176.9.78.123
176.9.109.126

1 Like
4

Absolutely great input from your side! I'm using --disable-http2 and strugglining with getting single-stream performance to Koofr servers... On all IPs you mentioned I get good stable 20-25ms

I have fiber-optic 200+Mbit that works like a charm in most cases. Just the single transfer upload to Koofr seems slow...

In some Reddit post, there was mention of 3 ways how to test their speed to Koofr (that should be located in Falkenstain, Hetzner operator):
A TechnikNews Speedtest V2 - The most accurate internet check
B Speedtest - DELTA BARTH Systemhaus GmbH
C Downloading single big file from: http://fsn.icmp.hetzner.com/

In my case only B produces decent 60Mbit+, but A & C is 8-16 Mbit tops on my fiber-optic, I have no idea why. Of course other German speedtest locations produce like 50-100Mbit transfers. Even when I'm connecting via LTE I get on A like 50Mbit (probably hitting LTE's max throughput in my location), but on my fiber-optics, nope :frowning: And you can guess what my ISP replied when I asked about this, just a small hint, it was typical response :smiley:

Below link to that reddit post just for reference:
https://www.reddit.com/r/koofrnet/comments/10cfc9b/koofr_uploading_slow/

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.