What is the problem you are having with rclone?
I've noticed that upload copy/sync of one single large file of size eg. 16GB is much slower than eg. Directory of size 16GB with 16 files. I would like to check if there is any way how to improve it. Myabe some additional flags/tweaks etc.
I'm using below flags:
"--transfers 16" nicely covers scenario of uploading/download huge dir with small and medium size files. All good.
"--multi-thread-streams and it's brother" nicely covers download of one large single file. All good as well.
However the scenario of single large file upload is kinda missing. I see there is single slow connection for rclone, no multi-threading. Are there any flags/tweaks that can be used here? Make it paralell? I'm using Koofr on Ubuntu 22.04
Run the command 'rclone version' and share the full output of the command.
- os/version: ubuntu 22.04 (64 bit)
- os/kernel: 5.15.0-75-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.20.2
- go/linking: static
- go/tags: none
Which cloud storage system are you using? (eg Google Drive)
The command you were trying to run (eg
rclone copy /tmp remote:tmp)
rclone -P \
--transfers 16 \
sync ./MY_DIR_WITH_ONE_LARGE_FILE remote:SOME_DIR
The rclone config contents with secrets removed.
type = koofr
provider = koofr
user = *** redacted ***
password = *** ENCRYPTED ***
A log from the command with the
without a debug log, hard to know what th exact issue.
looks like no multi-part upload and chunk size for koofr.
as compared to https://rclone.org/s3/#multipart-uploads
and based on this topic, might try
--disable-http2 is important to circumvent a bug in Go (which rclone uses) but AFAIR is only relevant to parallel transfers
The speed of a single thread mainly depends on your latency to Koofr servers*
Ping (ms) - MBps
15 - 4.395 <-- Germany VPS
40 - 1.020 <-- VDSL
63 - 0.894 <-- Turkey VPS
140 - 0.307 <-- VDSL (VMware VM, added 100 ms latency)
240 - 0.241 <-- VDSL (VMware VM, added 200 ms latency)
440 - 0.152 <-- VDSL (VMware VM, added 400 ms latency)
AFAIK there's not much else you can do than to reduce latency.
I never tried to split files though, maybe this produces several upload threads from a single file (in case you need no web access to said huge file).
* AFAIR IPs are
Absolutely great input from your side! I'm using --disable-http2 and strugglining with getting single-stream performance to Koofr servers... On all IPs you mentioned I get good stable 20-25ms
I have fiber-optic 200+Mbit that works like a charm in most cases. Just the single transfer upload to Koofr seems slow...
In some Reddit post, there was mention of 3 ways how to test their speed to Koofr (that should be located in Falkenstain, Hetzner operator):
A TechnikNews Speedtest V2 - The most accurate internet check
B Speedtest - DELTA BARTH Systemhaus GmbH
C Downloading single big file from: http://fsn.icmp.hetzner.com/
In my case only B produces decent 60Mbit+, but A & C is 8-16 Mbit tops on my fiber-optic, I have no idea why. Of course other German speedtest locations produce like 50-100Mbit transfers. Even when I'm connecting via LTE I get on A like 50Mbit (probably hitting LTE's max throughput in my location), but on my fiber-optics, nope And you can guess what my ISP replied when I asked about this, just a small hint, it was typical response
Below link to that reddit post just for reference:
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.