Rclone speed issue

Help and Support
1

What is the problem you are having with rclone?

I've got Nextcloud runing on a VPS. The files are stored on swift backend using rclone mount. That works pretty fine, but I'm facing speed issues. Let's explain.
On my home, I have a NAS and the VPS runs a script to backup the NAS to the swift backend via Nextcloud. This is the command I use :

rclone sync -c -P --transfers 10 --log-file /root/Documents/Oliver_save/transfert1.log Oliver:etc/ Nextcloud:etc/

When I run this command on my VPS, the speed transfer is about 330ko/s, which is very small.

On my house, I have an hypervisor. When I build a ubuntu VM and run the exact same command, the speed is about 1.8Mo/s, which is better but still not enough. My ISP real speed is 300Mbps, VPS real speed is 200Mbps so I hope I could transfer at ~15Mo/s.

Rclone version is the same on both side (VPS and hypervisor), see below.

An other information : on my hypervisor, I've rebuild the VPS instance with Nextcloud connected to the swift backend the exact same way. There is no more VPS between my home and the backend, and the speed is about 12Mo/s, which is what I expect.

Run the command 'rclone version' and share the full output of the command.

rclone v1.60.1

  • os/version: ubuntu 22.04 (64 bit)
  • os/kernel: 5.15.0-56-generic (x86_64)
  • os/type: linux
  • os/arch: amd64
  • go/version: go1.19.3
  • go/linking: static
  • go/tags: none

Which cloud storage system are you using? (eg Google Drive)

I use swift backend for the storage for Nextcloud, but not sure if it is important for this config. webdav is use for Nextcloud frontend, sftp for my NAS

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone sync -c -P --transfers 10 --log-file /root/Documents/Oliver_save/transfert1.log Oliver:etc/ Nextcloud:etc/

The rclone config contents with secrets removed.

[swift]
type = swift
user = xxx
key = xxx
auth = https://api.pub1.infomaniak.cloud/identity/v3
tenant = xxx
auth_version = 3
domain = Default
region = dc3-a

[Oliver]
type = sftp
host = oliver.xxx.fr
user = ulysse132
port = 2406
pass = xxx
use_insecure_cipher = true
md5sum_command = md5sum
sha1sum_command = sha1sum
shell_type = unix

[Nextcloud]
type = webdav
url = https://save.xxx.fr/remote.php/dav/files/ulysse132/
vendor = nextcloud
user = ulysse132
pass = xxx

Could you help me improve the transfer speed please ?

2

Hi Eric,

If I understand it correctly, then there are only performance issues when some/all data pass through Nextcloud on the (remotely hosted) VPS. That is the performance issue disappears when running the exact same Nextcloud setup in the (locally hosted) VM.

It therefore seems like there are no issues with Swift, NAS, Nextcloud or rclone, since we see optimal performance from all of them when the load/traffic is moved from the VPS to the VM.

The speed issue therefore seems like to be caused by some (currently unknown) difference between the VPS and the VM. It could be available resources (CPU, RAM, Disk), internet/network connections, setup of Nextcloud, or setup of the VPS itself. (I assume VPS and VM both use ubuntu 22.04)

The suspicion is strengthened by the observation that the transfer speed increases the more the load/traffic is moved from the VPS to the VM (0.3 -> 1.8 -> 15 MB/s)

Not that this analysis/summary is very helpful (yet), but is it correctly understood, and does it match your picture of the situation?

3

I and thank you for the reply.

It seems you're right but it strange, because when I run speed test on the vps, the speed is 200mbps.
Nextcloud and nginx are deployed through docker so I'm 100% sure they are the same.
I don't know what to check to find the speed issue. Any ideas?

4

I guess you already have used top on the VPS and within the docker contains to verify sufficient free CPU, RAM etc. during the transfers.

My next step would therefore be to check the transfer speeds one leg at the time with a minimum of extra software (docker, Nextcloud, etc.).

That is first test transfer speed from SFTP to VPS (outside docker) by executing something like this on the VPS:

rclone sync -P --transfers 10 --ignore-times Oliver:etc/ /some/temp/folder/ --log-file transfert2a.log

then test the transfer speed from VPS (outside docker) to Swift by executing something like this on the VPS:

rclone sync -P --transfers 10 --ignore-times /some/temp/folder/ swift:some/temp/folder --log-file transfert2b.log

If OK, then I would try executing the same two tests from within the Nextcloud container.

If OK, then I would try the first command within the Nextcloud container, but use a folder within the swift mount as target.

If OK, then I would try this command from the VPS (outside the docker):

rclone sync -P --transfers 10 --ignore-times /some/temp/folder/ Nextcloud:some/temp/folder --log-file transfert2e.log

I mainly work on Windows, so please excuse me if some of the above isn't possible/fully clear - I still hope you can use it as inspiration.

5

Ok, found something. The problem is between Oliver (my NAS) and the VPS. From the VPS to the swift backend through Nextcloud is full speed (about 20Mo/s)

I've tried setting Oliver via webdav, not via sftp. Speed is much better (about 10Mo/s).

So the problem is related to the protocol. SFTP is slow compared to webdav. But still no ideas why...

6

Perfect, what happens if you double --transfers to 20 on the sftp transfer? Does it roughly double speed?

7

I can't say for now. For my test, I only transfer 1GB file. It's one shot for one file only. I'll do more test this weeks, with more files at once.

8

I found this ressource :
https://support.cerberusftp.com/hc/en-us/articles/203333215-SFTP-transfer-speed-why-so-much-slower-than-FTP-or-FTPS-

I think this explain well why this is so slow. I switched to webdav in my scripts now.

9

I am happy we found the reason and a good work around for your issue!

However not sure your workaround will work as well for everybody. I searched the web and was unable to find any tests supporting a general speed difference between webdav and sftp (including high latency connections). So probably a matter of your NAS having hardware better equipped for webdav encryption (https) - or are you using webdav over the internet without encryption (http)?

10

Only https everywhere outside my network

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.