High CPU load when uploading to 1fichier via mount

[dietera]

What is the problem you are having with rclone?

High CPU load (~ 55% of 8 cores AMD EPYC 2nd @2.5 GHz) with 1fichier and rclone mount whereas upstream to 1fichier is 1/4 of the possible speed (tested with 1fichier speedtest) which can be achieved.

What is your rclone version (output from rclone version)

rclone v1.53.2
- os/arch: linux/amd64
- go version: go1.15.3

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Ubuntu 20.04.1 LTS

Which cloud storage system are you using? (eg Google Drive)

1fichier

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount 1fichier: /mnt/1fichier --daemon --vfs-cache-mode=full --vfs-cache-max-age=24h --allow-other --dir-perms=0777 --umask=0002 --gid=115 --uid=1001 --allow-non-empty --cache-dir=/mnt/to1

The rclone config contents with secrets removed.

[1fichier]
type = fichier
api_key = hidden

[ncw (Nick Craig-Wood)]

Can you try doing an upload not via the mount so, rclone copy file.bin 1fichier: and see if it uses the same amount of CPU.

I suspect it is the whirlpool hash 1fichier uses - it is extremely CPU intensive

Here is MD5 for comparison

$ time rclone hashsum MD5 /tmp/1G
cd573cfaace07e7949bc0c46028904ff  1G

real	0m2.286s
user	0m2.014s
sys	0m0.643s

And here is whirlpool

$ time rclone hashsum Whirlpool /tmp/1G
14a633081e96e68e2e32dfb1bea8865371d2538b10e90b4d2245e31c51af50aa0b1e69c253bb84515e81b008d9b6fe143ce2c6e9267e999dfc2b15004f783be2  1G

real	0m31.098s
user	0m30.852s
sys	0m0.776s

So my laptop can do 1G / 31.098s = 32 MByte/s for whirlpool per core.

Can you try this test on your machine with a large file?

[dietera]

Thank you for the fast answer. I will test your suggestion when my current transfer is completed.

So first I tested the hashing algorithms ~6.4G file:

~# time rclone hashsum MD5 OHO-052.mkv
4775edcc79568e3af856ad36364ff744  OHO-052.mkv

real    0m21.511s
user    0m19.841s
sys     0m6.328s
~# time rclone hashsum Whirlpool OHO-052.mkv
97d4ad4b82289b0c1a5345e9e52552555a508dd3a6c78127b649eb0bba685ac45f41a20edce0c39329870038d25dfc6ab9b000c48077d25d68c633ae9e0f6fbb  OHO-052.mkv

real    5m0.289s
user    4m58.262s
sys     0m10.719s

As you suspected the whirlpool hashing algorithm is very slow in comparsion to MD5.

I also tested rclone copy and that uses (all processes together) ~200% CPU at ~200 Mbit/s with the same file.

When I try an upload via the website of 1fichier I get ~550 Mbit/s using one connection.

[asdffdsa (jojothehumanmonkey)]

@ncw

rclone hashsum Whirlpool

Whirlpool is not mentioned on the docs
https://rclone.org/commands/rclone_hashsum/

[ncw (Nick Craig-Wood)]

@dietera

So first I tested the hashing algorithms ~6.4G file:

That seems to be saying your computer can whirlpool hash at about 21.8 MByte/s so that caps your upload rate to about 175 Mbit/s

@dietera

I also tested rclone copy and that uses (all processes together) ~200% CPU at ~200 Mbit/s with the same file.

Sounds about right!

@dietera

When I try an upload via the website of 1fichier I get ~550 Mbit/s using one connection.

I suspect uploading via the website doesn't compute the whirlpool hash. This means that you won't know if there has been data corruptions.

If you add

  --ignore-checksum   Skip post copy check of checksums.

Then that will stop rclone checking the checksum so it should increase the speed.

[ncw (Nick Craig-Wood)]

@asdffdsa

Whirlpool is not mentioned on the docs
https://rclone.org/commands/rclone_hashsum/

You are right, this is the current list

$ rclone hashsum
Supported hashes are:
  * MD5
  * SHA-1
  * Whirlpool
  * CRC-32
  * DropboxHash
  * MailruHash
  * QuickXorHash

Want to send a fix?

[asdffdsa (jojothehumanmonkey)]

sure, i will do that.

[dietera]

@ncw

I suspect uploading via the website doesn't compute the whirlpool hash. This means that you won't know if there has been data corruptions.

If you add

  --ignore-checksum   Skip post copy check of checksums.

Then that will stop rclone checking the checksum so it should increase the speed.

Thank you for the advice. I will test this flag and check, if it runs correctly with my setup. I'll let you know if I can find out anything better

[dietera]

The --ignore-checksum flag definitely works and now I get 500-550 Mbit/s Upstream to 1fichier even via rclone mount. So I think I am going to use this in the future.
Thank you very much!

[asdffdsa (jojothehumanmonkey)]

@ncw

rclone hashsum

hope this is what you wanted?
https://github.com/rclone/rclone/issues/4841

[ncw (Nick Craig-Wood)]

@dietera

The --ignore-checksum flag definitely works and now I get 500-550 Mbit/s Upstream to 1fichier even via rclone mount. So I think I am going to use this in the future.

Great - glad that works

[ncw (Nick Craig-Wood)]

@asdffdsa

hope this is what you wanted?
https://github.com/rclone/rclone/issues/4841

Do you want to edit the docs?

They need to be edited here: https://github.com/rclone/rclone/blob/86014cebd7aef901f196e938c7ce07d6fe262ef4/cmd/hashsum/hashsum.go#L36

Click the pencil in at the top right of the text, make the edit and submit with a comment

docs: fix hashes list in hashsum documention - fixes #4841

[asdffdsa (jojothehumanmonkey)]

sure, i will do that soon.

[asdffdsa (jojothehumanmonkey)]

is this correct?

https://github.com/rclone/rclone/compare/master...asdffdsa6131:patch-3

[ncw (Nick Craig-Wood)]

Looks perfect @asdffdsa