Rclone cache speed (test with Disk Speed Test by Blackmagic)

anon13086576 · March 21, 2021, 10:07pm

What is the problem you are having with rclone?

I'm using rclone to mount a Synology DS1821+ on my MacbookPro over my wifi network. I want to use it to mount remote photo library and get advantage of the rclone cache to work with Capture One.

If I run BlackMagic disk speed test on a local folder I can get impressive speed results (~2000MB/s read/write).
I've been curious to test it on a remote tmp folder and I see slow results (~40MB/s read/write).

I'm wondering if there is something I can improve to match the read/write speed of my ssd for the files that are cached.

I was convinced that caching worked a bit like Dropbox, i.e. giving the possibility to read/write to disk, and that rclone in the background would take care of the sync "unhurriedly"

Thanks for any suggestion on how to improve.

What is your rclone version (output from `rclone version`)

 ~/ rclone --version
rclone v1.54.1
- os/arch: darwin/amd64
- go version: go1.15.8

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Mac OSX 11.2.3 Big Sur

Which cloud storage system are you using? (eg Google Drive)

Synology DS1821+ in SHR2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone mount magnum:/ magnum --volname magnum --vfs-cache-mode full --log-level=DEBUG --log-file=rclone.log

The rclone config contents with secrets removed.

 ~/ cat .config/rclone/rclone.conf 
[magnum]
type = sftp
host = 192.168.1.2
user = fabio
port = 22
pass = xxx
key_file_pass = xxx
md5sum_command = none
sha1sum_command = none

A log from the command with the `-vv` flag

link to rclone.log on pastebin that will expire in 30 days: where you see [...] I've cut the log with duplicated lines. I can paste the complete one if needed.

Thanks.

ncw · March 22, 2021, 10:33am

Can you try an experiment?

Make an rclone mount of a local directory,

rclone mount /path/to/local/dir magnum --volname magnum --vfs-cache-mode full --log-level=DEBUG --log-file=rclone.log

And see what speed you get with that, with --vfs-cache-mode full and --vfs-cache-mode off - that will tell us where the bottleneck is hopefully.

anon13086576 · March 22, 2021, 11:00am

Hi @ncw
thanks for helping.

In both cases, I get only 40MB/s...

Logs:

rclone-cache-full.log
rclone-cache-off.log

Edit:

in the rclone-cache-full.log I just notice there is the first log I sent. The new one starts at 11:50.

ncw · March 23, 2021, 3:42pm

OK so this looks like it is FUSE being slow somehow.

The next thing to try would be the latest beta which has a reworked OSX FUSE interface - you should have the latest macfuse/osfuse 4.x to go with it.

anon13086576 · March 23, 2021, 4:17pm

thanks @ncw,
I've downloaded this version clone-v1.55.0-beta.5332.1406b6c3c-osx-amd64.zip and the lates osxfuse 4.1.0 version.
No improvements so far, ad with cache off is even worst (like 5MB/s...)

Logs:

ncw · March 23, 2021, 5:01pm

Can you try it just to local disk so we can compare it with the speeds you posted before?

anon13086576 · March 23, 2021, 7:34pm

you are right @ncw sorry, I didn't think of that.
No change so far, in both cases, still around 40MBb/s
Here the logs for the local mount testing:

ncw · March 24, 2021, 9:54am

I did a bit more research...

Try the local test with this additional parameter -o iosize=33554432 - that increases the buffer size to 32M which might make a difference.

anon13086576 · March 24, 2021, 10:25am

hi @ncw I tried this 2 commands:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --vfs-cache-mode full -o iosize=33554432 --log-level=DEBUG --log-file=rclone155beta-local-iosize.log 
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o iosize=33554432 --log-level=DEBUG --log-file=rclone155beta-local-iosize-only.log

no improvements so far, still around 40Mb/s. Let me know if there is something else I can test.

Here the logs:

https://drive.google.com/file/d/1YZTr7d3TkOoWwJ-3COimM2FE7sKorHXi/view?usp=sharing

https://drive.google.com/file/d/1WpXRz3fpJR1woJkLbmrn0gMMEFbBKher/view?usp=sharing

ncw · March 24, 2021, 10:48am

Looking at your log all the io is still being done in 4k chunks.

Can you try some different power of 2 sizes for iosize - ie some of these?

Try 16777216, 8388608 and 131072

You could also try experimenting with this value (without iosize)

  --max-read-ahead SizeSuffix              The number of bytes that can be prefetched for sequential reads. Not supported on Windows. (default 128k)

If you look through the log you'll see

$ grep 'Read: n' rclone155beta-local-iosize-only.log
2021/03/24 11:21:40 DEBUG : /tempAAA/DiskSpeedTestTemp: >Read: n=4096
2021/03/24 11:21:40 DEBUG : /tempAAA/DiskSpeedTestTemp: >Read: n=4096
...

You'll see all the Reads are 4096 - you want them to be bigger!

Likewise the Writes

$ grep 'Write: n' rclone155beta-local-iosize-only.log
2021/03/24 11:21:34 DEBUG : /tempAAA/DiskSpeedTestTemp: >Write: n=4096
2021/03/24 11:21:34 DEBUG : /tempAAA/DiskSpeedTestTemp: >Write: n=4096
...

anon13086576 · March 24, 2021, 11:36am

Looking at your log all the io is still being done in 4k chunks.

not sure, but this is probably due to the fact the Blackmagic Speedtest is stressing the disk writing/reading small files of that size.

I made the following tests, hoping I got well your suggestions:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o iosize=131072 --log-level=DEBUG --log-file=rclone155beta-local-iosize-131072.log
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o iosize=16777216 --log-level=DEBUG --log-file=rclone155beta-local-iosize-16777216.log
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o iosize=8388608 --log-level=DEBUG --log-file=rclone155beta-local-iosize-8388608.log
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --max-read-ahead 4k --log-level=DEBUG --log-file=rclone155beta-local-max-read-ahead-4k.log
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --max-read-ahead 128k --log-level=DEBUG --log-file=rclone155beta-local-max-read-ahead-128k.log 
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum  --log-level=DEBUG --log-file=rclone155beta-local-tranfer-huge-file.log

I wanted to do a different test then try to move a folder on the mounted folder it took around 4 minutes to move it:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --vfs-cache-mode full --log-level=DEBUG --log-file=rclone155beta-local-huge-file.log

The same folder moved directly on the Desktop/Tmp in less than 1minute:

Here the folder with all the logs above if you want to check (the last one is ~700MB). The chunk size is still 4096.

https://drive.google.com/file/d/1BR4T5oF-wDNkmpzTRenaRt0lKFWVOLlW/view?usp=sharing
https://drive.google.com/file/d/1gGOMnTYDMWi876qjsGMukeGWGr8QsFAy/view?usp=sharing
https://drive.google.com/file/d/10AxjamZvY9mExAxnItuUnttvNknnaidR/view?usp=sharing
https://drive.google.com/file/d/1TJZv02IHAG9wuC15cS8jll3jtkQT21Jh/view?usp=sharing
https://drive.google.com/file/d/1cAttdMNOS_iK7VChlqnQsbH2Orfh43CD/view?usp=sharing

and the huge folder moving log:

https://drive.google.com/file/d/1F-iLRRbRe6iWCc-5ilcWsoBH8Y3Z_Gyn/view?usp=sharing

Thanks,
Fabio

ncw · March 24, 2021, 4:28pm

I managed to replicate this on Linux. I've tried a lot of things to fix it but haven't succeeded yet. I've asked for help on the cgofuse issue tracker.

anon13086576 · March 24, 2021, 9:49pm

oh wow...this is a serious one then!
Looking forward to see the next update on this thread

And thanks for all the support.

ncw · March 25, 2021, 9:23am

Can you try with the option of -o direct_io and see if that makes a speed difference?

Thanks

anon13086576 · March 25, 2021, 11:19am

Hi @ncw, I tried with these 2 commands on the local folder:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o direct_io --log-level=DEBUG --log-file=rclone155beta-local-direct-io.log  

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --vfs-cache-mode full -o direct_io --log-level=DEBUG --log-file=rclone155beta-local-direct-io-cache-full.log

And no improvements, still around 40Mb/s. Here the logs:

darthShadow · March 25, 2021, 12:50pm

Can you try with -o big_writes too once?

anon13086576 · March 25, 2021, 1:32pm

hi @darthShadow ,

I tried this command on the local disk; let me know if this is what you are suggesting:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum --vfs-cache-mode full -o big_writes --log-level=DEBUG --log-file=rclone155beta-local-direct-big-writes.log

And this is the related log:

ncw · March 25, 2021, 2:17pm

That is still showing 4k writes... So I guess it was the same speed?

-o big_writes makes a lot of difference in Linux.

Can you try a test using cp to copy a big file?

darthShadow · March 25, 2021, 2:53pm

Ok, finally got it working on my Mac. Looks like osxfuse has an explicit blocksize option that is missing in libfuse on Linux.

Adding -o blocksize=1048576 should get it working. I didn't have any luck increasing it further than that.

anon13086576 · March 25, 2021, 3:57pm

hi @darthShadow can you share your command? I'm not lucky.

With the following command I got an error (see screenshot attached) and Blackmagic is not able to write the test files.

This is the log:

Last login: Thu Mar 25 12:15:43 on ttys000
 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o blocksize=1048576 --log-level=DEBUG --log-file=rclone155beta-local-blocksize.log