Rclone cache speed (test with Disk Speed Test by Blackmagic)

So, per the logs, atleast the writes are bigger now. Can you try adding --debug-fuse & --async-read=false in addition to the same command and try again?

Something like this:

rclone mount ~/Desktop/tmp magnum --volname magnum -o blocksize=1048576 --async-read=false --debug-fuse --log-level=DEBUG --log-file=rclone155beta-local-blocksize.log 

hi @ncw,
yes you are right, still 40Mb/s.

I made the test you suggested:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o big_writes --log-level=DEBUG --log-file=rclone155beta-local-big_writes.log

and this is the cp:

 ~/ time cp ~/Desktop/archive.zip magnum/tempAAA
cp ~/Desktop/archive.zip magnum/tempAAA  0.01s user 15.13s system 17% cpu 1:24.84 total

Archive.zip is a 3.1gb (3343531006) and to move it on a differente folder it takes less time:

 ~/ time cp ~/Desktop/archive.zip ~/Desktop/temporary 
cp ~/Desktop/archive.zip ~/Desktop/temporary  0.01s user 1.60s system 56% cpu 2.823 total

Here the logs:

Still no luck.

This is the command:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o blocksize=1048576 --async-read=false --debug-fuse --log-level=DEBUG --log-file=rclone155beta-local-debug-fuse.log 

The log:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o blocksize=1048576 --async-read=false --debug-fuse --log-level=DEBUG --log-file=rclone155beta-local-debug-fuse.log 

The error:

Can you try a normal cp once instead?

Looks faster!

Command:

 ~/ rclone mount ~/Desktop/tmp magnum --volname magnum -o blocksize=1048576 --async-read=false --debug-fuse --log-level=DEBUG --log-file=rclone155beta-local-debug-fuse-with-cp.log 

Copy time:

 ~/ time cp ~/Desktop/archive.zip magnum/tempAAA     
cp ~/Desktop/archive.zip magnum/tempAAA  0.01s user 2.97s system 19% cpu 15.475 total

And log:

I am guessing something BlackMagic does while testing doesn't agree with a pre-set block-size since I assume it sets it's own block-size and changes it for the various tests to find the best speeds.

If the cp is working fine, then I think the issue is solved or do you need BlackMagic to work?

Personally, I don't need blackmagic to work. I wanted only to test and see the rclone as a really advantage over monuting the nas on smb :slight_smile: The topic started for this reason. I'm doing other test that not involve Blackmagic.

I'll use

-o blocksize=1048576 --async-read=false

shall I use also the vfs cache parameter or not?

If you have the space for a local cache, then using full cache-mode is recommended. If not, then not specifying it should also work.

@darthShadow thanks, this is solving for me:

 ~/ rclone mount magnum:/ magnum --volname magnum --vfs-cache-mode full -o blocksize=1048576 --async-read=false

Using the Finder to move 3.7Gb file I had to try 5 times to get this screenshot:

rclone

It was quicker than me :smiley:

Amazing!

I'll continue testing and post an update later.

Kudos to you and @ncw for the help and support.

Great.

That was a tricky flag to find since it's not documented anywhere.

And look at this!

 ~/ rclone mount magnum:/ magnum --volname magnum --vfs-cache-mode full -o blocksize=1048576 --async-read=false --debug-fuse --daemon 

Mounting the nas folder, it works also blackmagic with very good speeds!

1 Like

hi @darthShadow @ncw

after some trial I noticed that the command shared before is not realiable.

Tying to explain some issues:

  • created a TMPAAA from my Mac on the folder mount via rclone
  • I moved a file from another pc on TMPAAA, but is not synced over rclone (I cannot see it on my mac)
  • I moved pdf (7948308bytes) and the file is corrupted (7946240bytes) (2068bytes difference)
  • I moved another file of 363946bytes and after is 360448 (3498 less)

I moved an empty file of 1MB that I created with this command:

mkfile -n 1m ~/Desktop/TESTFILE

and it this case size is the same (but this file contains just the same characters)

I copied Desktop/archive2.zip (3 343 531 006 bytes) to TMPAAA/archive2 (not the same size 3 343 527 936 bytes) but rclone is not able to sync in on the nas.
If I open the folder over SMB I can see the files being copied, but when the size reaches 200MB fails and restart. It looks like VFS fails the upload .

Before:
pdf-before

After:
pdf-after

Logs:

Are these failures repeatable?

Can you try without "-o blocksize=1048576" and see if you get the failure still? Maybe there is a reason that flag isn't documented...

hi @ncw I see what you mean; I was trying the -o flag because it was suggested in the thread. Without the flag works, but the speed is capped at 40Mb/s.

I currently switched to the stable 1.54 because I need to do some works a need reliability over speed. But I'll testing during this weekend.

If you want me to test something specific or if you want to have a call, I can screenshare live what I'm doing :slight_smile: I'm based in south of france (GMT+2)

1 Like

@ncw looking at new and previous logs I notice this message:

Statfs failed: your remote may not support About: failed to run "df -k /home": Permission denied, please try again.

Is this something that can prevent rclone to get info on my setup for example?

hi,
that problem is documented here
https://rclone.org/sftp/#limitations
"about will fail if it does not have shell access or if df is not in the remote's PATH."