Big files are slow to open

What is the problem you are having with rclone?

Hi, I'm using rclone as an SSD cache for my HDD array (yeah, it's a weird way to use rclone), it works great! But I'm having a little issue with big files.

My setup is: rclone mount -> samba -> my desktop PC to access the files
The operating system is Unraid

The parameters I'm using are:
rclone mount --cache-dir localcache/ --vfs-cache-mode full --vfs-read-ahead 128M --allow-other /mnt/disk1/ /mnt/rclonecache/

When I open a 1GB file, it hangs on "openPending" for about 10 seconds before starting to read the file (localcache directory stays empty, so it's not even pre-caching it).
The issue is only with big files, small files loads up almost instantly.

I've tried several options, including changing the buffer size, and the --no-checksum flag.
Any help would be appreciated, thanks in advance!

A log from the command with the -vv flag

2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: Open: flags=OpenReadOnly+OpenNonblock
2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: Open: flags=O_RDONLY|0x800
2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: newRWFileHandle: 
2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: >newRWFileHandle: err=<nil>
2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: >Open: fd=musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac (rw), err=<nil>
2022/03/21 12:28:21 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: >Open: fh=&{musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac (rw)}, err=<nil>
2022/03/21 12:28:22 DEBUG : &{musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac (rw)}: Read: len=131072, offset=0
2022/03/21 12:28:22 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac(0xc0008c15c0): _readAt: size=131072, off=0
2022/03/21 12:28:22 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac(0xc0008c15c0): openPending: 
2022/03/21 12:28:32 DEBUG : /: Attr: 
2022/03/21 12:28:32 DEBUG : /: >Attr: attr=valid=1s ino=0 size=0 mode=drwxrwxrwx, err=<nil>
2022/03/21 12:28:32 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: Attr: 
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: vfs cache: checking remote fingerprint "1983554435,2019-02-08 16:16:26.2152148 +0000 UTC,3b0065b464608a6855a25e7af6cac0ff" against cached fingerprint "1983554435,2019-02-08 16:16:26.2152148 +0000 UTC,3b0065b464608a6855a25e7af6cac0ff"
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: vfs cache: truncate to size=1983554435
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac: >Attr: a=valid=1s ino=0 size=1983554435 mode=-rw-rw-rw-, err=<nil>
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal: Added virtual directory entry vAddFile: "Sempiternal.flac"
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac(0xc0008c15c0): >openPending: err=<nil>
2022/03/21 12:28:34 DEBUG : vfs cache: looking for range={Pos:0 Size:131072} in [{Pos:0 Size:156229632} {Pos:1983524864 Size:29571}] - present true
2022/03/21 12:28:34 DEBUG : musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac(0xc0008c15c0): >_readAt: n=131072, err=<nil>
2022/03/21 12:28:34 DEBUG : &{musica/musica/Bring Me The Horizon/Sempiternal/Sempiternal.flac (rw)}: >Read: read=131072, err=<nil>
...

Run the command 'rclone version' and share the full output of the command.

rclone v1.58.0
- os/version: slackware 14.2+ (64 bit)
- os/kernel: 5.10.28-Unraid (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.8
- go/linking: static
- go/tags: none

Which cloud storage system are you using? (eg Google Drive)

none, I'm using local storage

The command you were trying to run (eg rclone copy /tmp remote:tmp)

rclone mount --cache-dir localcache/ --vfs-cache-mode full --vfs-read-ahead 128M --allow-other /mnt/disk1/ /mnt/rclonecache/

The rclone config contents with secrets removed.

Haven't used `rclone config`, so the file is empty.

This is causing the problem - it is rclone checking the hash of the remote file against that of the local file to see if they are identical and the file needs re-transferring.

Calculating a hash for a big file takes time.

The Fingerprint function has two modes, a fast mode and a slow mode. Fast being less accurate and slow being what it is using at the moment.

This only makes a difference in backends which set the SlowHash or SlowModtime feature flag.

Currently SlowHash is set by local and sftp and SlowModTime is set by ftp, qinqstore, s3 and swift.

Any other backend won't be affected.

I was my intention to use fast fingerprints for the VFS backend originally, but at some point I decided accuracy was more important.

Maybe it should be a user configuration.

In your case wrapping a local backend, the size and modification time are pretty much guaranteed to get updated if the file is changed.

As an experiment I made a --vfs-fast-fingerprint flag to control this. If you set it, you should find opening large files is much quicker. Note that setting this flag will invalidate all the files that are already in the cache as the fingerprints will no longer match. I need to write some docs about it but I'll do that if you tell me the feature works and is useful!

v1.59.0-beta.6038.08830e1fc.fix-vfs-fast-fingerprint on branch fix-vfs-fast-fingerprint (uploaded in 15-30 mins)

1 Like

I just tried it and the speed definitely improved! It's much better now, I think it can be useful if you work with big files

Thanks for testing.

I've merged this to master now which means it will be in the latest beta in 15-30 minutes and released in v1.59

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.