Question: what is status of rclone support for concurrent caching

[boxerab]

Following up on this topic

Need advice on rclone usage for decompressing images over network

I develop an image codec for jpeg 2000, and I would like to decompress large GIS images (100s of GB or more) stored in cloud storage. I just set up rclone to map my remote service as a drive rclone mount minio:grok ~/miniogrok --vfs-cache-mode full --cache-dir ~/miniogrok_cache In the codec, I access files by memory mapping them and accessing data as a buffer. The file access is sequential but the data is sparse - codec will often skip parts of the file. I want to get optimal performance -…

I am interested in learning more about thread safety of rclone vfs caching.
For regular file system, it is possible for two threads to open a file in read only mode and read some data, and the OS page cache will safely manage the caching of data in memory without corruption.

Is it currently safe to do this with rclone when disk is mounted with vfs cache ?
I want to use rclone caching to pre-fetch data in another thread while the main thread is reading and processing the file.

Any guidance or information on this would be greatly appreciated.

Thanks,
Aaron

[boxerab]

Bumping this again - where can I go to learn more about the status of rclone with vfs caching? How thread safe is it? Any clues would be much appreciated, thanks!

[boxerab]

Alright, to answer my own question, it appears that --vfs-cache-mode full is thread safe. I ran a script that

1. deleted all data from cache folder
2. launched 10 threads to copy the same file from rclone mount to a local folder
3. did binary diff of 10 copied files with original

There was no difference.

It would be cool if someone from the project could verify that this is indeed safe
by design, and it won't be flaky in production.

I visited the github page and it pointed me here to ask any questions I might have about the project.