Hi,
I am new to rclone but tested a lot to run my application on the Azure vfs remote.
I configered to use the full cache mode and also a fast ephemeral disk in a azure vm for the cache.
The throughput is good the IOPs are good, but the Application is handling the files with excessive locks.
The issue is, that the application reads the files, which are in best case already cached and then writes a .lock file. Most of the will be removed within 1 second because the file was read.
Unfortunatly the lock file will be transferred to the remote which takes about some parts of a second.
Not much, but I have several thousands of files and even 1ms each will take seconds in sum.
The ephemeral disk and also connection to azure blob storage is not the issue. If I run more searches in the application on the data, the result time is nearly the same.
Just for understanding:
To search for 60 Million Datasets in the application it takes about 4 seconds on normal disks
120 seconds if it is not cached on the local disk (I bet it is also because of the transfer of the lock, but i cannot promise)
After caching the files it takes about 80 seconds.
To read from disk is really fast, because if even a bigger file is read (few MB), it is closing it within on second.
So my question is:
Do i have the possibility to improve this? For cache there is a upload time setting to define but not for Azure vfs.
I also tested cache remote with almost the same settings and it was incredibly slow.
The Azure cache thing is the best what I can see right now, but I need to say that files which are written or changed in cache stays for some seconds before uploading to azure.
Thanks for you help