You probably do not want to use cache mode full - as that mode will need to completely download an entire file before accessing. For some files this won't matter much, but especially for streaming formats it is very much not ideal. The most commonly used cache modes are either off, or write - with write generally being the "best" in most cases for compatibility of all types of operations - at the cost of needing some local work-space. If you want to edit files or have more safety against failed uploads then try write.
Some lag on accessing is inevitable due to how the whole interface process works, but it should typically be on the order of a couple of seconds for small files or streaming formats. If you experience a lot more than that then there may be some other issue.
You may want to provide some more spesific info about what exactly your test is, what your config looks like (redact passwords) ect.