I think you are talking about two different things here.
vfs/refresh is just for metadata so directory and file structure.
Thats impacted by dir-cache-time and on a mount since Google Drive is a polling remote, you can have that value high.
I use that to precache my directory / file structure in memory so when my machine boots and starts the rclone mount, the directory listing is fast.
So when I run a directory listing, it comes back instantly.
felix@gemini:/GD$ time ls -alR | wc -l
The dir cache is lost when you reboot or restart the mount and the 'priming' of the mount needs to be done which is why I have it as part of my startup command to just make things faster for browsing. The time is takes is not dependent on the size but more so on the number of directories/files you have and how it's laid out.
Dir-cache-time has nothing directly to do with file based caching on disk with vfs-cache-mode full. That's for storing the actual data on the files and when you mount something with rclone, it only grabs when you ask it to grab. So if you read a file, it gets what you/the application has requested. The vfs-cache-mode data is persistent and sticks around on reboots.