poll-time is how often rclone asks drive “have you got any changes?”. dir-cache-time is how long rclone is willing to keep directory entries around before deciding they are too old and discarding them.
You likely want to set dir-cache-time high, or infinite and the poll time lower (say 1 minute).