You can set either to as large or as small as you want depending on what you are trying to accomplish. The backend matters as a polling backend usually means a larger directory cache time.
That is what I'd recommend. --attr-timeout controls caching in the kernel - it doesn't cost much for the kernel to ask rclone and it makes sure that the kernel is never out of data with respect to rclone.