2h of reading the manual later I think if the above command be run with --rc (flag enabling remote control) then running rclone rc vfs/refresh -v --fast-list recursive=true will precache all directories making the traversals much faster. Still need to test it though and find a way to integrate it with systemd