That seems to have fixed the issue for now. I set it for 3 minutes both on the http backend and on the mount.
But I still think it's very odd that changing from an domain to IP had a big change. @ncw does rclone respect the TTL of the domains ?
It's from my understanding that after the domain is resolved to the IP once, it should stay in the cache as long as the TTL allows, so changing from domain to IP shouldn't make any difference
Both the mount and the backend are on the same datacenter too.