So I did some further digging. This may have been all my fault.
My OAuth consent screen was still in testing under the Publishing status. I also found this
A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of "Testing" is issued a refresh token expiring in 7 days.
There is currently a limit of 50 refresh tokens per Google Account per OAuth 2.0 client ID. If the limit is reached, creating a new refresh token automatically invalidates the oldest refresh token without warning. This limit does not apply to service accounts.
I've now changed the Publishing status to In production. So we'll see if that makes a difference. If the token needs to be refreshed hopefully it's just one last time.