The Google Drive remote-backend has something called the "drive-pacer" which is basically flexible speed-limit that makes rclone try to more or less obey Google's limits in terms of API calls.
Google Photos used to be part of Google Drive I believe, so I would assume it either has a slightly tweaked version of the pacer, or simply uses the same. In short - I doubt it should be necessary to use such a severe limit. The general rule is that default settings (4 transfers, no tpslimit) should generally work for a remote out of the box. That said, Google Photos is still fairly new and might have quirks and issues.
Are you using a custom Oauth key? If you never made one and you still use the shared key then that COULD be related if and when lots of other rclone users are pushing the system. It could also be irrelevant to your spesific problem - but it's just generally advised to do this if you can.
Finally, there has been a whole slew of Google Photos related posts recently, due to it being fairly new. It would be worth doing a quick search-by-date and looking to see if anyone already reported similar issues.
Sorry that I don't have a more spesific solution to offer, but i have not yet used Google Photos extensively myself so my experience is limited with it.