Route Hex.pm requests through old badge server proxies

[calebcartwright]

Refs #4962 #4957 #4929 (comment)

The Hex.pm API has a rate limit of 100 requests per minute per IP Address, a limit Shields has previously exceeded (#1285 (comment)).

That was previously resolved by Hex whitelisting the Shields OVH production server ip addresses. Now that we've moved our runtime to Heroku the badge servers no longer have static ip's and we're subject to those rate limits again.

We ran into the same thing with the Discord badges and rolled out a workaround (#4956) that routes the requests to the upstream provider via the old badge servers. We can and should do the same for the Hex badges (https://github.com/badges/shields/pull/4956/files#diff-59e43644e767926a5be1ebbe6b3a36d0)

[PyvesB]

I've taken a look at our Grafana metrics, and temporarily tweaked the "All request rate" panel to only display the request rate for Hex.pm. Here's what I got for the past 48 hours:

That's a request rate per second averaged over a minute, so you just have to multiply by 60 to compare with the rate limit imposed by Hex.pm if I'm correct. We're almost always below 25 request per minute. Looking at spikes from various other days, I've never seen us go over 50 requests per minute. Bear in mind that we've got five dynos in production, so the requests would be spread across several IP addresses, we're probably around 10 requests per minute per IP during spikes.

As far as I can tell, we're very far from hitting the rate limit. Back in 2017 when the issue with Hex.pm was reported, we didn't have proper cache headers and our homepage would load the Hex.pm badges every single time. That significantly increased traffic to the service, but is no longer a problem nowadays.

I think we should not take any action, and close this issue.

[PyvesB]

I'm going to go ahead and close this one. If anyone notices broken Hex.pm badges, please do ping us.

[chris48s]

I was under the misguided impression that we were already routing the hex badges via OVH. If we've been running without doing that since we migrated to heroku I am sure someone would have raised an issue if we were hitting the rate limits.
Does that mean we can completely decommission the old servers now? (not that any of us actually have access to, but in principle...)