504 Gateway Timeout on badges inside GitHub README

[jcbcn]

Are you experiencing an issue with...

• shields.io

🪲
I am receiving 504 (Gateway Timeout) when viewing badges inside GitHub README's over a range of badge types. This is happening with least one badge in the README each time I try.

Possibly linked to #1568.

Thanks

[paulmelnikow]

Hi! Thanks for the report.

I haven't been able to reproduce this today on the Shields repos or any other repos that I've noticed.

Is this a public repo where you're seeing this happen?

[jcbcn]

Yeah, repo is here: https://github.com/pitch-game/pitch-api

I do have 17 in my README which may be the issue (maybe a few too many!). It is usually only happening on the test and coverage ones, rather than the three at the top.

I could move them into individual README's, but figured this could be a wider issue.

[paulmelnikow]

It should work, even with lots of badges…

I wonder if these are timing out because the upstream requests are slow. GitHub cuts things off at 4 seconds, so if the upstream requests take longer than that, the only thing we can do is put the right answer in the cache and then get it right on the next request (e.g. after refresh).

[paulmelnikow]

By the way, these are loading for me:

[jcbcn]

I'm starting to see this on first load of my frontend repo as well. I've just loaded up the README again this evening and am seeing the same behaviour:

It seems like it's slowly creeping back in again. After #1568 was fixed I haven't seen a broken badge for ages, but recently in the past week they seem to be more regular. I wonder if the IP ranges have changed which the cache was using?

However, you can see each request was taking ~8 seconds, so maybe it's like you said above and the downstream req is timing out.

[calebcartwright]

The other thing with both the Azure DevOps test and coverage badges is that we have to make two API calls to AzDO to get the actual data (the first to get the build id of the latest build and the second to get the respective data for that build)

[jcbcn]

Ah I see. The strange thing is this has been working flawlessly ever since #1568, just in the last week I've noticed it happening again. Obviously I'm not sure what's degraded, but because it had been fine for a while, I assumed it might be the caching again.

[platan]

There is something strange happening with our servers in the last hours:
https://metrics.shields.io/d/aESRBSjmz/services?orgId=1&from=now-3h&to=now
https://metrics.shields.io/d/g_1B7zhik/prom-client-default-metrics?orgId=1&refresh=10s&from=now-3h&to=now

[calebcartwright]

I believe @paulmelnikow ran a deploy at some point today (not sure when), but if there was a deployment during that same window that could potentially explain the spikes.

[jcbcn]

Seems to be fine since the rollback. Thanks again for looking into this. Much appreciated

EDIT: Spoke too soon. It seems to be happening again right now, if that helps to correlate with your logs

[paulmelnikow]

To clarify, I think @jcbcn's initial report may be related to a capacity issue, or else due to upstream services being slow.

While the symptom was the same, the issue that happened yesterday was probably unrelated.

[jcbcn]

Just to show another repo:

[paulmelnikow]

Have a link?

[jcbcn]

Whoops, sorry. https://github.com/pitch-game/pitch-frontend

[paulmelnikow]

They loaded for me just now.

Seems like this is happening intermittently. I'm curious how much this is related to Azure DevOps and how much it's intermittent server issues.

[calebcartwright]

Other than that short window after the last deploy, I haven't noticed anything either (including various Azure DevOps badges that are loading fine for me 🤷‍♂)

Either way though, it's probably worth revisiting the experiments we discussed in #3874 to update our prod environment

[jcbcn]

This looks to be resolved now. Thanks

[paulmelnikow]

Thanks! Please open a new issue if this recurs.