Self hosted AWS runners not being available in some GitHub Actions workflows

[karuppiah7890]

Below is a big list of workflows where the workflow executed for a long time with no runner available for it. Check out the links to get an idea of how sometimes the workflows went on for a few hours to a complete day without progressing and being in the queue

• https://github.com/vmware-tanzu/community-edition/actions/runs/1700138941/attempts/3
  • https://github.com/vmware-tanzu/community-edition/actions/runs/1700138941/attempts/2
  • https://github.com/vmware-tanzu/community-edition/actions/runs/1700138941/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1711276587/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1713918372/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1713918381/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1714319647/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1714370406/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1714383535/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1715455523/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1715566981/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1715642639/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1715676549/attempts/1
• https://github.com/vmware-tanzu/community-edition/actions/runs/1717708369/attempts/1

I also noticed that sometimes a simple re-run does to do the job and make it work. Gotta check what's going wrong here though. cc @dvonthenen

[davidvonthenen]

I think I already know what's wrong and I have run into the error before. I didn't save the error message the last time, but it's due to running out of quota for the instance type we are using. Next time it happens, the error should get captured again and I will increase the quota limits.

[davidvonthenen]

Ahhh I think I see the problem now. Working on a fix.