Fix self hosted runner Action field behavior change

[davidvonthenen]

What this PR does / why we need it

There were two issues:

• pushing multiple commits to the same job could have lead to the deletion of the running after the first run was complete. We now check to make sure the runner is idle before deleting: https://github.com/vmware-tanzu/community-edition/pull/2881/files#diff-210090a2d26e86b33d3cc35058efd559b9e3764511cc6da7474d403f5819094eR162
• there seems to have been a behavior change recently in which a job would not correctly fire off the in_progress status. So will key off the completed status instead: https://github.com/vmware-tanzu/community-edition/pull/2881/files#diff-210090a2d26e86b33d3cc35058efd559b9e3764511cc6da7474d403f5819094eR293

Additional changes:

• Added some debugging for tracing
• Put GitHub and EC2 clients into respective structs
• Added an "all else fails" clean up thread that checks for github clients that are idle and attempts to clean them up

Details for the Release Notes (PLEASE PROVIDE)

Fix self hosted runner Action field behavior change

Which issue(s) this PR fixes

Fixes: #2870

Describe testing done for PR

It's currently running on the webhook instances and fielding jobs

Special notes for your reviewer

NA

[davidvonthenen]

This is ready for review!

[karuppiah7890]

hey @dvonthenen 👋 I wanted to check how you debugged this issue, so that I could learn too. How did you get the logs of the TCE webhook service running in AWS? I noticed the 3 AWS EC2 instances named TCE Webhook, and I'm assuming those were running the service. I couldn't find any logs in CloudWatch. Did you SSH into the machine and check the logs? I think I can't SSH into it unless my public key is added to the ~/.ssh/authorized_keys file. I was wondering if we can use CloudWatch or something similar or if I can get SSH access to the machines so that I can help debug too 😄

Let me know what you think :)

[davidvonthenen]

Yea, I just ssh'ed in. We can talk offline and add your keys to those instances.

[stmcginnis]

Looks reasonable to me, and we can always follow up on anything else.