Backport of FLAKEY_TEST: Add retry to outbound request for ProxyLifecycleShutdown… into release/1.1.x

[hc-github-team-consul-core]

Backport

This PR is auto-generated from #3999 to be assessed for backporting due to the inclusion of the label backport/1.1.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@NiniOak
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/consul-k8s/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

Changes proposed in this PR

• Test fails intermittently in CI (for some reason it only fails in cloud environment not kind clusters) see example here
• Add retry to outbound API calls to improve chance of success.
  • if error occures, log the error and mark the current attempt as failed, trigger a retry.
  • If error does not exist, check if the output contains the specific error message. If the condition fails, log the error message and trigger a retry.

How I've tested this PR

CI should pass

How I expect reviewers to test this PR

CI should pass

Checklist

• Tests added
• CHANGELOG entry added

---

Overview of commits

• b547902 - b7ecab4 - 2fcccd2