add e2e test for leader election

[mhrivnak]

re #530

@hasbro17 commented:

Do you have any thoughts on how we can add an e2e test for this as a follow up?
It's a little tricky since we can't call leader.Become() from a test locally
I'm thinking in the test we would need to build and run a Deployment(of size 3) that does the following:

• call leader.Become()
• Set readiness probe to true once the lock is acquired.

And ensure that only 1 of the 3 pods become Ready(and is the leader).
And also ensure that it can step down after we kill that pod.

That sounds reasonable. I like the idea of the readiness probe. We can also verify that the pod claiming to be ready matches the owner reference on the lock. This is the sort of test that could be applied generically to any operator, as long as they follow the readiness pattern.

We could also make a test operator write its own hostname into some object, maybe even just the spec on any CR it sees. Then we could verify that the value we see there matches the owner reference observed on the operator's lock. That would help ensure that non-leaders are not doing reconciliation. We could even kill the pod, wait for a new leader, and verify that the value gets updated and again matches the ref on the lock.