Handle etcd status condition when node is not ready and disable etcd

[vitorsavian]

Proposed Changes

• Handle non ready nodes for etcd status condition
• Handle node when disable etcd is set

Types of Changes

• Feature

Verification

Create 2 or more servers in cluster mode
Take a snapshot to s3
Terminate all instances and bring up a new fresh instance
Restore s3 snapshot to the fresh instance

Then use

 kubectl get node {NodeName} -o=jsonpath='{range .status.conditions[*]}{"\n"}{"type: "}{.type},{"\n"}{"status: "}{.status},{"\n"}{"lastHeartbeatTime: "}{.lastHeartbeatTime},{"\n"}{"lastTransitionTime: "}{.lastTransitionTime},{"\n"}{"reason: "}{.reason},{"\n"}{"message: "}{.message},{"\n"}{end}'

Testing

Linked Issues

• New EtcdIsVoter Node condition should update status on a cluster-reset #9048

User-Facing Change

Further Comments

[brandond]

couple nits, looking good though!

[brandond]

Note that there is also SetNodeCondition that could probably save a fair bit of code here as well.

[brandond]

couple new nits, think we're close though.

[brandond]

Another ask from the issue was

I wonder if it's possible to do periodic checking from a healthy node to determine what is the actual state and update based on that.

Can we also use the etcd member health information to update the status if the node is a member of the cluster, but not healthy? For example, if it is a member of the cluster, but the pod is not running?

[vitorsavian]

Another ask from the issue was

I wonder if it's possible to do periodic checking from a healthy node to determine what is the actual state and update based on that.

Can we also use the etcd member health information to update the status if the node is a member of the cluster, but not healthy? For example, if it is a member of the cluster, but the pod is not running?

I was searching for a method to get the etcd member health the same way etcdctl does, it just tries to get a key and then shows if it is healthy or not, but I'm not sure if it's the best approach in this case. What do you think @brandond ?

[dereknola]

Drone is failing because its trying to run the broken secrets encryption test. You need to rebase off master to get the latest version that doesn't.

[codecov]

Codecov Report

Attention: 74 lines in your changes are missing coverage. Please review.

Comparison is base (eae221f) 41.46% compared to head (1f2aacb) 22.27%.
Report is 1 commits behind head on master.

Files	Patch %	Lines
pkg/etcd/etcd.go	0.00%	71 Missing ⚠️
pkg/etcd/metadata_controller.go	0.00%	3 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           master    #9084       +/-   ##
===========================================
- Coverage   41.46%   22.27%   -19.19%     
===========================================
  Files         151      148        -3     
  Lines       16048    16030       -18     
===========================================
- Hits         6654     3571     -3083     
- Misses       8259    11720     +3461     
+ Partials     1135      739      -396     

Flag	Coverage Δ
inttests	22.27% <0.00%> (-16.41%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.