fix: make controllers tolerant to spec marshalling errors

[jessesuen]

Resolves #389, #517

Uses a new, specialized "tolerant" dynamic informer which tolerates malformed objects that cannot unmarshal properly, preventing the issue where a unmarshallable object renders the entire controller unusable. When an object is discovered to be malformed, the informer will return a best effort version of the object via json unmarshalling (dropping all fields which could not be marshalled properly). This allows the controller to proceed as if the mistake was never made.

A consequence of this change is that the malformed field is silently dropped. But this is at least better behavior than stopping the controller.

[codecov-commenter]

Codecov Report

Merging #666 into master will decrease coverage by 0.24%.
The diff coverage is 74.00%.

@@            Coverage Diff             @@
##           master     #666      +/-   ##
==========================================
- Coverage   84.95%   84.70%   -0.25%     
==========================================
  Files          89       95       +6     
  Lines        8718     8918     +200     
==========================================
+ Hits         7406     7554     +148     
- Misses        934      965      +31     
- Partials      378      399      +21     

Impacted Files	Coverage Δ
utils/tolerantinformer/clusteranalysistemplate.go	72.41% <72.41%> (ø)
utils/tolerantinformer/analysisrun.go	72.97% <72.97%> (ø)
utils/tolerantinformer/analysistemplate.go	72.97% <72.97%> (ø)
utils/tolerantinformer/experiment.go	72.97% <72.97%> (ø)
utils/tolerantinformer/rollout.go	72.97% <72.97%> (ø)
utils/tolerantinformer/tollerantinformer.go	73.33% <73.33%> (ø)
utils/log/log.go	100.00% <100.00%> (ø)
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9be76fc...ab70632. Read the comment docs.

[khhirani]

LGTM