Sonobuoy 0.20.0 is missing required conformance tests

[adamwg]

What steps did you take and what happened:

We ran 1.20 conformance via Sonobuoy 0.20.0 and submitted the results to k8s-conformance here. The conformance bot flagged a missing required test, and it was determined that the test hadn't run due to a missing metadata update in k/k.

What did you expect to happen:

All required conformance tests for 1.20 to run.

Anything else you would like to add:

I'm not super clear on what's required to fix this. I suspect sonobuoy needs to be re-built (and a new release cut) with the updated test metadata, but the tool is relatively opaque to me, so please let me know if all I need to do is re-run.

Environment:

• Sonobuoy version: 0.20.0
• Kubernetes version: (use kubectl version): 1.20.2
• Kubernetes installer & version:
• Cloud provider or hardware configuration: DigitalOcean Kubernetes
• OS (e.g. from /etc/os-release):
• Sonobuoy tarball (which contains * below)

[wilsonehusin]

@adamwg thanks for filing the report! Looks like this is an integration problem from upstream k/k which should have been fixed by now. I would recommend trying it again.

Sonobuoy doesn't compile the conformance test suite, we simply orchestrate the flow of conformance image published upstream (k8s.gcr.io/conformance:<version>). If this issue has been fixed upstream then I would expect things to pass, or at least $ sonobuoy status returns success.

Let me know if it's still failing though, since the issue is still fresh upstream there might still be moving parts.

[brtkwr]

The k8s.gcr.io/conformance:v0.20.2 image hasnt been updated in 4 weeks so I doubt the recent merge will be reflected there. Maybe we'll need to wait for v1.20.3 to be released??

[wilsonehusin]

not really -- the version tag on k8s.gcr.io/conformance image maps to your cluster version, so if your cluster is 1.20.2, then it will pull the 1.20.2 image.

correct me if I'm wrong, but my understanding is that the failure at the moment is about CNCF integration checker bot, not conformance test suite itself -- so running sonobuoy status should still pass on a conformant cluster.

[adamwg]

Thanks for taking a look @wilsonehusin and @brtknr! From previous discussion on cncf/k8s-conformance#1308 it sounded like a sonobuoy update would be needed, but indeed it appears this was just a problem with the conformance checking bot, which is now resolved.