Conformance results for v1.16/inspur-iop-mips64el

[zhanw15]

This is the conformance results for v1.16/inspur-iop-mips64el.
Thank you!

[taylorwaggoner]

@zhanw15, thank you for submitting. Can you please confirm if this is a new product or one of the 3 that are already certified? https://landscape.cncf.io/category=platform&format=card-mode&organization=inspur-group

If it is a new product, please send in a signed Participation form.
Thanks!

[zhanw15]

@taylorwaggoner, this is not a new product, IOP supports deploy kubernetes cluster on arch mips64el, amd64 and arm64. In order to verify the availability of cluster on mips, we build needed images and complete the conformance.

[donaldliu]

@zhanw15 The existing amd64 ("Inspur Cloud Container Engine and Inspur Open Platform") and arm64 ("Inspur Open Platform for ARM") have their own participation forms. The added mips64el ("Inspur Open Platform for MIPS") product name will need its own participation form.

[timyinshi]

@donaldliu hi,the product (Inspur Open Platform for MIPS) belongs to inspur cloud,is a new product.Our company is inspur cloud service group.

[dankohn]

This is exciting. I’m not aware of much K8s work being done on mips. Did you encounter any particular problems?

[timyinshi]

@dankohn
this is realy exciting.our product (Inspur Open Platform for MIPS) is a new product,it is base on mips,we have passed e2e test case,we have some mips images for e2e.
due to passing the test certification,so our product(Inspur Open Platform for MIPS) is stable.
please merge our pr.
we can contribute our MIPS test image to cncf,then cncf can determine which products are stable on MIPS.

[timyinshi]

@zhanw15
please send in a signed Participation form.

[zhanw15]

@taylorwaggoner @donaldliu @dankohn I'm sorry to made a mistake byfore. I have complate the Certified Kubernetes Form and email to [email protected]. Help us merge please, thanks!

[timyinshi]

@zhanw15
the file of commit have some errors,please modify the error.
after the change,please give a message.

[zhanw15]

Yeah, I think I have fixed the url in PRODUCT.yaml, It's right now.

[timothysc]

My only concern would be to ensure all the container images for testing are generally available for reproducibility. We should open an issue in the k8s repo to add to the build artifacts.

cc @dims @justaugustus

[dims]

@zhanw15 @timyinshi @donaldliu - please work with the kubernetes community to add support for the container images, example see where we are talking to the RISC-V folks ( kubernetes/kubernetes#82349 (comment) )

[timyinshi]

@dims
ok,we can have a new pr abort how to build k8s-mips,
and we can have a new pr abort how to test k8s-mips.
thanks.

[ydcool]

@dims We wrote a blog about this, kubernetes/website#18229

[dims]

@ydcool questions below :)

1. Do you have PRs open against golang for the problems mentioned? During the compilation processes, we inevitably encountered many platform compatibility problems, such as a Golang system call compatibility problem (syscall), typecasting of syscall. Stat_t from uint32 to uint64, patching for EpollEvent, and so on.

2. You mentioned that so we have to replace the alpine to existing MIPS images, Do you have a list of images that you had to do that for?

3. The dependency on gcr.io/google-samples/gb-frontend:v6 is puzzling, do you know which conformance test needs this? (easy way is to just not add that image to the nodes and check which test fails)

Thanks,
Dims

[dankohn]

@dims Thanks for the feedback. Could you please review https://github.com/kubernetes/website/pull/18229/files and make any comments.

[ydcool]

@dims Thanks for your attention,

1. In fact we have PRs for those problems, one for docker Cast Dev and Rdev of Stat_t to uint64 for mips moby/moby#39646, and some of them have been fixed by other people, for example the EpollEvent was fixed here https://go-review.googlesource.com/c/sys/+/207278. But there are still some issues which is complex and hard to make a patch, like this one x/sys/unix: missing SIGSTKFLT for mips64le golang/go#33381.
2. Actually all the test images which are based on alpine should be rebuilt with our MIPS base image. We plan to push all the MIPS base images we used to docker hub in the near future.
3. The image gcr.io/google-samples/gb-frontend:v6 was required in /hack/testdata/frontend-controller.yaml and used in the case Guestbook application

[dims]

@ydcool i am hesitating to support this conformance result because one of the terms we have is End User Reproducibility. So it seems at this point it is very hard for someone in the general community to reproduce the results independently.

+1 to the blog post that reflects current state, will let the folks taking care of k/website to review it first.

If you see kubernetes/kubernetes#86011 we are asking the RISC-V folks to file a KEP so we could update all the infrastructure in kubernetes/kubernetes so it would support all the platforms including MIPS. So let's do that! looking forward to working with you all.

[dims]

@johnSchnake can you please check the discussion above? (especially the bit about needing gcr.io/google-samples/gb-frontend:v6)

[ydcool]

@dims yes we fully understanding your point and indeed we're working on patching and testing the Kubernetess on MIPS, for the reproducibility. We need a little more time. But no doubt that the Kubernetes on MIPS has already been reproduceable, besides, our team also has started working on the conformance test for Kubernetes v1.17 on MIPS, maybe we'll get a new test result in the near future. So, we'll be very grateful if you have confidence in our work and could let this PR merged as soom as possible ; )

[timyinshi]

@dims
We have used hyperkube-mips64el:v1.16.2 on the inspur devops cloud, we passed the e2e-test, and it has been running three months, our hyperkube-mips64el:v1.16.2 is stable.So I don't think you need to worry about reproducibility.

[dims]

Ack @timyinshi back to @dankohn and others for review.

[dankohn]

We really need a decision out of SIG Architecture on this one. Everyone agrees that Inspur is doing the right thing to get a MIPS build created as part of the upstream CI/CD processes. The issue is whether it's enough for the MIPS image to be publicly available in order for them to become Certified Kubernetes, or if they should need to use a MIPS image actually generated upstream.

[timyinshi]

@dankohn
We will not only public the MIPS e2e-test images, we will also public the code of building the MIPS e2e-test images and hyperkube. In the next two weeks, we will submit the PRs to the kubernetes community.then everyone can use our contributed code to get the hyperkube and e2e-test images for MIPS.

[johnSchnake]

The gb-frontend:v6 is required for the Guestbook app which is used in the test linked in the discussion above ([sig-cli] Kubectl client [k8s.io] Guestbook application should create and stop a working application [Conformance])

The manifest is loaded from this file: https://github.com/kubernetes/kubernetes/blob/v1.16.2/test/e2e/testing-manifests/guestbook/frontend-deployment.yaml.in#L19
and that variable is set/defined in the normal list of images here: https://github.com/kubernetes/kubernetes/blob/v1.16.2/test/e2e/common/util.go#L89

Note though, that this has been updated to use the agnhost image: https://github.com/kubernetes/kubernetes/blob/master/test/e2e/testing-manifests/guestbook/frontend-deployment.yaml.in#L19-L20

So if the agnhost image will/can support mips then this will fix itself.

[johnSchnake]

As far as the /hack/testdata/frontend-controller.yaml , I'm not certain that matters or gets used anywhere.

[johnSchnake]

I agree that the move to supporting a new platform that has some active interest is a good thing. However, my two cents is that the official support for the platform needs to precede the certification of that platform.

Otherwise, it seems like an odd situation that the CNCF would certify a custom build of k8s running a custom set of test images. That blurs the lines of what is really required if neither the build nor the tests have to be the officially released ones.

[ydcool]

Hi all, please take a look at this issue kubernetes/kubernetes#87056

[timyinshi]

@johnSchnake @dankohn In this way, hyperkube of MIPS can be reproducibility.
kubernetes/kubernetes#87056

[dankohn]

It looks like progress on this has unfortunately been blocked by the requirement to create a KEP for K8s to support new architectures:

kubernetes/kubernetes#87056
kubernetes/kubernetes#87058 (comment)

I'm going to close this for now, but I encourage you to work with the Kubernetes community to establish support and then we would be happy to approve your submission.

In particular, you would want to engage with SIG Architecture to discuss what would be entailed in having the Kubernetes community support an additional architecture, or whether there would be some alternative to doing so.

https://github.com/kubernetes/community/tree/master/sig-architecture#architecture-special-interest-group

[dims]

please see kubernetes/community#5014 for the WIP doc for guidance from sig-arch.