Kubebuilder is not listed in prerequisites

[deepak1725]

Bug Report

Kubebuilder is Tightly coupled in requirements, which is fine but...

1. kubebuilder is not listed in Prerequisites
2. make docker-build ... expects kubebuilder in usr/local/kubebuilder only otherwise it raises an error, which I don't think is a good idea. User may have kubebuilder anywhere in PATH.

What did you do?
make docker-build $IMG

What did you expect to see?
resolves kubebuilder from $PATH not from any Specific Location.

What did you see instead? Under which circumstances?

Failure [0.002 seconds]
[BeforeSuite] BeforeSuite 
/home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:52

  Unexpected error:
      <*fmt.wrapError | 0xc00046c2a0>: {
          msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory",
          err: {
              Op: "fork/exec",
              Path: "/usr/local/kubebuilder/bin/etcd",
              Err: 0x2,
          },
      }
      failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory
  occurred

  /home/deepak/projects/go/src/operator/newoperator3/final/crda-operator/controllers/suite_test.go:62
------------------------------


Ran 0 of 0 Specs in 0.002 seconds
FAIL! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (0.00s)
FAIL
coverage: 0.0% of statements
FAIL    github.com/deepak1725/crda-operator/controllers 0.010s
FAIL
make: *** [Makefile:30: test] Error 1

Environment

• operator-sdk version: 0.19

• go version: 0.14

• Kubernetes version information: 1.18

• Kubernetes cluster kind: minikube

• Are you writing your operator in ansible, helm, or go? go

Possible Solution

Additional context
Add any other context about the problem here.

[camilamacedo86]

Hi @deepak1725,

I am unable to reproduce this issue. Kubebuilder should not be a pre-requirement and shows that your project is asking its binary to setup the Env Test.

Could you please let us know the steps performed to generate this issue? Could you please let me know if you have kubebuilder installed when you create the project?

[deepak1725]

@camilamacedo86
My System didn't had kubebuilder

Steps:

• I tried make docker-build $IMG

• Got Error: As Specified above.

• then I installed it in the separate Path than specified in Official Kubebuilder docs.
  ie. I installed in /usr/local/bin/kubebuilder instead of /usr/local/kubebuilder/

• Exported Kubebuilder and it is recognized from every end of PC.

• I tried again make docker-build $IMG, but the error was still not resolved.

• Finally I have moved binaries from my custom path to officially specified Kubebuilder path, which finally Resolves Error.

• Conclusion acc. to me:
  SDK has Hard-Coded Kubebuilder path which relies solely on Official specified Path.

[camilamacedo86]

Hi @deepak1725,

Really thank you for raise that. As you know, SDK is in a process to be integrated with KB which means that its project layouts will be aligned. More info : Integrating Kubebuilder and Operator SDK. But, IMHO kubebuilder cannot be set as a pre-requirement because the kubebuilder version used by SDK as dependency can be diff of the version installed.

The problem faced by you is because Env Test requires the apiserver, kubectl and etcd binaries to run and these are shipped with kubebuilder binary. You can customize its set up to use them from another location as well. To understand it further as to see how you can change its conjugation see: kubernetes-sigs/kubebuilder#686 (comment).

We will discuss this scenario in the bug triage meeting on Monday. Feel free to attend if you wish.

[camilamacedo86]

PR to fix it for v3 in Kubebuilder; kubernetes-sigs/kubebuilder#1600

[camilamacedo86]

Hi @deepak1725,

See that we added a script to upstream to do this setup and we have a pr against master #3510 and against 0.19.x #3517 to update the docs.

[camilamacedo86]

All done. We can improve the docs in the future as well. Really tks for raise that.

[deepak1725]

@camilamacedo86 are we going to cherry-pick this in 0.19 later..?

[camilamacedo86]

It is applied to the 0.19 already. See; https://v0-19-x.sdk.operatorframework.io/docs/golang/references/env-test-setup/ and the PR: #3517

[camilamacedo86]

We are re-open this one. After the kubebuilder/controller-runtime meeting was decided that we should not re-build the bins and instead of that get them from https://storage.googleapis.com/kubebuilder-tools. @joelanford has been working in an script against controller-runtime: kubernetes-sigs/controller-runtime#1092

And then, we imagine as a final solution something such as:

• setup_envtest.sh:

ENVTEST_ASSETS_DIR=$(pwd)/test/assets
fetch_envtest_assets() {
    OS=$(uname -s | tr '[:upper:]' '[:lower:]')
    ARCH=$(uname -m | sed 's/x86_64/amd64/')
    mkdir -p $ENVTEST_ASSETS_DIR
    curl -fsL "https://storage.googleapis.com/kubebuilder-tools/kubebuilder-tools-1.16.4-${OS}-${ARCH}.tar.gz" | tar zx --strip-components=1 -C $ENVTEST_ASSETS_DIR
}
setup_envtest_env() {
    export TEST_ASSET_KUBE_APISERVER=$ENVTEST_ASSETS_DIR/bin/kube-apiserver
    export TEST_ASSET_ETCD=$ENVTEST_ASSETS_DIR/bin/etcd
    export TEST_ASSET_KUBECTL=$ENVTEST_ASSETS_DIR/bin/kubectl
}

Then in a makefile (or wherever)
source setup_envtest.sh; fetch_envtest_assets; setup_envtest_env

Also, in the PRs #3551 and #3543 we are marking the Env Test doc as TBD as the step to setup it until we have a better solution.

[estroz]

This shouldn't block v1.0 since the steps to get envtest tools are encoded in a script and documented. Moving this to v1.1.0.

[jimdickinson]

I just hit this on 1.0, and I don't see this explained in the docs. Is this the right page? https://sdk.operatorframework.io/docs/building-operators/golang/references/envtest-setup/

Might want to weave a link to that into the tutorial, although it's quite possible I missed it.

[camilamacedo86]

Hi @jimdickinson,

In the quickstart, you will see that a step was added to solve this problem for now. See: https://sdk.operatorframework.io/docs/building-operators/golang/quickstart/#configuring-your-test-environment

[zonnie]

@camilamacedo86 - the link you sent leads to the Quickstart page but the section is missing.
I tried to look for the workaround anyway but was unable to find it...any quick way to avoid this issue ?

[camilamacedo86]

The above link shows the latest version (1.1). For this version, the setup is no longer required. See that your project is built with the target as: https://github.com/operator-framework/operator-sdk/blob/master/testdata/go/memcached-operator/Makefile#L28-L33.

However, if you see any problem could you please raise a new issue.

[kansberry]

I am using operator-sdk version 1.3.0 and get this error message:
Unexpected error: <*fmt.wrapError | 0xc00045cd60>: { msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory", err: { Op: "fork/exec", Path: "/usr/local/kubebuilder/bin/etcd", Err: 0x2, }, } failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory occurred

and here are the lines in the Makefile that seem to be discussed above:
ENVTEST_ASSETS_DIR=$(shell pwd)/testbin test: generate fmt vet manifests mkdir -p ${ENVTEST_ASSETS_DIR} test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/v0.7.0/hack/setup-envtest.sh source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

Any suggestion as to what I am doing wrong would be appreciated.

[estroz]

@kansberry make test is giving you that error? If you add

SHELL = /bin/bash

to your Makefile, what happens?

[kansberry]

No, the command I am running is "make docker-build IMG=registry.hub.docker.com/$USERNAME/memcached-operator:v0.0.1". I also created an environment variable of SHELL=/bin/bash, as suggested, but this didn't fix the issue. I tried falling back to 1.2.0, and I get the same error.

CORRECTION. I missed the part about adding the value to the Makefile. I just did that, and it worked.

[estroz]

@kansberry can you file a new issue with a link to/paste of your Makefile. Just saw your edit above, this is a known problem upstream and is being worked on. See #4203

[Phiph]

@kansberry make test is giving you that error? If you add

SHELL = /bin/bash

to your Makefile, what happens?

Thanks Estroz. I was experiencing the same problem as the OP. Doing your suggestion solved the problem!

Environment -
Kali linux WSL2 instance.

[techjacker]

I'm still experiencing this issue

[ghost]

OS: WSL2

Still does not work!

Commit - https://github.com/evergreen-trading-systems/yta-operator/commit/64a13f71d2b05bc45e4285e6942e3249af85790a#diff-76ed074a9305c04054cdebb9e9aad2d818052b07091de1f20cad0bbac34ffb52

[StephanSalas]

Yep, I think this is a user-experience issue. If I have to do some research to rename a path for kubebuilder manually through some riga-maru, I don't think most users will find this edge case very friendly. I'd suggest an automated solution to prevent this error.

[dh-ironsec]

These helped me:

USE_EXISTING_CLUSTER (boolean): if set to true, envtest will use an existing cluster
TEST_ASSET_KUBE_APISERVER (string): path to the api-server binary to use
TEST_ASSET_ETCD (string): path to the etcd binary to use
TEST_ASSET_KUBECTL (string): path to the kubectl binary to use
KUBEBUILDER_ASSETS (string): directory containing the binaries to use (api-server, etcd and kubectl). Defaults to /usr/local/kubebuilder/bin.
KUBEBUILDER_CONTROLPLANE_START_TIMEOUT (string supported by time.ParseDuration): timeout for test control plane to start. Defaults to 20s.
KUBEBUILDER_CONTROLPLANE_STOP_TIMEOUT (string supported by time.ParseDuration): timeout for test control plane to start. Defaults to 20s.
KUBEBUILDER_ATTACH_CONTROL_PLANE_OUTPUT (boolean): if set to true, the control plane's stdout and stderr are attached to os.Stdout and os.Stderr