Pure Go

[xtellurian]

This PR is the main PR for the pure go work into master.

It includes:

• Changed build definition
• Go implementation
• Removal of python
• Upgrade to KubeBuilder 2

[xtellurian]

Tests in microsoft/azure-databricks-operator/pkg/controller/ seem to be failing.

go test ./pkg/... ./cmd/... -coverprofile cover.out
?   	microsoft/azure-databricks-operator/pkg/apis	[no test files]
?   	microsoft/azure-databricks-operator/pkg/apis/microsoft	[no test files]
ok  	microsoft/azure-databricks-operator/pkg/apis/microsoft/v1beta1	0.267s	coverage: 23.5% of statements
?   	microsoft/azure-databricks-operator/pkg/controller	[no test files]
E0621 06:02:05.524351   17904 runtime.go:69] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:76
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:65
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:51
/usr/local/go1.12/src/runtime/panic.go:522
/usr/local/go1.12/src/runtime/panic.go:82
/usr/local/go1.12/src/runtime/signal_unix.go:390
/home/vsts/work/1/s/gopath/src/microsoft/azure-databricks-operator/pkg/controller/notebookjob/notebookjob_controller_suite_test.go:54
/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:92
/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215
/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134
/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88
/usr/local/go1.12/src/runtime/asm_amd64.s:1337
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x111af00]

goroutine 109 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
	/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:58 +0x105
panic(0x1243aa0, 0x20c1f70)
	/usr/local/go1.12/src/runtime/panic.go:522 +0x1b5
microsoft/azure-databricks-operator/pkg/controller/notebookjob.SetupTestReconcile.func1(0xc0000bb819, 0x7, 0xc0000bb800, 0xa, 0xc0001126c0, 0xc000075d88, 0x18, 0xc000075d80)
	/home/vsts/work/1/s/gopath/src/microsoft/azure-databricks-operator/pkg/controller/notebookjob/notebookjob_controller_suite_test.go:55 +0x40
sigs.k8s.io/controller-runtime/pkg/reconcile.Func.Reconcile(0xc0004220c0, 0xc0000bb819, 0x7, 0xc0000bb800, 0xa, 0x20ecf40, 0xc000010008, 0xc000471e18, 0xfb7d1c)
	/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/reconcile/reconcile.go:92 +0x4e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00038bd60, 0x0)
	/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:215 +0x1cc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1()
	/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:158 +0x36
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0003dc020)
	/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x54
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0003dc020, 0x3b9aca00, 0x0, 0x1, 0xc00032a2a0)
	/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:134 +0xf8
k8s.io/apimachinery/pkg/util/wait.Until(0xc0003dc020, 0x3b9aca00, 0xc00032a2a0)
	/home/vsts/work/1/s/gopath/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:88 +0x4d
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start
	/home/vsts/work/1/s/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:157 +0x311
FAIL	microsoft/azure-databricks-operator/pkg/controller/notebookjob	0.286s
?   	microsoft/azure-databricks-operator/pkg/rand	[no test files]
?   	microsoft/azure-databricks-operator/pkg/webhook	[no test files]
?   	microsoft/azure-databricks-operator/cmd/manager	[no test files]
Makefile:14: recipe for target 'test' failed

[xtellurian]

#24

[xtellurian]

Above issue is resolved by adding DATABRICKS_HOST and DATABRICKS_TOKEN to the test environment.

[xtellurian]

Currently tests are failing with the following error:

--- FAIL: TestReconcile (104.09s)
    notebookjob_controller_test.go:146: 
        Timed out after 60.008s.
        Error: Expected an error, got nil
FAIL

[xinsnake]

Looking into it now. This issue seems to be intermittent.

[Azadehkhojandi]

Please update readme while I'm reviewing it

[xtellurian]

Tests are now passing.

We're going to add some utility binaries to the repository using git LFS. This will enable us to manage the binary versions more reliably, and enable us to run kustomize in the build pipeline. We need this to generate the setup.yaml people are expecting in the github release

[xinsnake]

It seems that LFS support in the pipeline is not enabled. @xtellurian can you please check the DevOps build?

[xinsnake]

@xtellurian Now upgraded to use kubebuilder v2 and embedded Kubernetes API server. Now build time is reduced from 9+ minutes to 6+ minutes. We should be able to further reduce it as currently, we are testing outside the container but building inside the container, which should be made consistent and will also save time.

ok  	github.com/microsoft/azure-databricks-operator/api/v1	7.716s	coverage: 20.4% of statements
ok  	github.com/microsoft/azure-databricks-operator/controllers	46.205s	coverage: 80.9% of statements

[xtellurian]

@xinsnake great work!

[Azadehkhojandi]

Awesome work @xinsnake :)
I'll update release pipeline