Skip to content
This repository has been archived by the owner on Jan 28, 2022. It is now read-only.

Commit

Permalink
Merge pull request #6 from microsoft/az/readme1
Browse files Browse the repository at this point in the history
update readme
  • Loading branch information
Azadehkhojandi authored May 22, 2019
2 parents 74d1812 + 4ec688d commit 4444c6a
Showing 1 changed file with 49 additions and 32 deletions.
81 changes: 49 additions & 32 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Introduction

Azure Databricks operator contains two projects. The golang application is a Kubernetes controller that watches CRDs that defines a Databricks job (input, output, functions, transformers, etc) and The Python Flask App sends commands to the Databricks.
Azure Databricks operator contains two projects. The golang application is a Kubernetes controller that watches CRDs that defines a Databricks job and The Python Flask App which sends commands to the Databricks.

![alt text](docs/images/azure-databricks-operator.jpg "high level architecture")

Expand Down Expand Up @@ -52,49 +52,65 @@ kubectl config set-context minikube --cluster=minikube --user=minikub
More info:

https://devkimchi.com/2018/06/05/running-kubernetes-on-wsl/

https://www.jamessturtevant.com/posts/Running-Kubernetes-Minikube-on-Windows-10-with-WSL/

### How to use operator

*Docs are work in progress*

Create .env file and set values of `DATABRICKS_HOST` and `DATABRICKS_TOKEN`
1. Create a secret set values of `DATABRICKS_HOST` and `DATABRICKS_TOKEN`

```
DATABRICKS_HOST=https://australiaeast.azuredatabricks.net
DATABRICKS_TOKEN=xxxx
```
```shell
kubectl create secret testdatabricks --from-literal=DatabricksHost="https://xxxx.azuredatabricks.net" --from-literal=DatabricksToken="xxxxx"
```

1. To install CRDs into a cluster : `kubectl apply -f databricks-operator/config/crds` or `make install -C databricks-operator`
2. To deploy controller in the configured Kubernetes cluster in ~/.kube/config `kustomize build databricks-operator/config | kubectl apply -f -`
3. Change NotebookJob name from `sample1run1` to your desired name and Update the values in `microsoft_v1beta2_notebookjob.yaml`

```
kubectl apply -f databricks-operator/config/samples/microsoft_v1beta2_notebookjob.yaml
kubectl get notebookjob
kubectl describe notebookjob kubectl sample1run1
```
Make sure your secrete name is set correctly in `databricks-operator/config/default/azure_databricks_api_image_patch.yaml`

Basic commands to check the new Notebookjob
```
kubectl get crd
kubectl -n databricks-operator-system get svc
kubectl -n databricks-operator-system get pod
kubectl -n databricks-operator-system describe pod databricks-operator-controller-manager-0
kubectl -n databricks-operator-system logs databricks-operator-controller-manager-0 -c dbricks -f
```
2. To install NotebookJob CRD in the configured Kubernetes cluster in ~/.kube/config, run `kubectl apply -f databricks-operator/config/crds` or `make install -C databricks-operator`

There is a also a Make file that you can use to install, test and deploy.
3. To deploy controller in the configured Kubernetes cluster in ~/.kube/config, run `kustomize build databricks-operator/config | kubectl apply -f -`

```
make -C databricks-operator
make docker-build IMG=azadehkhojandi/databricks-operator -C databricks-operator
make docker-push IMG=azadehkhojandi/databricks-operator -C databricks-operator
make deploy -C databricks-operator
4. Change NotebookJob name from `sample1run1` to your desired name, set Databricks notebook path and update the values in `microsoft_v1beta2_notebookjob.yaml`

```
```shell
kubectl apply -f databricks-operator/config/samples/microsoft_v1beta2_notebookjob.yaml
```

5. Basic commands to check the new Notebookjob

```shell
kubectl get crd
kubectl -n databricks-operator-system get svc
kubectl -n databricks-operator-system get pod
kubectl -n databricks-operator-system describe pod databricks-operator-controller-manager-0
kubectl -n databricks-operator-system logs databricks-operator-controller-manager-0 -c dbricks -f
kubectl get notebookjob
kubectl describe notebookjob kubectl sample1run1
```

### How to extend the opertaor and build your own images

#### Updating databricks operator:

This Repo is generated by [Kubebuilder](https://book.kubebuilder.io/).

To Extend the operator `databricks-operator`:

1. Run `dep ensure` to download dependencies. It doesn't show any progress bar and takes a while to download all of dependencies.
2. Update `pkg\apis\microsoft\v1beta1\notebookjob_types.go`.
3. Regenerate CRD `make manifests`.
4. Install updated CRD `make install`
5. Generate code `make generate`
6. Update operator `pkg\controller\notebookjob\notebookjob_controller.go`
7. Update tests and run `make test`
8. Build `make build`
9. Deploy
```shell
make docker-build IMG=azadehkhojandi/databricks-operator
make docker-push IMG=azadehkhojandi/databricks-operator
make deploy
```
## Main Contributors
Expand All @@ -105,7 +121,8 @@ make deploy -C databricks-operator
5. Rian Finnegan [Github](https://github.com/xtellurian), [Linkedin](https://www.linkedin.com/in/rian-finnegan-97651b55/)
6. Jason Goodselli [Github](https://github.com/JasonTheDeveloper), [Linkedin](https://www.linkedin.com/in/jason-goodsell-2505a3b2/)
7. Craig Rodger [Github](https://github.com/crrodger), [Linkedin](https://www.linkedin.com/in/craigrodger/)
8. Azadeh Khojandi [Github](https://github.com/Azadehkhojandi), [Linkedin](https://www.linkedin.com/in/azadeh-khojandi-ba441b3/)
8. Justin Chizer [Github](https://github.com/justinchizer), [Linkedin](https://www.linkedin.com/in/jchizer/)
9. Azadeh Khojandi [Github](https://github.com/Azadehkhojandi), [Linkedin](https://www.linkedin.com/in/azadeh-khojandi-ba441b3/)
## Contributing
Expand Down

0 comments on commit 4444c6a

Please sign in to comment.