Skip to content

Commit

Permalink
pushing doc changes so far
Browse files Browse the repository at this point in the history
  • Loading branch information
@jananivMS @frodopwns
jananivMS authored and frodopwns committed Oct 10, 2019
1 parent f12dc62 commit fbaccfb
Show file tree
Hide file tree
Showing 3 changed files with 381 additions and 349 deletions.
293 changes: 4 additions & 289 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
## Introduction

Kubernetes offers the facility of extending it's API through the concept of 'Operators' ([Introducing Operators: Putting Operational Knowledge into Software](https://coreos.com/blog/introducing-operators.html)).
Kubernetes offers the facility of extending it's API through the concept of 'Operators' ([Introducing Operators: Putting Operational Knowledge into Software](https://coreos.com/blog/introducing-operators.html)).

An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks.

Expand All @@ -22,297 +22,12 @@ The project was built using
[Kubebuilder](https://book.kubebuilder.io/)

## Azure Services supported
1. Resource Group

1. Resource Group
2. EventHub
3. Azure SQL

## Building and Testing the Operator

### Prerequisites And Assumptions

1. [GoLang](https://golang.org/dl/) is installed.
2. [Docker](https://docs.docker.com/install/) is installed and running.
3. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) Command-line tool is installed.
4. You have access to a Kubernetes cluster.
- It can be a local hosted Cluster like
[Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/),
[Kind](https://github.com/kubernetes-sigs/kind), or,
[Docker for desktop](https://blog.docker.com/2018/07/kubernetes-is-now-available-in-docker-desktop-stable-channel/) installed locally with RBAC enabled.
- If you opt for Azure Kubernetes Service ([AKS](https://azure.microsoft.com/en-au/services/kubernetes-service/)), you can set the current context to your cluster using the following command:

`az aks get-credentials --resource-group $RESOURCEGROUP_NAME --name $CLUSTER_NAME`

5. Install [Kubebuilder](https://book.kubebuilder.io/), following the linked installation instructions.
6. [Kustomize](https://github.com/kubernetes-sigs/kustomize) is also required. This must be installed via `make install-kustomize` (see section below).

Basic commands to check if you have an active Kubernetes cluster:

```shell
kubectl config get-contexts
kubectl cluster-info
kubectl version
kubectl get pods -n kube-system
```

### Quick start - Using VSCode with Remote-Containers extension

If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have you're environment set up and ready to go with everything you need to get started.

1. Open this project in VSCode.
2. Inside `.devcontainer`, create a file called `.env` and using the following template, copy your Service Principal's details.

```txt
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_SUBSCRIPTION_ID=
AZURE_TENANT_ID=
```
3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...` and hit enter.
4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing.
5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment.
**Note**: if you do not want to create a kind cluster when starting the devcontainer, comment out `"postCreateCommand": "make set-kindcluster",` in `.devcontainer/devcontainer.json` and reopen the devcontainer.
### Running the operator locally for testing
1. Clone the repository into the following folder `<GOPATH>/src/github.com/Azure`.
2. Make sure the environment variable `GO111MODULE` is set to `on`.
```bash
export GO111MODULE=on
```
3. Create a Service Principal
If you don't have a Service Principal create one from Azure CLI:
```bash
az ad sp create-for-rbac --role Contributor
```
Then make sure this service principal has rights assigned to provision resources in your Azure Subscription.
4. Set the environment variables `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_SUBSCRIPTION_ID`, `REQUEUE_AFTER`.
```shell
export AZURE_TENANT_ID=xxxxxxx
export AZURE_CLIENT_ID=yyyyyyy
export AZURE_CLIENT_SECRET=zzzzzz
export AZURE_SUBSCRIPTION_ID=aaaaaaa
export REQUEUE_AFTER=30
```
If running on Windows, the environment variables should not have quotes and should be set like below.
```shell
set AZURE_TENANT_ID=xxxxxxx
set AZURE_CLIENT_ID=yyyyyyy
set AZURE_CLIENT_SECRET=zzzzzz
set AZURE_SUBSCRIPTION_ID=aaaaaaa
set REQUEUE_AFTER=30
```
VSCode or `make run` should be run from the same session/command/terminal window where the environment variables are set.
5. Install [kustomize](https://github.com/kubernetes-sigs/kustomize) using `make install-kustomize`.
6. Install test certificates using `make generate-test-certs`.
7. Install the CRDs defined in the config/crd/bases folder using the command
```make install```
You will see output as below
```shell
kubectl apply -f config/crd/bases
customresourcedefinition.apiextensions.k8s.io/eventhubnamespaces.azure.microsoft.com created
customresourcedefinition.apiextensions.k8s.io/eventhubs.azure.microsoft.com created
customresourcedefinition.apiextensions.k8s.io/resourcegroups.azure.microsoft.com created
customresourcedefinition.apiextensions.k8s.io/sqldatabases.azure.microsoft.com created
customresourcedefinition.apiextensions.k8s.io/sqlfirewallrules.azure.microsoft.com created
customresourcedefinition.apiextensions.k8s.io/sqlservers.azure.microsoft.com configured
```
8. Run the operator locally using
```make run```
This will cause the operator to run and "watch" for events on this terminal. You will need to open a new terminal to trigger the creation of a custom resource.
You will see something like this on the terminal window indicating that the controller is running.
```shell
go fmt ./...
go vet ./...
go run ./main.go
2019-09-24T12:18:10.419-0600 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhub", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "resourcegroup", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhubnamespace", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/mutate-azure-microsoft-com-v1-eventhubnamespace"}
2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/validate-azure-microsoft-com-v1-eventhubnamespace"}
2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/mutate-azure-microsoft-com-v1-eventhub"}
2019-09-24T12:18:10.424-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/validate-azure-microsoft-com-v1-eventhub"}
2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlserver", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqldatabase", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlfirewallrule", "source": "kind source: /, Kind="}
2019-09-24T12:18:10.424-0600 INFO setup starting manager
2019-09-24T12:18:10.424-0600 INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhub"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "resourcegroup"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqldatabase"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlfirewallrule"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhubnamespace"}
2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlserver"}
2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Updated current TLS certiface
2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Starting certificate watcher
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqldatabase", "worker count": 1}
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "resourcegroup", "worker count": 1}
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlfirewallrule", "worker count": 1}
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhub", "worker count": 1}
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlserver", "worker count": 1}
2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhubnamespace", "worker count": 1}
```

9. Open a new terminal window. Trigger the creation of a custom resource using kubectl and the sample YAML file provided.
For instance, you would use the following command to create a SQL server:
```bash
kubectl apply -f config/samples/azure_v1_sqlserver.yaml
sqlserver.azure.microsoft.com/sqlserver-sample created
```

10. You should see logs on the other terminal from the operator when this custom resource is being created.
``` bash
2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Updated", "message": "finalizer sqlserver.finalizers.azure.com added"}
2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Submitting", "message": "starting resource reconciliation"}
2019-09-24T12:27:17.129-0600 INFO controllers.SqlServer mutating secret bundle
2019-09-24T12:27:17.144-0600 INFO controllers.SqlServer waiting for provision to take effect {"sqlserver": "default/sqlserver-sample1"}
2019-09-24T12:27:17.350-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194437"}, "reason": "Checking", "message": "instance in NotReady state"}
2019-09-24T12:27:17.359-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"}
2019-09-24T12:27:17.564-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in NotReady state"}
2019-09-24T12:27:17.570-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"}
2019-09-24T12:28:17.805-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in Ready state"}
2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"}
2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "}
2019-09-24T12:28:19.202-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Checking", "message": "instance in Ready state"}
2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"}
2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "}```
### Developing - Using VSCode with Remote-Containers extension
If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have your environment set up and ready to go, with everything you need to get started.
1. Open this project in VSCode.
2. Inside the folder `.devcontainer`, create a file called `.env` and using the following template, copy your environment variable details.
```txt
AZURE_CLIENT_ID=
AZURE_CLIENT_SECRET=
AZURE_SUBSCRIPTION_ID=
AZURE_TENANT_ID=
```
3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...`, select the ```Azure-service-operator``` folder and hit enter.
4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing.
5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment via the terminal inside VSCode.
**Note**: after the DevContainer has finished building, the kind cluster will start initialising and installing the Azure Service Operator in the background. This will take some time before it is available.
To see when the kind cluster is ready, use `docker ps -a` to list your running containers, look for `IMAGE` with the name `azure-service-operator_devcontainer_docker-in-docker...`. Using that image's `CONTAINER ID`, use `docker logs -f CONTAINER ID` to view the logs from the container setting up your cluster.
6. Use ```kubectl apply``` with the sample YAML files to create custom resources for testing.
For eg., use ```kubectl apply -f config/samples/azure_v1_sqlserver.yaml``` from the terminal to create a SQL server using the operator.
```kubectl describe SqlServer``` would show the events that indicate if the resource is created or being created.
### Deploying the operator on a Kubernetes cluster
1. Get your Kubernetes cluster setup:
(i) If you use Kind, install [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/)
```shell
GO111MODULE="on" go get sigs.k8s.io/[email protected] && kind create cluster
kind create cluster
export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info
IMG="docker.io/yourimage:tag" make docker-build
kind load docker-image docker.io/yourimage:tag --loglevel "trace"
make deploy
```
(ii) If you use Docker for Desktop, enable [Kubernetes](https://docs.docker.com/docker-for-mac/#kubernetes). For Windows users, you can follow [these](https://blog.docker.com/2018/01/docker-windows-desktop-now-kubernetes/) steps
2. Set up the Cluster
If you are using Kind:
```shell
make set-kindcluster
```
If you are not using Kind, it's a manual process, as follows:
a. Create the namespace you want to deploy the operator to. Skip this step if you use the ```default``` namespace
**Note** If you deploy the operator to any other namespace other than ```default```, you will need to give the operator ```cluster-admin``` permissiosn to be able to install the resources in a different namespace than the operator (default is the ```default``` namespace)
```shell
kubectl create namespace azureoperator-system
```
b. Set the ```azureoperatorsettings``` secret. Run the below command from the same terminal where you have set the environment variables for these values.
```shell
kubectl --namespace <Namespace_Operator_Is_Deployed_To> \
create secret generic azureoperatorsettings \
--from-literal=AZURE_CLIENT_ID="$AZURE_CLIENT_ID" \
--from-literal=AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET" \
--from-literal=AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID" \
--from-literal=AZURE_TENANT_ID="$AZURE_TENANT_ID"
```
c. Install [Cert Manager](https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html)
```shell
make install-cert-manager
```
8. Install the azure_v1_eventhub CRD in the configured Kubernetes cluster folder ~/.kube/config,
Use `kubectl apply -f config/crd/bases` or `make install`
## Add support for a new Azure service
TODO: Fill out
## How to extend the operator and build your own images
### Updating the Azure operator
This repository is generated by [Kubebuilder](https://book.kubebuilder.io/).
To Extend the operator `github.com/Azure/azure-service-operator`:
1. Run `go mod download` to download dependencies. It doesn't show any progress bar and takes a while to download all of dependencies.
2. Update `api\v1\eventhub_types.go`.
3. Regenerate CRD `make manifests`.
4. Install updated CRD `make install`
5. Generate code `make generate`
6. Update operator `controller\eventhub_controller.go`
7. Update tests and run `make test`
8. Deploy `make deploy`
If you make changes to the operator and want to update the deployment without recreating the cluster (when testing locally), you can use the `make update` to update your Azure Operator pod. If you need to rebuild the docker image without cache, use `make ARGS="--no-cache" update`.
[Building, testing and running the operator](/docs/development.md)

## Testing

Expand Down
Loading

0 comments on commit fbaccfb

Please sign in to comment.