diff --git a/README.md b/README.md index 0171ab14643..d228322db3a 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ ## Introduction -Kubernetes offers the facility of extending it's API through the concept of 'Operators' ([Introducing Operators: Putting Operational Knowledge into Software](https://coreos.com/blog/introducing-operators.html)). +Kubernetes offers the facility of extending it's API through the concept of 'Operators' ([Introducing Operators: Putting Operational Knowledge into Software](https://coreos.com/blog/introducing-operators.html)). An Operator is an application-specific controller that extends the Kubernetes API to create, configure, and manage instances of complex stateful applications on behalf of a Kubernetes user. It builds upon the basic Kubernetes resource and controller concepts but includes domain or application-specific knowledge to automate common tasks. @@ -22,297 +22,12 @@ The project was built using [Kubebuilder](https://book.kubebuilder.io/) ## Azure Services supported -1. Resource Group + +1. Resource Group 2. EventHub 3. Azure SQL -## Building and Testing the Operator - -### Prerequisites And Assumptions - -1. [GoLang](https://golang.org/dl/) is installed. -2. [Docker](https://docs.docker.com/install/) is installed and running. -3. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) Command-line tool is installed. -4. You have access to a Kubernetes cluster. - - It can be a local hosted Cluster like - [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/), - [Kind](https://github.com/kubernetes-sigs/kind), or, - [Docker for desktop](https://blog.docker.com/2018/07/kubernetes-is-now-available-in-docker-desktop-stable-channel/) installed locally with RBAC enabled. - - If you opt for Azure Kubernetes Service ([AKS](https://azure.microsoft.com/en-au/services/kubernetes-service/)), you can set the current context to your cluster using the following command: - - `az aks get-credentials --resource-group $RESOURCEGROUP_NAME --name $CLUSTER_NAME` - -5. Install [Kubebuilder](https://book.kubebuilder.io/), following the linked installation instructions. -6. [Kustomize](https://github.com/kubernetes-sigs/kustomize) is also required. This must be installed via `make install-kustomize` (see section below). - -Basic commands to check if you have an active Kubernetes cluster: - -```shell - kubectl config get-contexts - kubectl cluster-info - kubectl version - kubectl get pods -n kube-system -``` - -### Quick start - Using VSCode with Remote-Containers extension - -If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have you're environment set up and ready to go with everything you need to get started. - -1. Open this project in VSCode. -2. Inside `.devcontainer`, create a file called `.env` and using the following template, copy your Service Principal's details. - - ```txt - AZURE_CLIENT_ID= - - AZURE_CLIENT_SECRET= - - AZURE_SUBSCRIPTION_ID= - - AZURE_TENANT_ID= - ``` - -3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...` and hit enter. -4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing. -5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment. - -**Note**: if you do not want to create a kind cluster when starting the devcontainer, comment out `"postCreateCommand": "make set-kindcluster",` in `.devcontainer/devcontainer.json` and reopen the devcontainer. - - -### Running the operator locally for testing - -1. Clone the repository into the following folder `/src/github.com/Azure`. - -2. Make sure the environment variable `GO111MODULE` is set to `on`. - ```bash - export GO111MODULE=on - ``` - -3. Create a Service Principal - If you don't have a Service Principal create one from Azure CLI: - - ```bash - az ad sp create-for-rbac --role Contributor - ``` - - Then make sure this service principal has rights assigned to provision resources in your Azure Subscription. - -4. Set the environment variables `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_SUBSCRIPTION_ID`, `REQUEUE_AFTER`. - - ```shell - export AZURE_TENANT_ID=xxxxxxx - export AZURE_CLIENT_ID=yyyyyyy - export AZURE_CLIENT_SECRET=zzzzzz - export AZURE_SUBSCRIPTION_ID=aaaaaaa - export REQUEUE_AFTER=30 - ``` - - If running on Windows, the environment variables should not have quotes and should be set like below. - - ```shell - set AZURE_TENANT_ID=xxxxxxx - set AZURE_CLIENT_ID=yyyyyyy - set AZURE_CLIENT_SECRET=zzzzzz - set AZURE_SUBSCRIPTION_ID=aaaaaaa - set REQUEUE_AFTER=30 - ``` - VSCode or `make run` should be run from the same session/command/terminal window where the environment variables are set. - -5. Install [kustomize](https://github.com/kubernetes-sigs/kustomize) using `make install-kustomize`. - -6. Install test certificates using `make generate-test-certs`. - -7. Install the CRDs defined in the config/crd/bases folder using the command - ```make install``` - You will see output as below - - ```shell - kubectl apply -f config/crd/bases - customresourcedefinition.apiextensions.k8s.io/eventhubnamespaces.azure.microsoft.com created - customresourcedefinition.apiextensions.k8s.io/eventhubs.azure.microsoft.com created - customresourcedefinition.apiextensions.k8s.io/resourcegroups.azure.microsoft.com created - customresourcedefinition.apiextensions.k8s.io/sqldatabases.azure.microsoft.com created - customresourcedefinition.apiextensions.k8s.io/sqlfirewallrules.azure.microsoft.com created - customresourcedefinition.apiextensions.k8s.io/sqlservers.azure.microsoft.com configured - ``` - -8. Run the operator locally using - ```make run``` - This will cause the operator to run and "watch" for events on this terminal. You will need to open a new terminal to trigger the creation of a custom resource. - - You will see something like this on the terminal window indicating that the controller is running. - - ```shell - go fmt ./... - go vet ./... - go run ./main.go - 2019-09-24T12:18:10.419-0600 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} - - 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhub", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "resourcegroup", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhubnamespace", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/mutate-azure-microsoft-com-v1-eventhubnamespace"} - 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/validate-azure-microsoft-com-v1-eventhubnamespace"} - 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/mutate-azure-microsoft-com-v1-eventhub"} - 2019-09-24T12:18:10.424-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/validate-azure-microsoft-com-v1-eventhub"} - 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlserver", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqldatabase", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlfirewallrule", "source": "kind source: /, Kind="} - 2019-09-24T12:18:10.424-0600 INFO setup starting manager - 2019-09-24T12:18:10.424-0600 INFO controller-runtime.manager starting metrics server {"path": "/metrics"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhub"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "resourcegroup"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqldatabase"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlfirewallrule"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhubnamespace"} - 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlserver"} - 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Updated current TLS certiface - 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Starting certificate watcher - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqldatabase", "worker count": 1} - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "resourcegroup", "worker count": 1} - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlfirewallrule", "worker count": 1} - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhub", "worker count": 1} - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlserver", "worker count": 1} - 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhubnamespace", "worker count": 1} - ``` - -9. Open a new terminal window. Trigger the creation of a custom resource using kubectl and the sample YAML file provided. - For instance, you would use the following command to create a SQL server: - ```bash - kubectl apply -f config/samples/azure_v1_sqlserver.yaml - sqlserver.azure.microsoft.com/sqlserver-sample created - ``` - -10. You should see logs on the other terminal from the operator when this custom resource is being created. - ``` bash - 2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Updated", "message": "finalizer sqlserver.finalizers.azure.com added"} - 2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Submitting", "message": "starting resource reconciliation"} - 2019-09-24T12:27:17.129-0600 INFO controllers.SqlServer mutating secret bundle - 2019-09-24T12:27:17.144-0600 INFO controllers.SqlServer waiting for provision to take effect {"sqlserver": "default/sqlserver-sample1"} - 2019-09-24T12:27:17.350-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194437"}, "reason": "Checking", "message": "instance in NotReady state"} - 2019-09-24T12:27:17.359-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} - 2019-09-24T12:27:17.564-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in NotReady state"} - 2019-09-24T12:27:17.570-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} - 2019-09-24T12:28:17.805-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in Ready state"} - 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"} - 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "} - 2019-09-24T12:28:19.202-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Checking", "message": "instance in Ready state"} - 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"} - 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "}``` - - - - -### Developing - Using VSCode with Remote-Containers extension - -If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have your environment set up and ready to go, with everything you need to get started. - -1. Open this project in VSCode. -2. Inside the folder `.devcontainer`, create a file called `.env` and using the following template, copy your environment variable details. - - ```txt - AZURE_CLIENT_ID= - - AZURE_CLIENT_SECRET= - - AZURE_SUBSCRIPTION_ID= - - AZURE_TENANT_ID= - ``` - -3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...`, select the ```Azure-service-operator``` folder and hit enter. - -4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing. - -5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment via the terminal inside VSCode. - -**Note**: after the DevContainer has finished building, the kind cluster will start initialising and installing the Azure Service Operator in the background. This will take some time before it is available. - -To see when the kind cluster is ready, use `docker ps -a` to list your running containers, look for `IMAGE` with the name `azure-service-operator_devcontainer_docker-in-docker...`. Using that image's `CONTAINER ID`, use `docker logs -f CONTAINER ID` to view the logs from the container setting up your cluster. - -6. Use ```kubectl apply``` with the sample YAML files to create custom resources for testing. -For eg., use ```kubectl apply -f config/samples/azure_v1_sqlserver.yaml``` from the terminal to create a SQL server using the operator. -```kubectl describe SqlServer``` would show the events that indicate if the resource is created or being created. - -### Deploying the operator on a Kubernetes cluster - -1. Get your Kubernetes cluster setup: - - (i) If you use Kind, install [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) - - ```shell - GO111MODULE="on" go get sigs.k8s.io/kind@v0.4.0 && kind create cluster - kind create cluster - export KUBECONFIG="$(kind get kubeconfig-path --name="kind")" - kubectl cluster-info - IMG="docker.io/yourimage:tag" make docker-build - kind load docker-image docker.io/yourimage:tag --loglevel "trace" - make deploy - ``` - - (ii) If you use Docker for Desktop, enable [Kubernetes](https://docs.docker.com/docker-for-mac/#kubernetes). For Windows users, you can follow [these](https://blog.docker.com/2018/01/docker-windows-desktop-now-kubernetes/) steps - - -2. Set up the Cluster - - If you are using Kind: - - ```shell - make set-kindcluster - ``` - - If you are not using Kind, it's a manual process, as follows: - - a. Create the namespace you want to deploy the operator to. Skip this step if you use the ```default``` namespace - - **Note** If you deploy the operator to any other namespace other than ```default```, you will need to give the operator ```cluster-admin``` permissiosn to be able to install the resources in a different namespace than the operator (default is the ```default``` namespace) - - ```shell - kubectl create namespace azureoperator-system - ``` - - b. Set the ```azureoperatorsettings``` secret. Run the below command from the same terminal where you have set the environment variables for these values. - - ```shell - kubectl --namespace \ - create secret generic azureoperatorsettings \ - --from-literal=AZURE_CLIENT_ID="$AZURE_CLIENT_ID" \ - --from-literal=AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET" \ - --from-literal=AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID" \ - --from-literal=AZURE_TENANT_ID="$AZURE_TENANT_ID" - ``` - - c. Install [Cert Manager](https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html) - - ```shell - make install-cert-manager - ``` - -8. Install the azure_v1_eventhub CRD in the configured Kubernetes cluster folder ~/.kube/config, - - Use `kubectl apply -f config/crd/bases` or `make install` - -## Add support for a new Azure service - -TODO: Fill out - -## How to extend the operator and build your own images - -### Updating the Azure operator - -This repository is generated by [Kubebuilder](https://book.kubebuilder.io/). - -To Extend the operator `github.com/Azure/azure-service-operator`: - -1. Run `go mod download` to download dependencies. It doesn't show any progress bar and takes a while to download all of dependencies. -2. Update `api\v1\eventhub_types.go`. -3. Regenerate CRD `make manifests`. -4. Install updated CRD `make install` -5. Generate code `make generate` -6. Update operator `controller\eventhub_controller.go` -7. Update tests and run `make test` -8. Deploy `make deploy` - -If you make changes to the operator and want to update the deployment without recreating the cluster (when testing locally), you can use the `make update` to update your Azure Operator pod. If you need to rebuild the docker image without cache, use `make ARGS="--no-cache" update`. +[Building, testing and running the operator](/docs/development.md) ## Testing diff --git a/docs/azuresql/azuresql.md b/docs/azuresql/azuresql.md index 5d044894b1b..6af5ccfe3f1 100644 --- a/docs/azuresql/azuresql.md +++ b/docs/azuresql/azuresql.md @@ -1,19 +1,87 @@ -# Eventhub Operator +# Azure SQL Operator -## Resources supported -1. Eventhub -2. Eventhub Namespace -3. Consumer groups +## Resources Supported +The Azure SQL operator can be used to provision the following resources -## Deploying resources +1. Azure SQL server +2. SQL database +3. SQL firewall rule +4. Action (Rolling user credentials for the SQL server) -kubectl Apply -kubectl Delete - Talk about deleting parent deletes child +## Deploying SQL Resources -## Troubleshooting resource provisioning -kubectl describe to see events +We will use two terminal windows to deploy SQL resouces. +- Terminal Window #1 - used to run the operator +- Terminal Window #2 - used to create new Azure resources. -# How would you use the Eventhub Operator from a real application +In the first terminal window, run the operator by running the following commands: -describe demo app +```bash +make install +make run +``` + +Now that the operator is running, open your second terminal window. Here we will use our yaml files to create different Azure resources. + +### 1. Create a Resource Group + +Create a resource group by running the following yaml file. Update the name variable to your preferred resource group name. + +```bash +kubectl create -f config/samples/azure_v1_resourcegroup.yaml +``` + +### 2. Create SQL resources + +Create your SQL resources by running the following yaml files. Update the resource group name, and SQL resource names to your preferred names. + +```bash +kubectl create -f config/samples/azure_v1_sqldatabase.yaml +``` + +```bash +kubectl create -f config/samples/azure_v1_sqlfirewallrule.yaml +``` + +```bash +kubectl create -f config/samples/azure_v1_sqlserver.yaml +``` + + + +### View SQL Resources + +To view your created SQL resources, such as sqlserver, run the following command: + +```bash +kubectl get sqlserver +``` + +Your servers should be displayed with their name and age. + +### Delete a SQL Resource + +To delete an existing resource from Kubernetes and Azure, such as SQL server, run: + +```bash +kubectl edit sqlserver $sqlservername +``` + +Remove the lines under the finalizer, and then delete the sql server instance: + +```bash +kubectl delete sqlserver $sqlservername +``` + +The following message should appear: + +`sqlserver.azure.microsoft.com "$sqlservername" deleted.` + +## Troubleshooting Provisioning Resources + +kubectl describe to see events + +## Demo App + +Watch this demo to observe how you would you use the Azure SQL Operator from a real application. diff --git a/docs/development.md b/docs/development.md index c06ba3be01e..4a3cf2c714e 100644 --- a/docs/development.md +++ b/docs/development.md @@ -1,102 +1,351 @@ -# Development +# Building and testing the Operator -## Prerequisites +## Running the operator -* a Kubernetes cluster to run against. You can use [KIND](https://sigs.k8s.io/kind) to get a local cluster for testing, or run against a remote cluster, e.g. [Azure Kubernetes Service](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough). -* [kubebuilder](https://book.kubebuilder.io/quick-start.html#installation) -* [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) +### Prerequisites And Assumptions -## Deploy Operator and Test +1. [GoLang](https://golang.org/dl/) is installed. +2. [Docker](https://docs.docker.com/install/) is installed and running. +3. [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) Command-line tool is installed. +4. You have access to a Kubernetes cluster. + - It can be a local hosted Cluster like + [Minikube](https://kubernetes.io/docs/tasks/tools/install-minikube/), + [Kind](https://github.com/kubernetes-sigs/kind), or, + [Docker for desktop](https://blog.docker.com/2018/07/kubernetes-is-now-available-in-docker-desktop-stable-channel/) installed locally with RBAC enabled. + - If you opt for Azure Kubernetes Service ([AKS](https://azure.microsoft.com/en-au/services/kubernetes-service/)), you can set the current context to your cluster using the following command: -### Test it locally + `az aks get-credentials --resource-group $RESOURCEGROUP_NAME --name $CLUSTER_NAME` -1. Create Cluster. +5. Install [Kubebuilder](https://book.kubebuilder.io/), following the linked installation instructions. +6. [Kustomize](https://github.com/kubernetes-sigs/kustomize) is also required. This must be installed via `make install-kustomize` (see section below). - ``` - kind create cluster - export KUBECONFIG="$(kind get kubeconfig-path --name="kind")" +Basic commands to check if you have an active Kubernetes cluster: + +```shell + kubectl config get-contexts kubectl cluster-info + kubectl version + kubectl get pods -n kube-system +``` + +### Running the operator locally for testing + +1. Clone the repository into the following folder `/src/github.com/Azure`. + +2. Make sure the environment variable `GO111MODULE` is set to `on`. + ```bash + export GO111MODULE=on + ``` + +3. Create a Service Principal + If you don't have a Service Principal create one from Azure CLI: + + ```bash + az ad sp create-for-rbac --role Contributor ``` -1. Install CRDs. + Then make sure this service principal has rights assigned to provision resources in your Azure Subscription and resource group. + +4. Set the environment variables `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`, `AZURE_SUBSCRIPTION_ID`, `REQUEUE_AFTER`. + ```shell + export AZURE_TENANT_ID=xxxxxxx + export AZURE_CLIENT_ID=yyyyyyy + export AZURE_CLIENT_SECRET=zzzzzz + export AZURE_SUBSCRIPTION_ID=aaaaaaa + export REQUEUE_AFTER=30 ``` - make install + + If running on Windows, the environment variables should not have quotes and should be set like below. + + ```shell + set AZURE_TENANT_ID=xxxxxxx + set AZURE_CLIENT_ID=yyyyyyy + set AZURE_CLIENT_SECRET=zzzzzz + set AZURE_SUBSCRIPTION_ID=aaaaaaa + set REQUEUE_AFTER=30 ``` + VSCode or `make run` should be run from the same session/command/terminal window where the environment variables are set. -1. Run Controller. +5. Install [kustomize](https://github.com/kubernetes-sigs/kustomize) using `make install-kustomize`. - Setup the environment variables: +6. Install test certificates using `make generate-test-certs`. +7. Install the CRDs defined in the config/crd/bases folder using the command + ```make install``` + You will see output as below + + ```shell + kubectl apply -f config/crd/bases + customresourcedefinition.apiextensions.k8s.io/eventhubnamespaces.azure.microsoft.com created + customresourcedefinition.apiextensions.k8s.io/eventhubs.azure.microsoft.com created + customresourcedefinition.apiextensions.k8s.io/resourcegroups.azure.microsoft.com created + customresourcedefinition.apiextensions.k8s.io/sqldatabases.azure.microsoft.com created + customresourcedefinition.apiextensions.k8s.io/sqlfirewallrules.azure.microsoft.com created + customresourcedefinition.apiextensions.k8s.io/sqlservers.azure.microsoft.com configured ``` - export CLOUD_NAME=AzurePublicCloud - export TENANT_ID= - export SUBSCRIPTION_ID= - export CLIENT_ID= - export CLIENT_SECRET= + +8. Run the operator locally using + ```make run``` + This will cause the operator to run and "watch" for events on this terminal. You will need to open a new terminal to trigger the creation of a custom resource. + + You will see something like this on the terminal window indicating that the controller is running. + + ```shell + go fmt ./... + go vet ./... + go run ./main.go + 2019-09-24T12:18:10.419-0600 INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"} + + 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhub", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "resourcegroup", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.419-0600 INFO controller-runtime.controller Starting EventSource {"controller": "eventhubnamespace", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/mutate-azure-microsoft-com-v1-eventhubnamespace"} + 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=EventhubNamespace", "path": "/validate-azure-microsoft-com-v1-eventhubnamespace"} + 2019-09-24T12:18:10.420-0600 INFO controller-runtime.builder Registering a mutating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/mutate-azure-microsoft-com-v1-eventhub"} + 2019-09-24T12:18:10.424-0600 INFO controller-runtime.builder Registering a validating webhook {"GVK": "azure.microsoft.com/v1, Kind=Eventhub", "path": "/validate-azure-microsoft-com-v1-eventhub"} + 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlserver", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqldatabase", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.424-0600 INFO controller-runtime.controller Starting EventSource {"controller": "sqlfirewallrule", "source": "kind source: /, Kind="} + 2019-09-24T12:18:10.424-0600 INFO setup starting manager + 2019-09-24T12:18:10.424-0600 INFO controller-runtime.manager starting metrics server {"path": "/metrics"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhub"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "resourcegroup"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqldatabase"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlfirewallrule"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "eventhubnamespace"} + 2019-09-24T12:18:10.526-0600 INFO controller-runtime.controller Starting Controller {"controller": "sqlserver"} + 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Updated current TLS certiface + 2019-09-24T12:18:10.527-0600 INFO controller-runtime.certwatcher Starting certificate watcher + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqldatabase", "worker count": 1} + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "resourcegroup", "worker count": 1} + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlfirewallrule", "worker count": 1} + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhub", "worker count": 1} + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "sqlserver", "worker count": 1} + 2019-09-24T12:18:10.626-0600 INFO controller-runtime.controller Starting workers {"controller": "eventhubnamespace", "worker count": 1} + ``` + +9. Open a new terminal window. Trigger the creation of a custom resource using kubectl and the sample YAML file provided. + For instance, you would use the following command to create a SQL server: + ```shell + kubectl apply -f config/samples/azure_v1_sqlserver.yaml + sqlserver.azure.microsoft.com/sqlserver-sample created + ``` + +10. You should see logs on the other terminal from the operator when this custom resource is being created. + ``` shell + 2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Updated", "message": "finalizer sqlserver.finalizers.azure.com added"} + 2019-09-24T12:27:12.450-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194427"}, "reason": "Submitting", "message": "starting resource reconciliation"} + 2019-09-24T12:27:17.129-0600 INFO controllers.SqlServer mutating secret bundle + 2019-09-24T12:27:17.144-0600 INFO controllers.SqlServer waiting for provision to take effect {"sqlserver": "default/sqlserver-sample1"} + 2019-09-24T12:27:17.350-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194437"}, "reason": "Checking", "message": "instance in NotReady state"} + 2019-09-24T12:27:17.359-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} + 2019-09-24T12:27:17.564-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in NotReady state"} + 2019-09-24T12:27:17.570-0600 INFO controllers.SqlServer Got ignorable error {"sqlserver": "default/sqlserver-sample1", "type": "ResourceNotFound"} + 2019-09-24T12:28:17.805-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194439"}, "reason": "Checking", "message": "instance in Ready state"} + 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"} + 2019-09-24T12:28:19.010-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "} + 2019-09-24T12:28:19.202-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Checking", "message": "instance in Ready state"} + 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.controller Successfully Reconciled {"controller": "sqlserver", "request": "default/sqlserver-sample1"} + 2019-09-24T12:28:20.331-0600 DEBUG controller-runtime.manager.events Normal {"object": {"kind":"SqlServer","namespace":"default","name":"sqlserver-sample1","uid":"ed3774af-def8-11e9-90c4-025000000001","apiVersion":"azure.microsoft.com/v1","resourceVersion":"194518"}, "reason": "Provisioned", "message": "sqlserver sqlserver-sample1 provisioned "}``` + + + + + + +If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have your environment set up and ready to go, with everything you need to get started. + +1. Open this project in VSCode. +2. Inside the folder `.devcontainer`, create a file called `.env` and using the following template, copy your environment variable details. + + ```txt + AZURE_CLIENT_ID= + + AZURE_CLIENT_SECRET= + + AZURE_SUBSCRIPTION_ID= + + AZURE_TENANT_ID= ``` - Run your controller (this will run in the foreground, so switch to a new terminal if you want to leave it running): +3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...`, select the ```Azure-service-operator``` folder and hit enter. - ``` - make run +4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing. + +5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment via the terminal inside VSCode. + +**Note**: after the DevContainer has finished building, the kind cluster will start initialising and installing the Azure Service Operator in the background. This will take some time before it is available. + +To see when the kind cluster is ready, use `docker ps -a` to list your running containers, look for `IMAGE` with the name `azure-service-operator_devcontainer_docker-in-docker...`. Using that image's `CONTAINER ID`, use `docker logs -f CONTAINER ID` to view the logs from the container setting up your cluster. + +6. Use ```kubectl apply``` with the sample YAML files to create custom resources for testing. +For eg., use ```kubectl apply -f config/samples/azure_v1_sqlserver.yaml``` from the terminal to create a SQL server using the operator. +```kubectl describe SqlServer``` would show the events that indicate if the resource is created or being created. + +### Deploying the operator on a Kubernetes cluster + +1. Create your Kubernetes cluster + + (i) If you use Kind, install [Kind](https://kind.sigs.k8s.io/docs/user/quick-start/) + + ```shell + GO111MODULE="on" go get sigs.k8s.io/kind@v0.4.0 && kind create cluster + kind create cluster + export KUBECONFIG="$(kind get kubeconfig-path --name="kind")" + kubectl cluster-info + IMG="docker.io/yourimage:tag" make docker-build + kind load docker-image docker.io/yourimage:tag --loglevel "trace" + make deploy ``` - Refer to [kubebuilder's doc](https://book.kubebuilder.io/quick-start.html#test-it-out-locally). + (ii) If you use Docker for Desktop, enable [Kubernetes](https://docs.docker.com/docker-for-mac/#kubernetes). For Windows users, you can follow [these](https://blog.docker.com/2018/01/docker-windows-desktop-now-kubernetes/) steps -1. Create a Custom Resource. + (iii) If you use Azure Kubernetes Service, create a cluster from the Azure portal or using [Azure CLI](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#create-aks-cluster) - Create your CR (make sure to edit them first to specify the fields). Example: + ```shell + az aks create -g -n + ``` + You could also create the AKS cluster from the Azure portal. Once you create the cluster, use the following command to set it as the current cluster. + + ```shell + az aks get-credentials -g -n ``` - kubectl apply -f examples/service/v1alpha1/storage.yaml + +2. Set up the Cluster + + If you are using Kind: + + ```shell + make set-kindcluster ``` -### Test it on a remote cluster + If you are not using Kind, it's a manual process, as follows: -1. Create Cluster. + a. Create the namespace you want to deploy the operator to. + + **Note** The scripts currently are configured to deploy to the ```azureoperator-system``` namespace + ```shell + kubectl create namespace azureoperator-system ``` - az aks create -g -n - az aks get-credentials -g -n - kubectl cluster-info + + b. Set the ```azureoperatorsettings``` secret. + + Run the below command from the same terminal where you have set the environment variables for these values. If you haven't done this yet, first do that before running the below command. + + ```shell + kubectl --namespace azureoperator-system \ + create secret generic azureoperatorsettings \ + --from-literal=AZURE_CLIENT_ID="$AZURE_CLIENT_ID" \ + --from-literal=AZURE_CLIENT_SECRET="$AZURE_CLIENT_SECRET" \ + --from-literal=AZURE_SUBSCRIPTION_ID="$AZURE_SUBSCRIPTION_ID" \ + --from-literal=AZURE_TENANT_ID="$AZURE_TENANT_ID" ``` -1. Install CRDs. + c. Install [Cert Manager](https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html) - ``` - make install + ```shell + make install-cert-manager ``` -1. Build and Push the image. +8. Build the image and push it to docker hub. + ```shell + docker login + IMG=/: make docker-build + IMG=/: make docker-push ``` - IMG= make build-and-push + +9. Give the default service account on the azureoperator-system namespace "cluster-admin" access to the "default" namespace where the operator deploys the resources by default. + +**Note** If you deploy resources to a different namespace than default then you will need to provide permission to that namespace too. + + ```shell + kubectl create rolebinding default-admin-binding --clusterrole=cluster-admin --user=system:serviceaccount:azureoperator-system:default --namespace=default ``` +10. Install the Custom Resource Definitions (CRDs) in the configured Kubernetes cluster - Update kustomize image patch file `config/default/manager_image_patch.yaml` for manager resource manually. + Use `make install` -1. Run Controller. +11. Deploy the operator to the Kubernetes cluster - Update `config/manager/manager.yaml` with your service principal. + ```shell + IMG=/: make deploy + ``` +12. Check that the operator is deployed to the cluster using the following commands. + ```shell + kubectl get namespaces + kubectl get pods -n azureoperator-system ``` - make deploy + +13. You can view the logs from the operator using the following command. The `podname` is the name of the pod in the output from `kubectl get pods -n azureoperator-system`, `manager` is the name of the container inside the pod. + + ```shell + kubectl logs -c manager -n azureoperator-system ``` -1. Create a Custom Resource. - Create your CR (make sure to edit them first to specify the fields). Example: +This repository is generated by [Kubebuilder](https://book.kubebuilder.io/). + +To Extend the operator `github.com/Azure/azure-service-operator`: + +1. Run `go mod download` to download dependencies. It doesn't show any progress bar and takes a while to download all of dependencies. +2. Update `api\v1\eventhub_types.go`. +3. Regenerate CRD `make manifests`. +4. Install updated CRD `make install` +5. Generate code `make generate` +6. Update operator `controller\eventhub_controller.go` +7. Update tests and run `make test` +8. Deploy `make deploy` + +If you make changes to the operator and want to update the deployment without recreating the cluster (when testing locally), you can use the `make update` to update your Azure Operator pod. If you need to rebuild the docker image without cache, use `make ARGS="--no-cache" update`. + +## Developing + +### Using VSCode with Remote-Containers extension + +If you're using VSCode with [Remote - Containers](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers) extensions installed, you can quickly have your environment set up and ready to go, with everything you need to get started. + +1. Open this project in VSCode. +2. Inside the folder `.devcontainer`, create a file called `.env` and using the following template, copy your environment variable details. + + ```txt + AZURE_CLIENT_ID= + + AZURE_CLIENT_SECRET= + + AZURE_SUBSCRIPTION_ID= + + AZURE_TENANT_ID= ``` - kubectl apply -f examples/service/v1alpha1/storage.yaml - ``` + +3. Open the Command Pallet (`Command+Shift+P` on MacOS or `CTRL+Shift+P` on Windows), type `Remote-Containers: Open Folder in Container...`, select the ```Azure-service-operator``` folder and hit enter. + +4. VSCode will relaunch and start building our development container. This will install all the necessary dependencies required for you to begin developing. + +5. Once the container has finished building, you can now start testing your Azure Service Operator within your own local kubernetes environment via the terminal inside VSCode. + +**Note**: after the DevContainer has finished building, the kind cluster will start initialising and installing the Azure Service Operator in the background. This will take some time before it is available. + +To see when the kind cluster is ready, use `docker ps -a` to list your running containers, look for `IMAGE` with the name `azure-service-operator_devcontainer_docker-in-docker...`. Using that image's `CONTAINER ID`, use `docker logs -f CONTAINER ID` to view the logs from the container setting up your cluster. + +6. Use ```kubectl apply``` with the sample YAML files to create custom resources for testing. +For eg., use ```kubectl apply -f config/samples/azure_v1_sqlserver.yaml``` from the terminal to create a SQL server using the operator. +```kubectl describe SqlServer``` would show the events that indicate if the resource is created or being created. ## Add a New Custom Resource +If you want to create a new custom resource for a new Azure service, you will need to follow the following steps: + +1. Add a New API - + ### 1. Add a New API -``` +```shell kubebuilder create api --group service --version v1alpha1 --kind ``` @@ -113,9 +362,9 @@ Refer to [kubebuilder's doc](https://book.kubebuilder.io/cronjob-tutorial/api-de Note: -* Don't forget to add `// +kubebuilder:subresource:status` if we want a status subresource. +- Don't forget to add `// +kubebuilder:subresource:status` if we want a status subresource. -* Run `make manifests` if you find the property you add doesn't work. +- Run `make manifests` if you find the property you add doesn't work. ### 3. Delete external resource