From cf566f01aa3a10d6828966b2944bc24712c83db9 Mon Sep 17 00:00:00 2001 From: Alessandro De Maria Date: Wed, 11 Dec 2024 09:40:18 +0000 Subject: [PATCH 1/2] Initial update --- .../{kapitan_overview.md => core_concepts.md} | 23 +++-- docs/pages/inventory/backends.md | 12 +++ docs/pages/inventory/introduction.md | 84 +++++++++++++------ mkdocs.yml | 6 +- 4 files changed, 91 insertions(+), 34 deletions(-) rename docs/pages/{kapitan_overview.md => core_concepts.md} (78%) create mode 100644 docs/pages/inventory/backends.md diff --git a/docs/pages/kapitan_overview.md b/docs/pages/core_concepts.md similarity index 78% rename from docs/pages/kapitan_overview.md rename to docs/pages/core_concepts.md index 0313444d9..bb19f801d 100644 --- a/docs/pages/kapitan_overview.md +++ b/docs/pages/core_concepts.md @@ -93,23 +93,36 @@ Let's explore these concepts in a way that's accessible to new users: ## [**Inventory**](/pages/inventory/introduction/) At the core of **Kapitan** lies the **Inventory**, a structured database of variables meticulously organized in YAML files. -This hierarchical setup serves as the single source of truth (SSOT) for your system's configurations, making it easier to manage and reference the essential components of your infrastructure. Whether you're dealing with Kubernetes configurations, Terraform resources, or even business logic, the Inventory allows you to define and store these elements efficiently. This central repository then feeds into **Kapitan**'s templating engines, enabling seamless reuse across various applications and services. - +This hierarchical setup serves as the single source of truth (SSOT) for your system's configurations, making it easier to manage and reference the essential components of your infrastructure. + +Whether you're dealing with Kubernetes configurations, Terraform resources, or even business logic, the Inventory allows you to define and store these elements efficiently. This central repository then feeds into **Kapitan**'s templating engines, enabling seamless reuse across various applications and services. + ## [**Input Types**](/pages/input_types/introduction/) -**Kapitan** takes the information stored in the Inventory and brings it to life through its templating engines upon compilation. This process transforms static data into dynamic configurations, capable of generating a wide array of outputs like Kubernetes manifests, Terraform plans, documentation, and scripts. It's about making your configurations work for you, tailored to the specific needs of your projects. +**Kapitan** takes the information stored in the Inventory and brings it to life through its templating engines upon compilation. Some of the supported input types are: **Python**, **Jinja2**, **Jsonnet**, **Helm**, and we're adding more soon. + +This process transforms static data into dynamic configurations, capable of generating a wide array of outputs like Kubernetes manifests, Terraform plans, documentation, and scripts. + +It's about making your configurations work for you, tailored to the specific needs of your projects. See [**Input Types**](/pages/input_types/introduction/) for more + ### [**Generators**](https://generators.kapitan.dev) -[**Generators**](https://generators.kapitan.dev) offer a straightforward entry point into using **Kapitan**, requiring minimal to no coding experience. These are essentially pre-made templates that allow you to generate common configuration files, such as Kubernetes manifests, directly from your Inventory data. **Kapitan** provides a wealth of resources, including the [**Kapitan Reference**](https://github.com/kapicorp/kapitan-reference) GitHub repository and various blog posts, to help users get up and running with generators. +[**Generators**](https://generators.kapitan.dev) offer a straightforward entry point into using **Kapitan**, requiring minimal to no coding experience. + +These are essentially pre-made templates that allow you to generate common configuration files, such as Kubernetes manifests, directly from your Inventory data. + +**Kapitan** provides a wealth of resources, including the [**Kapitan Reference**](https://github.com/kapicorp/kapitan-reference) GitHub repository and various blog posts, to help users get up and running with generators. ### [**Kadet**](https://github.com/kapicorp/kapitan-reference) -For those looking to leverage the full power of **Kapitan**, [**Kadet**](https://github.com/kapicorp/kapitan-reference) introduces a method to define and reuse complex configurations through Python. This internal library facilitates the creation of JSON and YAML manifests programmatically, offering a higher degree of customization and reuse. [**Kadet**](https://github.com/kapicorp/kapitan-reference) empowers users to craft intricate configurations with the simplicity and flexibility of Python. +For those looking to leverage the full power of **Kapitan**, [**Kadet**](https://github.com/kapicorp/kapitan-reference) introduces a method to define and reuse complex configurations through Python. + +This internal library facilitates the creation of JSON and YAML manifests programmatically, offering a higher degree of customization and reuse. [**Kadet**](https://github.com/kapicorp/kapitan-reference) empowers users to craft intricate configurations with the simplicity and flexibility of Python. ## [**References**](/references) diff --git a/docs/pages/inventory/backends.md b/docs/pages/inventory/backends.md new file mode 100644 index 000000000..b2e1c5049 --- /dev/null +++ b/docs/pages/inventory/backends.md @@ -0,0 +1,12 @@ +# :kapitan-logo: **Alternative Inventory Backends** + +We provide pluggable backend alternatives for the Inventory: + +* `reclass`(**default**): The original kapitan inventory (see [reclass](https://github.com/kapicorp/reclass)) +* `reclass-rs`: The Rust drop in replacement (see [reclass-rs](reclass-rs.md)) +* `omegaconf`: An alternative inventory solution based on [Omegaconf](https://github.com/omry/omegaconf) + +You can switch the backend to by: + +* (**recommended**) Define the backend in the [.kapitan config file](../commands/kapitan_dotfile.md). +* Passing the `--inventory-backend=backend` command line option when calling `kapitan`. diff --git a/docs/pages/inventory/introduction.md b/docs/pages/inventory/introduction.md index 5e0cc318f..6e13c2ae8 100644 --- a/docs/pages/inventory/introduction.md +++ b/docs/pages/inventory/introduction.md @@ -1,43 +1,73 @@ -## Overview - - +# :kapitan-logo: **What is the inventory?** The **Inventory** is a core component of Kapitan: this section aims to explain how it works and how to best take advantage of it. The **Inventory** is a hierarchical `YAML` based structure which you use to capture anything that you want to make available to **Kapitan**, so that it can be passed on to its templating engines. -The first concept to learn about the **Inventory** is the [**target**](#targets). A target is a file, found under the [`inventory/targets`](#targets) substructure, that tells Kapitan what you want to compile. It will usually map to something you want to do with **Kapitan**. +``` +inventory/ +├── classes/ +│ ├── common/ +│ │ └── base.yml # Foundational configurations +│ ├── components/ +│ │ └── web/ # Component-specific settings +│ └── environments/ +│ └── production.yml # Environment-specific configurations +└── targets/ + └── production/ + └── web.yml # Specific deployment configurations +``` -For instance, you might want to define a [**target**](#targets) for each environment that you want to deploy using **Kapitan**. +The **Kapitan** inventory is divided between [**targets**](#targets) and [**classes**](#classes). -The **Inventory** lets you also define and reuse common configurations through YAML files that are referred to as [**classes**](#classes): by listing classes into [**target**](#targets), their content gets merged together and allows you to compose complex configurations without repetitions. +Both classes and targets are yaml file with the same structure -By combining [**target**](#targets) and [**classes**](#classes), the **Inventory** becomes the SSOT for your whole configuration, and learning how to use it will unleash the real power of **Kapitan**. +```yaml +classes: + - common + - my.other.class -!!! info - The **Kapitan** **Inventory** is based on an open source project called [reclass](https://github.com/kapicorp/reclass) and you can find the full documentation on our Github clone. However we discourage you to look directly at the reclass documentation before you learn more about **Kapitan**, because **Kapitan** uses a fork of reclass and greatly simplifies the reclass experience. +parameters: + key: value + key2: + subkey: value +``` -!!! info - Kapitan allows users to switch the inventory backend to [reclass-rs](https://github.com/projectsyn/reclass-rs). You can switch the backend to reclass-rs by passing `--inventory-backend=reclass-rs` on the command line. Alternatively, you can define the backend in the [.kapitan config file](../commands/kapitan_dotfile.md). +## [**Classes**](#classes) - See the [reclass-rs inventory backend](reclass-rs.md) documentation for more details. +[**Classes**](#classes) are found by default under the [`inventory/classes`](#classes) directory and define common settings and data that you define once and can be included in other files. This promotes consistency and reduces duplication. -!!! note - Kapitan enforces very little structure for the **Inventory**, so that you can adapt it to your specific needs: this might be overwhelming at the beginning: don’t worry, we will explain best practice and give guidelines soon. +Classes are identified with a name that maps to the directory structure they are nested under. +In this example, the `kapicorp.common` class represented by the file `classes/kapicorp/common.yml` -By default, Kapitan will search for its **Inventory** under [`inventory/classes`](#classes) and [`inventory/targets`](#targets). +``` +# classes/kapicorp/common.yml +classes: + - common +parameters: + namespace: ${target_name} + base_docker_repository: quay.io/kapicorp + base_domain: kapicorp.com ``` -inventory/ -├── classes -│ ├── applications -│ ├── components -│ ├── features -│ ├── kapitan -│ ├── projects -│ └── terraform -└── targets - ├── examples - ├── kapicorp - └── terraform + +## [**Targets**](#targets) + +[**Targets**](#targets) are found by default under the [`inventory/targets`](#targets) directory and represent the different environments or components you want to manage. Each target is a YAML file that defines a set of configurations. + +For example, you might have targets for **`production`**, **`staging`**, and **`development`** environments. + +``` +# targets/production.yml +classes: + - kapicorp.common + - components.web + - environments.production + +parameters: + target_name: web ``` + + + +By combining [**target**](#targets) and [**classes**](#classes), the **Inventory** becomes the SSOT for your whole configuration, and learning how to use it will unleash the real power of **Kapitan**. diff --git a/mkdocs.yml b/mkdocs.yml index 04aad8edd..6bf6ab222 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -73,15 +73,17 @@ nav: - Home: README.md - Trusted by: ADOPTERS.md - Getting started: getting_started.md + - Core Concepts: pages/core_concepts.md - Documentation: - - Core Concepts: pages/kapitan_overview.md - Inventory: - Introduction: pages/inventory/introduction.md - Targets: pages/inventory/targets.md - Classes: pages/inventory/classes.md - Parameters Interpolation: pages/inventory/parameters_interpolation.md - Advanced: pages/inventory/advanced.md - - reclass-rs Backend: pages/inventory/reclass-rs.md + - Backends: + - Introduction: pages/inventory/backends.md + - reclass-rs: pages/inventory/reclass-rs.md - Input Types: - Introduction: pages/input_types/introduction.md - Kadet: pages/input_types/kadet.md From 7469be0ce52ca18b242171289be3469678f4cff8 Mon Sep 17 00:00:00 2001 From: Alessandro De Maria Date: Wed, 11 Dec 2024 10:44:12 +0000 Subject: [PATCH 2/2] improve documentation --- docs/pages/inventory/introduction.md | 4 - docs/pages/inventory/targets.md | 142 ++++++++++----------------- 2 files changed, 52 insertions(+), 94 deletions(-) diff --git a/docs/pages/inventory/introduction.md b/docs/pages/inventory/introduction.md index 6e13c2ae8..8222417fa 100644 --- a/docs/pages/inventory/introduction.md +++ b/docs/pages/inventory/introduction.md @@ -33,8 +33,6 @@ parameters: subkey: value ``` -## [**Classes**](#classes) - [**Classes**](#classes) are found by default under the [`inventory/classes`](#classes) directory and define common settings and data that you define once and can be included in other files. This promotes consistency and reduces duplication. Classes are identified with a name that maps to the directory structure they are nested under. @@ -51,8 +49,6 @@ parameters: base_domain: kapicorp.com ``` -## [**Targets**](#targets) - [**Targets**](#targets) are found by default under the [`inventory/targets`](#targets) directory and represent the different environments or components you want to manage. Each target is a YAML file that defines a set of configurations. For example, you might have targets for **`production`**, **`staging`**, and **`development`** environments. diff --git a/docs/pages/inventory/targets.md b/docs/pages/inventory/targets.md index 439bcb000..41c99ca47 100644 --- a/docs/pages/inventory/targets.md +++ b/docs/pages/inventory/targets.md @@ -1,111 +1,73 @@ -# Targets +# :kapitan-logo: **Targets** -## Usage +In the world of Kapitan, a target represents a specific environment or deployment scenario where you want to apply your configurations. -A target is a file that lives under the [`inventory/targets`](#targets) subdirectory, and that tells **Kapitan** what you want it to do for you. +Think of it as a blueprint that defines all the necessary settings and parameters for a particular deployment. - **Kapitan** will recognise all YAML files in the [`inventory/targets`](#targets) subtree as targets. +For instance, you might have separate targets for **`production`**, **`staging`**, and **`development`** environments, each with its own unique configurations. -!!! note - Only use **`.yml`** as extension for **Inventory** files. `.yaml` will not be recognised as a valid **Inventory** file. +## Defining targets +Targets are defined as YAML files within the `inventory/targets/` directory of your Kapitan project. Each target file typically includes: -What you do with a [**target**](#targets) is largely up to you and your setup. Common examples: +``` +classes: + # A list of classes to inherit configurations from. + # This allows you to reuse common settings and avoid repetition + - -* **clusters**: Map each [**target**](#targets) to a cluster, capturing all configurations needed for a given cluster. For instance: `targets/clusters/production-cluster1.yml` -* **applications**: When using **Kapitan** to manage **Kubernetes** applications, you might define a [**target**](#targets) for everything that you would normally deploy in a single namespace, including all its resources, scripts, secrets and documentation. For instance: `targets/mysql.yml` -* **environments**: You might have want to define a different [**target**](#targets) for each environment you have, like `dev.yml`, `test.yml` and `prod.yml` -* **cloud projects**: When working with **Terraform**, it may be convenient to group [**target**](#targets) by cloud project. For instance: `targets/gcp/projects/engineering-prod.yml`. -* **single tenancy**: When deploying a single-tenancy application, you might combine the approaches above, and have a [**target**](#targets) `acme.yml` that is used to define both **Terraform** and **Kubernetes** resources for a given tenant, perhaps also with some **ArgoCD** or **Spinnaker** pipelines to go with it. +parameters: + # file parameters that override or extend the parameters inherited from previously loaded classes +``` +Example: -!!! example +```yaml +# inventory/targets/production/web.yml +classes: + - common + - components.nginx - If you have configured your kapitan repository like in [Quick Start](/kapitan_overview/#setup-your-repository) instructions, you can run the commands we give during the course of this documentation. +parameters: + environment: production + description: ${environment} environment + nginx: + replicas: 3 + namespace: ${environment} +``` - !!! quote "" +In this example, the `production.web` target: - `kapitan compile` +* Inherits configurations from the common and components.nginx classes. +* Sets the `environment` parameter to **`production`**. +* Overrides (if defined) the `replicas` for the `nginx` component to `3`. +* Defines the namespace as **`production`** using variable interpolation. +* Creates a dynamic description based on the content of the environment variable. - ```shell - Compiled gke-pvm-killer (0.09s) - Compiled vault (0.18s) - Compiled pritunl (0.17s) - Compiled mysql (0.07s) - Compiled examples (0.25s) - Compiled postgres-proxy (0.06s) - Compiled echo-server (0.08s) - Compiled global (0.05s) - Compiled tutorial (0.09s) - Compiled guestbook-argocd (0.08s) - Compiled sock-shop (0.30s) - Compiled kapicorp-demo-march (0.04s) - Compiled kapicorp-project-123 (0.03s) - Compiled kapicorp-terraform-admin (0.08s) - Compiled tesoro (0.09s) - Compiled prod-sockshop (0.34s) - Compiled dev-sockshop (0.41s) - Compiled argocd (2.53s) - ``` +## Compiling targets +When you run kapitan compile -t , Kapitan: - When you run `kapitan compile`, you instruct **Kapitan** to generate for each given [**target**](#targets) a directory under `compiled` with the same name. Under this directory you will find all the files that have been generated by **Kapitan** for that target. +* Reads the target file: Kapitan parses the YAML file for the specified target. +* Merges configurations: It merges the parameters from the included classes with the target-specific parameters, giving priority to the target's values. +* Generates output in `compiled/target/path/targetname`: It uses this merged configuration data, along with the input types and generators, to create the final configuration files for the target environment. - !!! quote "" +When you run `kapitan` without the selector, it will run compile for all targets it discovers under the `inventory/targets` subdirectory. - `tree compiled/mysql/` +## Target directory structure +Targets are not limited to living directly within the `inventory/targets` directory. - ```shell - compiled/mysql/ - ├── argocd - ├── docs - │ ├── mysql-readme.md - │ └── README.md - ├── manifests - │ ├── mysql-bundle.yml - │ ├── mysql-config.yml - │ ├── mysql-namespace.yml - │ └── mysql-secret.yml - ├── pre-deploy - ├── rabbitmq - ├── scripts - └── terraform +They can be organized into subdirectories to create a more structured and hierarchical inventory. This is particularly useful for managing large and complex projects. - 7 directories, 6 files - ``` +When targets are organized in subdirectories, Kapitan uses the full path from the `targets/` directory to create the target name. This name is then used to identify the target during compilation and in the generated output. -## Definition +Example for target `clusters.production.my-cluster` -A typical [**target**](#targets) might look like this: +```shell +inventory/ +└── targets/ + └── clusters/ + └── production/ + └── my-cluster.yml +``` -!!! example "`inventory/targets/acme/dev.yaml`" - - ```yaml - classes: - - common - - components.acme.frontend - - components.acme.backend - - parameters: - target_name: dev - ``` - -Note that it is made of 2 sections: - -* `classes` is a list of class files you will want to import. -* `parameters` allows for local override of what is unique to this target. - -!!! info - - the `kapitan` key under the root `parameters` is reserved for kapitan usage. Some examples: - - ```yaml - parameters: - kapitan: - compile: # input types configuration section - dependencies: # dependencies configuration section to download resources - secrets: # secret encryption/decryption configuration section - validate: # items which indicate which compiled output to validate - vars: # which are also passed down to input types as context - ``` - -*[SSOT]: Single Source Of Truth +In this example, the `my-cluster.yml` target file is located within the `clusters/production/` subdirectory, and can be identified with `clusters.production.my-cluster`.