Big Bang Component Extension (#1094)

Introduces the first Zarf Extension: `bigbang`. This PR establishes the
pattern for create-time extensions and provides a simple/reliable way to
deploy [Big Bang](https://p1.dso.mil/products/big-bang) using Zarf. In
addition to collecting all the necessary assets, including Flux, the
extension adds [component
actions](https://docs.zarf.dev/docs/user-guide/component-actions) to
verify the proper deployment of Big Bang Packages. In the event of a
failure during deployment, details support data is also provided to the
user to help expedite troubleshooting.

## Example usage:
```yaml
components:
  - name: bigbang
    required: true
    extensions:
      bigbang:
        version: 1.53.0
        valuesFiles:
          # Istio configs
          - config/ingress.yaml
          # Use Kyverno instead of Gatekeeper
          - config/kyverno.yaml
```

Conversation around #1093

The following things should be completed in future issues
- [ ] Ensure Cleanup works consistently and smoothly when un-installing
- [ ] Add labels to services for `zarf connect`
- [ ] Perform upgrade test after Zarf CRD upgrade support + Big Bang
1.55 are released.

---------

Signed-off-by: Tom Runyon <[email protected]>
Co-authored-by: corang <[email protected]>
Co-authored-by: Megamind <[email protected]>
Co-authored-by: Andy Roth <[email protected]>
Co-authored-by: Jeff McCoy <[email protected]>
Co-authored-by: Wayne Starr <[email protected]>
Co-authored-by: razzle <[email protected]>
Co-authored-by: Wayne Starr <[email protected]>
Signed-off-by: Wayne Starr <[email protected]>

.github/workflows/test-bigbang.yml

@@ -0,0 +1,52 @@
+name: Test Big Bang extension
+on:
+  pull_request:
+    paths-ignore:
+      - "**.md"
+      - "**.jpg"
+      - "**.png"
+      - "**.gif"
+      - "**.svg"
+      - "adr/**"
+      - "docs/**"
+      - "package.json"
+      - "package-lock.json"
+      - "CODEOWNERS"
+
+# Abort prior jobs in the same workflow / PR
+concurrency:
+  group: e2e-bb-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  validate:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Setup golang
+        uses: ./.github/actions/golang
+
+      - name: Setup NodeJS
+        uses: ./.github/actions/node
+
+      - name: Build binary and zarf packages
+        uses: ./.github/actions/packages
+
+      - name: Setup K3d
+        uses: ./.github/actions/k3d
+
+      - name: "Login to Iron Bank"
+        uses: docker/login-action@v2
+        with:
+          registry: registry1.dso.mil
+          username: ${{ secrets.IRONBANK_USERNAME }}
+          password: ${{ secrets.IRONBANK_PASSWORD }}
+
+      - name: Run tests
+        run: go test ./src/extensions/bigbang/test -failfast -v -timeout 30m
+
+      - name: Save logs
+        if: always()
+        uses: ./.github/actions/save-logs

.gitignore

@@ -35,3 +35,4 @@ zarf-pki
 zarf-sbom/
 *.part*
 test-*.txt
+__debug_bin

.grype.yaml

@@ -1,10 +1,6 @@
 # Ignore file for false positives from protobuf, see the following for more information:
 #   https://github.com/anchore/grype/issues/558
 ignore:
-  # False positives from CPE confusion of Go vs C (https://github.com/anchore/grype/issues/558)
-  - vulnerability: CVE-2021-22570
-  - vulnerability: CVE-2015-5237
-
   # False positive from CPE confusion of NPM vs Rust Crates (https://github.com/anchore/grype/issues/931)
   - vulnerability: CVE-2017-18589
 
@@ -14,8 +10,8 @@ ignore:
   # False positive from CPE confusion of svelte and svelte's extension
   - vulnerability: CVE-2021-29261
 
-  # From yargs - only used through quicktype as a dev dependency
-  - vulnerability: GHSA-p9pc-299p-vxgp
-
-  # Grype being stupid again, https://nvd.nist.gov/vuln/detail/CVE-2018-25076 is a php vulnerability 🤦
+  # False positive from CPE confusion of a php vulnerability , https://nvd.nist.gov/vuln/detail/CVE-2018-25076
   - vulnerability: CVE-2018-25076
+
+  # From rouille - The Zarf injector does not expose endpoints that use multipart form data
+  - vulnerability: GHSA-mc8h-8q98-g5hr

adr/0013-bigbang-as-a-noun.md

@@ -0,0 +1,42 @@
+# 13. Big Bang as a Noun
+
+Date: 2023-01-18
+
+## Status
+
+Accepted
+
+## Context
+
+One primary application component that end users of Zarf are deploying is [Big Bang](https://repo1.dso.mil/big-bang/bigbang). The installation of Big Bang is complicated for several reasons:
+
+- It requires Flux to be installed to deploy correctly due to the use of Flux CRDs.
+- The [images](https://umbrella-bigbang-releases.s3-us-gov-west-1.amazonaws.com/umbrella/1.51.0/package-images.yaml) defined within Big Bang are normally a superset of the images needed for any individual deployment.
+- All images that Big Bang might need takes 10s of gigabytes of storage to include in a Zarf package.
+- The git repositories defined within Big Bang are normally a superset of the git repositories needed for any individual deployment.
+- Injecting a `values.yaml` file into the [default deployment structure](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/base/kustomization.yaml) is complicated and the discovery of which images are needed is a function of the values that are provided to the Big Bang chart
+
+## Decision
+
+Deployments of Big Bang can be managed with a new `bigbang` noun in the zarf.yaml that manages the complexity of the deployment. This capability will take the values provided to the big bang chart, template them during the package phase to identify which [Big Bang packages](https://repo1.dso.mil/big-bang/bigbang/-/blob/master/docs/packages.md) are being configured in the Zarf package. The code then includes only the git repositories and images needed for the configured packages, and does not include the git repositories and images for packages that would not be deployed.
+
+The `bigbang` section will provide the following configurations for managing a big bang deployment:
+
+- `version` - Identifies the particular version of Big Bang to deploy, which corresponds to git tags in the provided `repo`. See versions of Big Bang [here](https://repo1.dso.mil/big-bang/bigbang/-/releases).
+- `repo` - Identifies the git repository Big Bang is hosted on. Defaults to https://repo1.dso.mil/big-bang/bigbang.git
+- `valuesFiles` - list of local files that get passed to the Big Bang helm chart for deployment.
+- `skipFlux` - boolean to determine if the flux installation for Big Bang should be skipped. Only set this to true if flux has been deployed in a different way already in the cluster.
+
+## Implementation
+
+The Big Bang component in the original zarf.yaml will update the component to install Flux (assuming the `skipFlux` flag is not set to true) and Big Bang. The flux deployment is just a remote Kustomization pointing at the [corresponding version of flux](https://repo1.dso.mil/big-bang/bigbang/-/tree/master/base/flux), with Big Bang resulting in the following objects being defined as manifests:
+
+1. The `bigbang` `Namespace`
+2. One `Secret` for each file provided in `valuesFiles`
+3. `GitRepository` object pointing at the provided `repo`.
+4. `HelmRelease` object deploying the chart from the `GitRepository` configured with the secrets as values files.
+
+## Consequences
+
+- By doing package time rendering and discovery of images for inclusion into the zarf package, the flexibility for deploy time configuration is limited since new parts of Big Bang can't be added arbitrarily, since the necessary artifacts to deploy those Big Bang packages won't be present in the zarf package
+- Big Bang is ever changing and improving, and while it is available as open source, we do not control the change in how the deployment is handled as [Big Bang 2.0](https://repo1.dso.mil/groups/big-bang/-/epics/217) is progressing. This creates a burden on the Zarf team to ensure new changes in Big Bang do not break how Big Bang is deployed, and a burden to ensure as the way Big Bang gets deployed is changed, it does not break older versions of deploying Big Bang.

docs/13-walkthroughs/5-big-bang.md

@@ -0,0 +1,197 @@
+# Using Big Bang with Zarf
+
+## Introduction
+
+This walkthrough describes how to use Big Bang with Zarf for Air Gap deployments through the use of the Big Bang Zarf extension. If you are not familiar with Big Bang you can learn more about it here: https://p1.dso.mil/products/big-bang, but in short it is a DevSecOps platform that contains many useful tools for building, managing, and running software projects while adhering to the [United States Department of Defense DevSecOps Reference Design](https://public.cyber.mil/devsecops/).
+
+Zarf integrates with Big Bang through the use of an extension that simplifies the selection of Big Bang packages and the pulling of the required artifacts to deploy those packages in an Air Gap.
+
+### Known Issues
+
+The current version of this extension works best with Big Bang version `1.54.0` or later, and is not fully integrated into the `zarf package remove` lifecycle.  We will be looking to remove these limitations in a future release.
+
+## System Requirements
+
+Big Bang requires a reasonably powerful `amd64` system that scales up with the number of components deployed.  We recommend at least `32 GB` of RAM and a high-speed internet connection to complete this walkthrough.
+
+To learn more about Big Bang's requirements in general, see their documentation: https://docs-bigbang.dso.mil/latest/docs/prerequisites/minimum-hardware-requirements/
+
+## Pre-requisites
+
+Before beginning this walkthrough you will need the following:
+
+- A local copy of the Zarf repository
+  - `git clone https://github.com/defenseunicorns/zarf.git`
+- A kubernetes cluster onto which you can deploy Zarf and Big Bang
+  - Follow steps 3 through 5 of the [Big Bang Quick Start](https://docs-bigbang.dso.mil/1.54.0/docs/guides/deployment-scenarios/quickstart/) to setup `docker` and a `k3d` cluster.
+- The latest version of the Zarf `cli`
+  - Follow instructions on https://zarf.dev/install/
+- An account on `https://registry1.dso.mil` to retrieve Big Bang images
+  - You can register for an account [here](https://login.dso.mil/auth/realms/baby-yoda/protocol/openid-connect/registrations?client_id=account&response_type=code)
+
+:::note
+
+Big Bang requires some additional [configuration options](https://docs-bigbang.dso.mil/1.54.0/docs/guides/deployment-scenarios/quickstart/#Explanation-of-k3d-Command-Flags-Relevant-to-the-Quick-Start) to be passed to `k3d` than is required in most other Zarf walkthroughs.  Below are some examples:
+
+```bash
+  # Required by the PLG stack
+  --volume /etc/machine-id:/etc/machine-id
+
+  # Required for Istio ingress
+  --k3s-arg "--disable=traefik@server:0"
+  --port 80:80@loadbalancer
+  --port 443:443@loadbalancer
+
+  # Required for TLS to work correctly with kubectl
+  --k3s-arg "--tls-san=$SERVER_IP@server:0"
+  --api-port 6443
+```
+
+If you tweak the packages that are deployed there may be other configuration options you need to specify, please refer to the [Big Bang documentation](https://docs-bigbang.dso.mil/1.54.0/docs/) for more details.
+
+:::
+
+## Package Creation
+
+The below section covers creating and tuning the Big Bang package:
+
+### Setup
+
+By default, Big Bang uses images from [Iron Bank](https://p1.dso.mil/products/iron-bank) which will require you to set your login credentials for [Registry One](https://registry1.dso.mil) (see [pre-requisites](#pre-requisites) for information on account setup).
+
+```bash
+# Authenticate to https://registry1.dso.mil/, then retrieve your CLI secret from your User Profile and run the following:
+set +o history
+export REGISTRY1_USERNAME=<REPLACE_ME>
+export REGISTRY1_CLI_SECRET=<REPLACE_ME>
+echo $REGISTRY1_CLI_SECRET | zarf tools registry login registry1.dso.mil --username $REGISTRY1_USERNAME --password-stdin
+set -o history
+```
+
+Now navigate to the `examples/big-bang` folder within the Zarf repository you cloned in the [pre-requisites](#pre-requisites) section.
+
+### Configure Big Bang
+
+Within the `examples/big-bang` folder you will see a `zarf.yaml` that has the following [component](../4-user-guide/2-zarf-packages/2-zarf-components.md) defined:
+
+```yaml
+components:
+  - name: bigbang
+    required: true
+    extensions:
+      bigbang:
+        version: 1.54.0
+        skipFlux: false
+        valuesFiles:
+          - config/minimal.yaml #turns on just istio
+          - config/ingress.yaml # adds istio certs for *.bigbang.dev
+          - config/kyverno.yaml # turns on kyverno
+          - config/loki.yaml # turns on loki and monitoring
+```
+
+This component uses the `bigbang` extension to define the version of Big Bang to use and the values files to apply.  Feel free to inspect and configure the values.yaml files as you wish and to learn more about Big Bang's configuration see their values guide: https://docs-bigbang.dso.mil/latest/docs/guides/using-bigbang/values-guide/
+
+:::note
+
+The `valuesFiles` are applied from top to bottom and will apply the last value that was provided for any given key.
+
+:::
+
+:::note
+
+This extension works best with Big Bang version `1.54.0` or later.  Version `1.53.0` requires manual patches to images to function correctly.
+
+:::
+
+
+### Package Big Bang
+
+When you're ready to continue you can create a Big Bang package by running the following command in `examples/big-bang`:
+
+```bash
+zarf package create
+```
+
+Now wait for the package creation to complete and you should see a `zarf-package-big-bang-example-amd64-x.x.x.tar.zst` file in the directory.
+
+
+## Package Deployment
+
+The below section covers deploying the Big Bang package from the previous section:
+
+### Initialize Zarf
+
+Before you can deploy the Big Bang package you must first initialize Zarf on the cluster you created in the [pre-requisites](#pre-requisites) section.  To do so you can run the following:
+
+```bash
+# Initialize Zarf (interactively)
+zarf init
+# Make these choices at the prompts
+# ? Do you want to download this init package? Yes
+# ? Deploy this Zarf package? Yes
+# ? Deploy the k3s component? No
+# ? Deploy the logging component? No
+# ? Deploy the git-server component? Yes
+
+# (Optional) Inspect the results
+zarf tools k9s
+```
+
+:::note
+
+The `git-server` component is required by Big Bang as it uses it as a source for Flux deployments.
+
+:::
+
+
+### Deploy Big Bang
+
+Now you are ready to deploy Big Bang, and can do so with the following in the `examples/big-bang` directory:
+
+```bash
+# Deploy Big Bang (interactively)
+zarf package deploy
+# Make these choices at the prompts
+# ? Choose or type the package file [tab for suggestions] zarf-package-big-bang-example-amd64-x.x.x.tar.zst
+# ? Deploy this Zarf package? Yes
+```
+
+### See The Results
+
+Once the install completes you can inspect the results and watch the Big Bang components deploy using the following:
+
+```bash
+zarf tools k9s
+
+# To view different k8s objects you can use the following:
+
+# Helm Releases:
+# :hr [Enter]
+# Pods:
+# :pods [Enter]
+# Services:
+# :svc [Enter]
+# Secrets:
+# :secret [Enter]
+# ConfigMaps:
+# :configmap [Enter]
+
+# When you are done use the following to quit
+# :q [Enter]
+```
+
+## Package Removal
+
+The Big Bang package is not fully integrated into the Zarf package remove lifecycle (see [known issues](#known-issues)), but for the purposes of this walkthrough you can simply remove your k3d cluster:
+
+```bash
+k3d cluster delete
+```
+
+## Troubleshooting
+
+See the Troubleshooting section of the Big Bang Quick Start for help troubleshooting the Big Bang deployment: https://repo1.dso.mil/big-bang/bigbang/-/blob/master/docs/guides/deployment-scenarios/quickstart.md#troubleshooting
+
+Also, ensure that you have followed all of the steps required in the [pre-requisites](#pre-requisites) section and have reviewed the [known issues](#known-issues).
+
+If you feel that the error you are encountering is one with Zarf feel free to [open an issue](https://github.com/defenseunicorns/zarf/issues/new/choose) or reach out via [slack](https://kubernetes.slack.com/archives/C03B6BJAUJ3).