Standalone Egress NAT

Signed-off-by: Patryk Strusiewicz-Surmacki <[email protected]> Co-authored-by: Tomoya Terashima <[email protected]>
cybozu-go · Jan 22, 2025 · edae8ae · edae8ae
1 parent 200731d
commit edae8ae
Show file tree

Hide file tree

Showing 82 changed files with 2,505 additions and 789 deletions.
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
@@ -13,12 +13,16 @@ env:
 jobs:
   test:
     name: Small test
+    strategy:
+      matrix:
+        test-ipam: ["true", "false"]
     runs-on: ubuntu-24.04
     steps:
     - uses: actions/checkout@v4
     - uses: actions/setup-go@v5
       with:
         go-version: ${{ env.go-version }}
+        cache-dependency-path: "**/go.sum"
     - name: Cache tools
       id: cache-tools
       uses: actions/cache@v4
@@ -29,7 +33,7 @@ jobs:
         key: cache-${{ env.cache-version }}-go-${{ env.go-version }}-${{ hashFiles('v2/Makefile') }}
     - run: make setup
       if: steps.cache-tools.outputs.cache-hit != 'true'
-    - run: make test
+    - run: make test TEST_IPAM=${{ matrix.test-ipam }} TEST_EGRESS=true
     - run: make test-nodenet
       timeout-minutes: 10
     - run: make test-founat
@@ -40,35 +44,48 @@ jobs:
     strategy:
       matrix:
         kindest-node: ["1.28.15", "1.29.12", "1.30.8"]
-        ip-version: ["ipv4", "ipv6"]
+        with-ipam: ["false", "true"]
+        ipv6: ["false", "true"]
     runs-on: ubuntu-24.04
     steps:
     - uses: actions/checkout@v4
     - uses: actions/setup-go@v5
       with:
         go-version: ${{ env.go-version }}
+        cache-dependency-path: "**/go.sum"
     - run: make image
     - run: make certs
     - name: Enable docker IPv6 mode
-      if: matrix.ip-version == 'ipv6'
+      if: matrix.ipv6 == 'true'
       working-directory: v2/e2e
       run: |
         sudo mkdir -p /etc/docker
         sudo cp daemon.json /etc/docker/daemon.json
         sudo systemctl restart docker.service
         sleep 10
         echo TEST_IPV6=true >> $GITHUB_ENV
-    - run: make start KUBERNETES_VERSION=${{ matrix.kindest-node }}
+    - run: make start KUBERNETES_VERSION=${{ matrix.kindest-node }} WITH_KINDNET=false TEST_IPV6=${{ matrix.ipv6 }}
+      if: matrix.with-ipam == 'true'
+      working-directory: v2/e2e
+    - run: make start KUBERNETES_VERSION=${{ matrix.kindest-node }} WITH_KINDNET=true TEST_IPV6=${{ matrix.ipv6 }}
+      if: matrix.with-ipam == 'false'
       working-directory: v2/e2e
     - run: make install-coil
+      if: matrix.with-ipam == 'true'
+      working-directory: v2/e2e
+    - run: make install-coil-egress-v4
+      if: matrix.with-ipam == 'false' && matrix.ipv6 == 'false'
+      working-directory: v2/e2e
+    - run: make install-coil-egress-v6
+      if: matrix.with-ipam == 'false' && matrix.ipv6 == 'true'
       working-directory: v2/e2e
-    - run: make test
+    - run: make test TEST_IPAM=${{ matrix.with-ipam }} TEST_EGRESS=true TEST_IPV6=${{ matrix.ipv6 }}
       working-directory: v2/e2e
     - run: make logs
       working-directory: v2/e2e
       if: always()
     - uses: actions/upload-artifact@v4
       if: always()
       with:
-        name: logs-${{ matrix.ip-version }}-${{ matrix.kindest-node }}.tar.gz
+        name: logs-ipv6-${{ matrix.ipv6 }}-with-ipam-${{ matrix.with-ipam }}-${{ matrix.kindest-node }}.tar.gz
         path: v2/e2e/logs.tar.gz
diff --git a/docs/cmd-coil-egress-controller.md b/docs/cmd-coil-egress-controller.md
@@ -0,0 +1,27 @@
+coil-egress-controller
+===============
+
+`coil-egress-controller` is a Kubernetes controller for Coil custom resources related to on-demand NAT egress.
+It is intended to be run as a Pod in `kube-system` namespace.
+
+
+## Egress
+
+`coil-egress-controller` creates **Deployment** and **Service** for each Egress.
+
+It also creates `coil-egress` **ServiceAccount** in the namespace of Egress,
+and binds it to the **ClusterRoles** for `coil-egress`.
+
+## Command-line flags
+
+```
+Flags:
+      --cert-dir string        directory to locate TLS certs for webhook (default "/certs")
+      --egress-port int32      UDP port number used by coil-egress (default 5555)
+      --gc-interval duration   garbage collection interval (default 1h0m0s)
+      --health-addr string     bind address of health/readiness probes (default ":9387")
+  -h, --help                   help for coil-egress-controller
+      --metrics-addr string    bind address of metrics endpoint (default ":9386")
+  -v, --version                version for coil-egress-controller
+      --webhook-addr string    bind address of admission webhook (default ":9443")
+```
diff --git a/docs/cmd-coil-controller.md → docs/cmd-coil-ipam-controller.md b/docs/cmd-coil-controller.md → docs/cmd-coil-ipam-controller.md
@@ -1,41 +1,33 @@
-coil-controller
+coil-ipam-controller
 ===============
 
-`coil-controller` is a Kubernetes controller for Coil custom resources.
+`coil-ipam-controller` is a Kubernetes controller for Coil IPAM related custom resources.
 It is intended to be run as a Pod in `kube-system` namespace.
 
 ## AddressPool and AddressBlock
 
-`coil-controller` has an in-memory database of address pools and
+`coil-ipam-controller` has an in-memory database of address pools and
 address blocks to allocate address blocks quickly.
 
 ## BlockRequest
 
-`coil-controller` watches newly created block requests and carve out
+`coil-ipam-controller` watches newly created block requests and carve out
 address blocks from the requested pool.
 
-## Egress
-
-`coil-controller` creates **Deployment** and **Service** for each Egress.
-
-It also creates `coil-egress` **ServiceAccount** in the namespace of Egress,
-and binds it to the **ClusterRoles** for `coil-egress`.
-
 ## Garbage collection
 
-`coil-controller` periodically checks orphaned address blocks and deletes them.
+`coil-ipam-controller` periodically checks orphaned address blocks and deletes them.
 
 ## Command-line flags
 
 ```
 Flags:
       --cert-dir string        directory to locate TLS certs for webhook (default "/certs")
-      --egress-port int32      UDP port number used by coil-egress (default 5555)
       --gc-interval duration   garbage collection interval (default 1h0m0s)
       --health-addr string     bind address of health/readiness probes (default ":9387")
-  -h, --help                   help for coil-controller
+  -h, --help                   help for coil-ipam-controller
       --metrics-addr string    bind address of metrics endpoint (default ":9386")
-  -v, --version                version for coil-controller
+  -v, --version                version for coil-ipam-controller
       --webhook-addr string    bind address of admission webhook (default ":9443")
 ```
 

diff --git a/docs/cni-grpc.md b/docs/cni-grpc.md
@@ -7,6 +7,7 @@
     - [AddResponse](#pkg-cnirpc-AddResponse)
     - [CNIArgs](#pkg-cnirpc-CNIArgs)
     - [CNIArgs.ArgsEntry](#pkg-cnirpc-CNIArgs-ArgsEntry)
+    - [CNIArgs.InterfacesEntry](#pkg-cnirpc-CNIArgs-InterfacesEntry)
     - [CNIError](#pkg-cnirpc-CNIError)
 
     - [ErrorCode](#pkg-cnirpc-ErrorCode)
@@ -57,6 +58,8 @@ https://pkg.go.dev/github.com/containernetworking/[email protected]/pkg/skel?tab=doc#Cm
 | args | [CNIArgs.ArgsEntry](#pkg-cnirpc-CNIArgs-ArgsEntry) | repeated | Key-Value pairs parsed from CNI_ARGS |
 | path | [string](#string) |  |  |
 | stdin_data | [bytes](#bytes) |  |  |
+| ips | [string](#string) | repeated |  |
+| interfaces | [CNIArgs.InterfacesEntry](#pkg-cnirpc-CNIArgs-InterfacesEntry) | repeated |  |
 
 
 
@@ -79,6 +82,22 @@ https://pkg.go.dev/github.com/containernetworking/[email protected]/pkg/skel?tab=doc#Cm
 
 
 
+<a name="pkg-cnirpc-CNIArgs-InterfacesEntry"></a>
+
+### CNIArgs.InterfacesEntry
+
+
+
+| Field | Type | Label | Description |
+| ----- | ---- | ----- | ----------- |
+| key | [string](#string) |  |  |
+| value | [bool](#bool) |  |  |
+
+
+
+
+
+
 <a name="pkg-cnirpc-CNIError"></a>
 
 ### CNIError

diff --git a/docs/design.md b/docs/design.md
@@ -73,7 +73,8 @@ Now that we have learned how to do these things, and want to add rich features s
 
 Coil v2 will consist of the following programs:
 
-- `coil-controller`: Kubernetes controller managing custom resources.
+- `coil-ipam-controller`: Kubernetes controller managing IPAM related custom resources.
+- `coil-egress-controller`: Kubernetes controller managing on-demand NAT egress related custom resources.
 - `coild`: Daemon program running on nodes.
 - `coil`: CNI interface that delegates requests from `kubelet` to `coild`.
 - `coil-egress`: Administration program running in Egress pods.
@@ -102,11 +103,11 @@ To make things simple, the default pool is the pool whose name is `default`.
 To reduce the number of advertised routes, addresses in an address pool are divided into fixed-size blocks.
 These blocks are called _address blocks_, and assigned to nodes.  Since all IP addresses in an address block are routed to the same node, only one route per address block need to be advertised.
 
-For example, if an address pool defines that the size of an address block is 2<sup>5</sup>, `coil-controller` will carve an address block for IPv4 with `/27` subnet mask out of the pool, and assigns it to a node.
+For example, if an address pool defines that the size of an address block is 2<sup>5</sup>, `coil-ipam-controller` will carve an address block for IPv4 with `/27` subnet mask out of the pool, and assigns it to a node.
 
 
 In general, avoiding immediate reuse of IP addresses is better not to confuse other software or components.
-To avoid such immediate reuse, `coil-controller` remembers the last used address, and it assigns the address from the next address.
+To avoid such immediate reuse, `coil-ipam-controller` remembers the last used address, and it assigns the address from the next address.
 
 The same problem may occur when we use address blocks of the size `/32`.
 In this case, there is a high chance of reusing the same address immediately.
@@ -333,7 +334,7 @@ Therefore, when the deletion of the owning `AddressPool` is directed, all `Addre
 That said, an `AddressBlock` should not be deleted until there are no more Pods with an address in the block.
 For this purpose, Coil adds a finalizer to each `AddressBlock`.  **`coild` checks the usage of addresses in the block**, and once there are no more Pods using the addresses, it removes the finalizer to delete the `AddressBlock`.
 
-`AddressBlock` should also be deleted when `Node` that acquired the block is deleted.  Since `coild` running as a DaemonSet pod cannot do this, **`coil-controller` watches Node deletions and removes `AddressBlocks`**.  `coil-controller` periodically checks dangling `AddressBlocks` and removes them.
+`AddressBlock` should also be deleted when `Node` that acquired the block is deleted.  Since `coild` running as a DaemonSet pod cannot do this, **`coil-ipam-controller` watches Node deletions and removes `AddressBlocks`**.  `coil-ipam-controller` periodically checks dangling `AddressBlocks` and removes them.
 
 `coild` also deletes `AddressBlock` when it frees the last IP address used in the block.  At startup, `coild` also checks each `AddressBlock` for the Node, and if no Pod is using the addresses in the block, it deletes the `AddressBlock`.
 
@@ -342,7 +343,7 @@ Note that Coil does not include `Node` in the list of owner references of an `Ad
 ### AddressPool
 
 Similar to an `AddressBlock` and its addresses, an `AddressPool` should not be deleted until there are no more `AddressBlock`s derived from the pool.
-For this purpose, Coil adds a finalizer to each `AddressPool`.  `coil-controller` checks the usage of blocks in the pool.
+For this purpose, Coil adds a finalizer to each `AddressPool`.  `coil-ipam-controller` checks the usage of blocks in the pool.
 
 Note that `blockOwnerDeletion: true` in `AddressBlock`'s `ownerReferences` does not always block the deletion of the owning `AddressPool`.
 This directive has effect only when foreground cascading deletion is adopted.
@@ -397,7 +398,7 @@ skinparam rectangle {
 together {
 actor User
 package Deployment {
-    [coil-controller] as controller
+    [coil-ipam-controller] as controller
 }
 database kube-apiserver as apiserver #lightblue {
     [AddressPool] as pool1

diff --git a/docs/setup.md b/docs/setup.md
@@ -17,6 +17,8 @@ You can tweak optional parameters by editing [`kustomization.yaml`](../v2/kustom
   - [IPv4/v6 dual stack pool](#ipv4v6-dual-stack-pool)
 - [(Option) Configure BIRD](#option-configure-bird)
 - [Note on CRI runtime compatibility](#note-on-cri-runtime-compatibility)
+- [Standalone egress](#standalone-egress)
+  - [Configuration](#configuration)
 
 ## Install `kustomize`
 
@@ -37,7 +39,8 @@ This should generate the following PEM files:
 
 ```console
 $ ls config/default/*.pem
-config/default/cert.pem  config/default/key.pem
+config/default/cert.pem         config/default/egress-key.pem  config/default/ipam-key.pem
+config/default/egress-cert.pem  config/default/ipam-cert.pem   config/default/key.pem
 ```
 
 ## Edit `kustomization.yaml`
@@ -57,7 +60,7 @@ $ vi kustomization.yaml
 (actually, the example is a network configuration list).
 
 You may edit the file to, say, add Cilium for network policies or to tune MTU.
-Note that `coil` must be the first in the plugin list.
+Note that `coil` must be the first in the plugin list if IPAM is enabled.
 
 ```console
 vi netconf.json
@@ -76,7 +79,7 @@ The following example adds `tuning` and `bandwidth` plugins.
   "plugins": [
     {
       "type": "coil",
-      "socket": "/run/coild.sock"
+      "socket": "/run/coild.sock",
     },
     {
       "type": "tuning",
@@ -231,3 +234,86 @@ host directory.
 
 [netconf]: https://github.com/containernetworking/cni/blob/spec-v0.4.0/SPEC.md#network-configuration
 [BIRD]: https://bird.network.cz/
+
+## Standalone egress
+
+Coil can be run as standalone egress NAT controller, using CNI chaining with another CNI providing base connectivity. This chapter will guide you on how to achieve this.
+
+### Configuration
+
+To deploy Coil with only egress feature enabled the following changes are required in the configuration files:
+
+1. Comment all IPAM related pieces in the following `kustomization.yaml` files:
+    - `v2/config/crd/kustomization.yaml`
+    - `v2/config/default/kustomization.yaml`
+    - `v2/config/pod/kustomization.yaml`
+    - `v2/config/rbac/kustomization.yaml`
+
+1. Comment unnecessary resources in `config/crd/patches/remove_status.yaml`.
+1. Add following arguments to the `coild` contianer executable in `config/pod/coild.yaml`
+    ```yaml
+    containers:
+      - name: coild
+        image: coil:dev
+        command: ["coild"]
+        args:
+          - --zap-stacktrace-level=panic
+          - --enable-ipam=false
+          - --enable-egress=true
+          - --pod-table-id=0 # 255 if IPv6 is being used
+          - --protocol-id=2
+    ```
+1. Set CNI config filename using environment variable for init contianer `coil-installer` in `config/pod/coild.yaml`:
+    ```yaml
+    env:
+    - name: CNI_CONF_NAME
+      value: "01-coil.conflist"
+    ```
+1. Add configuration of your chosen CNI to `v2/netconf.json` before `coil` related configuration.
+1. Deploy `coil` to existing cluster as described in [Compile and apply the manifest](#compile-and-apply-the-manifest).
+
+### Testing standalone egress
+
+#### Testing with Kindnet using IPv4
+1. Generate certificates using `v2/Makefile`.
+    ```bash
+    cd v2 && make certs
+    ```
+1. Go to `v2/e2e`
+    ```bash
+    cd e2e
+    ```
+1. Create IPv4 based Kind cluster with Kindnet CNI deployed:
+    ```bash
+    WITH_KINDNET=true TEST_IPV6=false make start
+    ```
+1. Install Coil on the cluster:
+    ```bash
+    make install-coil-egress-v4
+    ```
+1. Run egress-only IPv4 tests:
+    ```bash
+    TEST_IPAM=false TEST_EGRESS=true TEST_IPV6=false make test
+    ```
+
+#### Testing with Kindnet using IPv6
+1. Generate certificates using `v2/Makefile`.
+    ```bash
+    cd v2 && make certs
+    ```
+1. Go to `v2/e2e`
+    ```bash
+    cd e2e
+    ```
+1. Create IPv6 based Kind cluster with Kindnet CNI deployed:
+    ```bash
+    WITH_KINDNET=true TEST_IPV6=true make start
+    ```
+1. Install Coil on the cluster:
+    ```bash
+    make install-coil-egress-v6
+    ```
+1. Run egress-only IPv6 tests:
+    ```bash
+    TEST_IPAM=false TEST_EGRESS=true TEST_IPV6=true make test
+    ```
diff --git a/v2/.gitignore b/v2/.gitignore
@@ -1,6 +1,9 @@
 /bin
-/config/default/webhook_manifests_patch.yaml
+/config/default/ipam/webhook_manifests_patch.yaml
+/config/default/egress/webhook_manifests_patch.yaml
 /include
 /testbin
 /tmp
 /work
+/e2e/tmp
+/e2e/kindnet-conf