This blueprint will teach you to:
- Deploy and configure a DigitalOcean Droplet to act as an Egress gateway for your DOKS cluster, via Crossplane.
- Deploy the Static Routes Operator, and configure static routes on your DOKS cluster worker nodes to egress workloads traffic to all public IP ranges (some restrictions apply), or to specific destinations only.
What is an Egress gateway and why is it important?
No matter where you have your resources (DOKS, Droplets, etc) deployed and running, they live in a private network, or VPC. The private network can be at home behind your ISP router, a private data center (on-premise), or in a cloud based environment. Main role of a VPC is to isolate different networks across different regions. Different VPCs can talk to each other via gateways (which is a router in essence).
Next, you need to get familiar with some terminology associated with inbound and outbound traffic, explained below:
Ingressdeals withinboundtraffic, entering your VPC.Egressdeals withoutboundtraffic, exiting your VPC.
When using Kubernetes, you manage incoming traffic via an Ingress resource. But, for outgoing traffic there is no Egress resource in the Kubernetes spec. It can be implemented in the CNI layer. For example Cilium, which is used by DOKS, has an Egress Gateway spec. But, in case of DigitalOcean Kubernetes it's not quite stable or production ready yet.
There are three key aspects related to egress functionality:
- Restricting egress traffic (not covered in this guide).
- NAT outgoing.
- Egress gateways.
Restricting egress (or outgoing) traffic is not covered in this blueprint, but in essence is a way of restricting outbound traffic for cluster Pods. It can be achieved via network policies, or firewalls. Firewall is the most common use case, where you allow connections only to a particular external IP address range, or to external services. Firewalls cannot distinguish between individual Pods, so the rules apply equally to all.
To build an egress gateway you need NAT functionality, implemented using a NAT gateway In essence, a NAT gateway sits at the edge of your private network (or VPC), through which outbound (or egress) traffic flows to the public internet. The main role of NAT, which stands for Network Address Translation, is to make network packets routable outside your private network (or VPC). The process needs to work in reverse order as well, meaning incoming response packets, need to be routed back to the originating private IP inside your VPC.
Your VPC uses a specific IP range (e.g. 10.116.0.0/20), which needs to be translated to a public IP address so that packets can flow to a specific destination outside your private network (i.e. Internet). When a response packet comes in, the NAT layer must translate the public IP address from the packet header to the private network address of the host where traffic originated. This is what NAT is for.
A dedicated machine configured for this purpose is called a NAT gateway. When attached to a DOKS cluster, it can route all (or specific destinations only) traffic via a single routable public IP. Hence, we can call it an Egress Gateway in this context.
Moving further with a practical example, suppose that you need to use an external service such as a database. The database is usually located in another data center outside your private network (or VPC). The database administrator configured the database firewall so that only specific public IPs are allowed to connect. You already egress from your DOKS cluster nodes because they are connected directly to the Internet, but it's not practical because nodes are volatile, hence the public IPs will change.
As a consequence, on the other end (i.e. database service), you need to change network ACLs again to allow the new public IPs - not good. An egress gateway makes it so that all traffic coming from your application Pods inside the Kubernetes cluster, is seen as coming from a single public IP. That is, the public IP of the egress gateway. You can go even further and use a reserved IP for the egress gateway, so that it will never change.
Below is a diagram, showing the main setup for egressing DOKS cluster traffic to an external service (i.e. database):
- Overview
- Prerequisites
- Step 1 - Introducing Crossplane
- Step 2 - Installing Crossplane
- Step 3 - Introducing Static Routes Operator
- Step 4 - Installing Static Routes Operator
- Step 5 - Creating an Egress Gateway using Crossplane
- Step 6 - Configuring Static Routes for your Egress Gateway
- Step 7 - Testing the DOKS Cluster Egress Setup
- Cleaning Up
- FAQ
- Troubleshooting
- Summary
To complete this tutorial, you will need:
- Basic knowledge on how to run and operate
DOKSclusters. You can learn more here. - Basic knowledge on how to create and manage
Droplets. You can learn more here. - Basic knowledge and experience with shell commands (e.g.
bash). - Kubectl CLI, to interact with Kubernetes clusters. Make sure it is configured to point to your DOKS cluster, as explained here.
- Helm (version 3.x is required), to install Crossplane.
- Doctl for interacting with DigitalOcean API.
The main idea behind Crossplane is infrastructure management the Kubernetes way. It means, you can define and create CRDs using a declarative approach, and let Crossplane deal with the inner details. With Crossplane it's possible to create Droplets, Managed Databases, Load Balancers, even Kubernetes clusters (DOKS), via the DigitalOcean provider. Crossplane was designed with flexibility in mind, and it can be extended via providers.
Next, it's important to understand some important concepts behind Crossplace to create DigitalOcean resources (e.g. Droplets). There are four main concepts to know about:
- Packages allow Crossplane to be extended to include new functionality. You can think of packages the same way as Linux distributions pack and distribute applications. So, Crossplane packages distribute a set of CRDs and associated controllers to provision infrastructure resources.
- Providers represent packages that enable Crossplane to provision infrastructure (e.g. DigitalOcean Provider).
- Managed Resources are Kubernetes custom resources (CRDs) that represent infrastructure primitives. For example, to provision a Droplet, you would use a Droplet CRD.
- Composite Resources represents a special kind of custom resource. Main purpose is to compose together more managed resources into a higher level infrastructure unit which can be reused. For example, you can create a composite resource which consists of a managed database (e.g. DigitalOcean PostgreSQL) and a firewall resource, acting as a single deployable unit.
Below picture shows a simplified operational overview for Crossplane:
A typical Droplet CRD consumed by Crossplane looks like below:
apiVersion: compute.do.crossplane.io/v1alpha1
kind: Droplet
metadata:
name: egress-gw-nyc1
spec:
forProvider:
region: nyc1
size: s-1vcpu-1gb
image: ubuntu-20-04-x64
sshKeys:
- "7e:9c:b7:ee:74:16:a5:f7:62:12:b1:72:dc:51:71:85"
providerConfigRef:
name: do-provider-configExplanations for the above configuration:
spec.forProvider- defines all metadata required by the DigitalOcean provider to provision a new Droplet, such as:region,size,image, etc. Fields value map directly to the DigitalOcean Droplet specification. Also, if you have SSH keys deployed to your DigitalOcean account, you can specify the fingerprint in thesshKeysfield from the spec.spec.providerConfigRef- specifies a reference to a provider configuration CRD (explained in Step 3 - Creating an Egress Gateway using Crossplane). The provider configuration instructs the Droplet CRD how to connect to the DigitalOcean REST API, and what credentials to use (e.g. DO API token).
Hint:
You can also check this nice CRD viewer to see all the fields available for the Droplet kind, in a human readable format (available only for latest released version, which is 0.1.0 at this time of writing).
Under the hood, Crossplane is delegating the real work to the DigitalOcean provider, which in turn will use the provider REST API to create a new droplet based on the requirements from the Droplet CRD spec field.
With all features presented above at hand, you are able to build your own cloud platform using one or multiple providers. The possibilities are almost limitless.
Please visit the official documentation page for more information about the product and available features. The DigitalOcean provider home page for Crossplane is available here.
In the next step, you will learn how to install and configure Crossplane for your DOKS cluster using Helm.
Crossplane is available as a Helm chart for easy installation, as well as for future upgrades. Follow below steps to install Crossplane, via Helm (version 3.x is required):
-
Add and update the Crossplane Helm repository:
helm repo add crossplane-stable https://charts.crossplane.io/stable helm repo update
-
Search the
crossplane-stableHelm repository for available charts to install:helm search repo crossplane-stable
The output looks similar to:
NAME CHART VERSION APP VERSION DESCRIPTION crossplane-stable/crossplane 1.9.0 1.9.0 Crossplane is an open source Kubernetes add-on ... -
Deploy Crossplane to your DOKS cluster:
HELM_CHART_VERSION="1.16.0" helm install crossplane crossplane-stable/crossplane \ --version "${HELM_CHART_VERSION}" \ --namespace crossplane-system \ --create-namespace
Note:
A specific version for the Crossplane Helm chart is used. In this case
1.16.0is picked, which maps to the1.16.0version of the application. It’s good practice in general, to lock on a specific version. This helps to have predictable results, and allows versioning control viaGit.
Now, check if the Crossplane Helm chart was deployed to your cluster:
helm ls -n crossplane-systemThe output looks similar to (STATUS column value should be set to deployed):
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
crossplane crossplane-system 1 2022-09-10 20:40:20.903871 +0300 EEST deployed crossplane-1.9.0 1.9.0
Finally, verify Crossplane deployment status:
kubectl get deployments -n crossplane-systemThe output looks similar to:
NAME READY UP-TO-DATE AVAILABLE AGE
crossplane 1/1 1 1 3d19h
crossplane-rbac-manager 1/1 1 1 3d19h
All pods must be up and running (check the READY column). In the next step, a short introduction is given about the static routes operator used in this guide.
The main role of the Static Routes Operator is to manage entries in the Linux routing table of each worker node, based on CRD spec. It is deployed as a DaemonSet, hence it will run on each node of your DOKS cluster.
Below diagram illustrates the operational concept:
Configuring static routes is done via StaticRoute CRDs. A typical example is shown below:
apiVersion: networking.digitalocean.com/v1
kind: StaticRoute
metadata:
name: static-route-ifconfig.me
spec:
destinations:
- "34.160.111.145"
gateway: "10.116.0.5"Explanations for the above configuration:
spec.destinations- A list of host IPs (or subnet CIDRs) to route through the gateway.spec.gateway- Gateway IP address used for routing the host(s)/subnet(s), specified in thedestinationsfield.
Because the operator has access to the Linux routing table, special care must be taken and policies set so that only administrators have access. It's very easy to misconfigure the routing table, rendering the DOKS cluster unstable or unusable.
VERY IMPORTANT TO REMEMBER:
You need to make sure to NOT ADD static routes containing CIDRs which overlap with DigitalOcean REST API endpoints (including DOKS) ! Doing so, will affect DOKS cluster functionality (Kubelets), and/or other internal services (e.g. Crossplane).
In the next step, you will learn how to install and configure the static routes operator.
Static routes operator is available as a single manifest file, and it is installed via kubectl. A dedicated namespace, named static-routes, is created as well. Please follow below steps to install the static routes controller:
-
Deploy the latest version release using kubectl. Below example is using the
1.0.0version:kubectl apply -f https://raw.githubusercontent.com/digitalocean/k8s-staticroute-operator/main/releases/v1/k8s-staticroute-operator-v1.0.0.yaml
Hint:
You can check the latest version in the releases path from the k8s-staticroute-operator GitHub repo.
-
Check if operator Pods are up and running:
kubectl get pods -l name=k8s-staticroute-operator -n static-routes
Output looks similar to:
NAME READY STATUS RESTARTS AGE k8s-staticroute-operator-9vp7g 1/1 Running 0 22m k8s-staticroute-operator-mlfff 1/1 Running 0 22m -
Check operator logs - no exceptions should be reported:
kubectl logs -f ds/k8s-staticroute-operator -n static-routes
Output looks similar to:
[2022-08-24 14:42:13,625] kopf._core.reactor.r [DEBUG ] Starting Kopf 1.35.6. [2022-08-24 14:42:13,625] kopf._core.engines.a [INFO ] Initial authentication has been initiated. [2022-08-24 14:42:13,626] kopf.activities.auth [DEBUG ] Activity 'login_via_pykube' is invoked. [2022-08-24 14:42:13,628] kopf.activities.auth [DEBUG ] Pykube is configured in cluster with service account. [2022-08-24 14:42:13,629] kopf.activities.auth [INFO ] Activity 'login_via_pykube' succeeded. [2022-08-24 14:42:13,629] kopf.activities.auth [DEBUG ] Activity 'login_via_client' is invoked. [2022-08-24 14:42:13,631] kopf.activities.auth [DEBUG ] Client is configured in cluster with service account. [2022-08-24 14:42:13,632] kopf.activities.auth [INFO ] Activity 'login_via_client' succeeded. [2022-08-24 14:42:13,632] kopf._core.engines.a [INFO ] Initial authentication has finished. [2022-08-24 14:42:13,789] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for customresourcedefinitions.v1.apiextensions.k8s.io cluster-wide. [2022-08-24 14:42:13,791] kopf._cogs.clients.w [DEBUG ] Starting the watch-stream for staticroutes.v1.networking.digitalocean.com cluster-wide. ...
If the output looks like above, you installed the static routes operator successfully. In the next step, you will learn how to provision your first egress gateway Droplet using Crossplane.
To provision an Egress gateway Droplet on the DigitalOcean platform using Kubernetes and Crossplane, you need to follow a few steps:
- Install the DigitalOcean provider package.
- Configure the DigitalOcean provider so that it can talk with the provider REST API, and manage resources on your behalf.
- Provision infrastructure using the designated CRDs, such as Droplet kind, DODatabaseCluster kind, etc.
Next, you will learn how to accomplish each of the above steps, to better understand the concepts and separation of concerns.
Important note:
For each step, you will be using the default namespace in order to simplify testing. In practice, is recommended to create the resources in a dedicated namespace, with proper RBAC policies set.
In Crossplane terms, a provider package is a bundle of assets required by Crossplane to provide additional functionality, such as CRD manifests and associated controller(s). Under the hood everything is packed in a Docker image, and distributed as such.
Typical Crossplane provider package CRD looks like below:
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
name: provider-do
spec:
package: "xpkg.upbound.io/digitalocean/provider-digitalocean:v0.2.0"Explanations for the above configuration:
spec.package- defines the provider package to download, which in essence is a Docker image, so it follows the standard naming convention.
To install the DigitalOcean provider used in this guide, please follow below steps:
-
Fetch the provider manifest from the
container-blueprintsrepository, usingcurl:curl -O https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/do-provider-install.yaml
-
Open and inspect the manifest file, using a text editor of your choice (the default values are usually fine, unless you require a specific provider package version). For example, you can use Visual Studio Code, with YAML linting support:
code do-provider-install.yaml
-
Apply the manifest using
kubectl:kubectl apply -f do-provider-install.yaml
Or, directly from the
container-blueprintsrepo (if you're OK with the default values):kubectl apply -f https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/do-provider-install.yaml
Finally, check if the provider was installed successfully in your DOKS cluster:
kubectl get providersThe output looks similar to:
NAME INSTALLED HEALTHY PACKAGE AGE
provider-do True True crossplane/provider-digitalocean:v0.2.0-rc.0.42.g0932045-main 4d17h
The INSTALLED and HEALTHY columns should both report True.
Next, you will learn how to configure the DigitalOcean provider to have access to the DigitalOcean REST API, and manage resources on your behalf.
The DigitalOcean provider package installed previously needs a ProviderConfig CRD to operate properly. In order to perform REST operations, it needs to authenticate against the DigitalOcean REST API endpoint. For that, a valid DO API token is required, stored in a Kubernetes secret (as a base64 encoded value).
Typical ProviderConfig CRD looks like below:
apiVersion: do.crossplane.io/v1alpha1
kind: ProviderConfig
metadata:
name: do-provider-config
spec:
credentials:
source: Secret
secretRef:
namespace: crossplane-system
name: do-api-token
key: tokenExplanations for the above configuration:
spec.credentials.source- defines a source where credentials are stored (e.g. Kubernetes Secret).spec.credentials.secretRef- tells the provider how to access the secret, such as whatnamespaceit was created, itsname, and whatkeycontains the DO API token value as a base64 encoded string.
To install the DigitalOcean provider configuration used in this guide, please follow below steps:
-
Fetch the provider configuration manifest from the
container-blueprintsrepository, usingcurl:curl -O https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/do-provider-config.yaml
-
Open and inspect the manifest file, using a text editor of your choice. For example, you can use Visual Studio Code, with YAML linting support:
code do-provider-config.yaml
-
Replace the
<>placeholders in thedo-api-tokenKubernetes Secret CRD. Please run below command, to generate the base64 encoded string from your DO API token:echo "<YOUR_DO_API_TOKEN_HERE>" | base64
-
Save the provider configuration file, and apply using
kubectl:kubectl apply -f do-provider-config.yaml
Finally, check if the provider configuration resource was created successfully:
kubectl get providerconfigs -o wideThe output looks similar to:
NAME AGE SECRET-NAME
do-provider-config 4d18h do-api-token
If the output looks like above, then you configured the DigitalOcean provider successfully. Notice the SECRET-NAME column value - it references the secret storing your DO API token.
Please note that all of the steps taken so far need to be performed only once. Once a provider is installed and configured, you can reference it in any resource you want to create, such as Droplets, managed databases, etc. The only intervention required over time, is when you need to upgrade a provider package to get new functionality, or if you need to update your DO API token.
Next, you will take the final step and provision the egress gateway droplet for your egress setup.
IMPORTANT:
The egress gateway Droplet and DOKS cluster must be in the same VPC (or on the same network) for the whole setup to work !!!
Now that the DigitalOcean provider is installed and properly configured, you can proceed and create the actual Droplet resource to act as an egress gateway for the demo used this blueprint.
The Crossplane Droplet CRD used in this blueprint looks like below:
apiVersion: compute.do.crossplane.io/v1alpha1
kind: Droplet
metadata:
name: egress-gw-nyc1
spec:
forProvider:
region: nyc1
size: s-1vcpu-1gb
image: ubuntu-20-04-x64
vpcUuid: 4b5e125e-c52e-4578-93a7-01341ee927ac
sshKeys:
- "7e:9c:b7:ee:74:16:a5:f7:62:12:b1:72:dc:51:71:85"
userData: |
#!/usr/bin/env bash
# Install dependencies
echo iptables-persistent iptables-persistent/autosave_v4 boolean true | debconf-set-selections
echo iptables-persistent iptables-persistent/autosave_v6 boolean true | debconf-set-selections
apt-get update
apt-get -y install iptables iptables-persistent curl
# Enable IP forwarding
echo "net.ipv4.ip_forward=1" >> /etc/sysctl.conf
sysctl -p /etc/sysctl.conf
# Configure iptables for NAT
PRIVATE_NETWORK_INTERFACE_IP="$(curl -s http://169.254.169.254/metadata/v1/interfaces/private/0/ipv4/address)"
PRIVATE_NETWORK_CIDR="$(ip route show src $PRIVATE_NETWORK_INTERFACE_IP | awk '{print $1}')"
PUBLIC_INTERFACE_NAME="$(ip route show default | awk '{print $5}')"
iptables -t nat -A POSTROUTING -s "$PRIVATE_NETWORK_CIDR" -o "$PUBLIC_INTERFACE_NAME" -j MASQUERADE
iptables-save > /etc/iptables/rules.v4
providerConfigRef:
name: do-provider-configThe fields meaning is the same as explained at the beginning of this blueprint guide, when Crossplane was introduced). The only exception is the spec.forProvider.userData field. What this field does is, it allows you to specify custom cloud init scripts for the Droplet (e.g. Bash scripts, denoted by the first line value - #!/usr/bin/env bash).
In the above example, the cloud init script main role is to initialize the Droplet to act as a NAT gateway. The example is heavily based on this guide provided by DigitalOcean.
- The first line denotes the fact that the cloud init script needs to be interpreted as a Bash script -
#!/usr/bin/env bash. - Next, the required packages are installed, such as:
iptables,iptables-persistent(persists iptables rules on reboot), etc. Please bear in mind that the installation commands are Linux distribution specific (above example is Ubuntu specific, which in turn is based on Debian). - Then, IP forwarding is enabled for the Linux Kernel via the
net.ipv4.ip_forwardsysctl setting. Configuration is persisted in the/etc/sysctl.conffile to survive machine reboots. - Next, iptables is configured with required NAT rules, to provide egress functionality in combination with IP forwarding.
- Finally, NAT rules are saved using the
iptables-savecommand, to persist on reboots.
To create the egress gateway Droplet via Kubernetes, please follow below steps:
-
Fetch the Droplet CRD manifest from the
container-blueprintsrepository, usingcurl:curl -O https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/egress-gw-droplet.yaml
-
Open and inspect the manifest file, using a text editor of your choice (the default values are usually fine, unless you require something specific). For example, you can use Visual Studio Code, with YAML linting support:
code egress-gw-droplet.yaml
Hints:
- If you have specific SSH keys you want to add at Droplet creation time, you can do so by uncommenting the
sshKeysfield from the spec. Then, replace the<>placeholders with your SSH key fingerprint. To list the available SSH keys associated with your account and their fingerprint, you can do so by issuing the following command -doctl compute ssh-key list. - The egress gateway Droplet and DOKS cluster must be in the same VPC. Use the following command to get your VPC ID -
doctl vpcs listand set this by uncommenting thevpcUuidfield from the spec.
- If you have specific SSH keys you want to add at Droplet creation time, you can do so by uncommenting the
-
Save and apply the manifest using
kubectl:kubectl apply -f egress-gw-droplet.yaml
Or, directly from the
container-blueprintsrepo (if you're OK with the default values):kubectl apply -f https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/egress-gw-droplet.yaml
Finally, check if the Droplet Kubernetes resource was created successfully:
kubectl get droplets -o wideThe output looks similar to:
NAME PRIVATE IPV4 PUBLIC IPV4 READY REGION SIZE SYNCED
egress-gw-nyc1 10.116.0.4 192.81.213.125 True nyc1 s-1vcpu-1gb True
Important notes:
-
Please bear in mind that it takes some time for the Droplet to be provisioned, so please be patient.
-
The
PRIVATE IPV4andPUBLIC IPV4columns will receive values only after the Droplet is successfully provisioned. -
You can always check how the Droplet resource is progressing, or if something goes wrong by inspecting the
Eventsoutput from below command:kubectl describe droplet egress-gw-nyc1
The READY and SYNCED columns should print True. If the output looks like above, you configured the Droplet CRD successfully. Also, if you navigate to the Droplets web panel from your DigitalOcean account, you should see the new resource created. You should receive an email as well, with additional details about your newly created Droplet.
Finally, you can check if the cloud init script ran successfully, and did the required changes to enable IP forwarding and NAT functionality. Please follow below steps (all commands must be run as root, or via sudo):
-
Check if IP forwarding is set:
cat /proc/sys/net/ipv4/ip_forward # Should output: 1, meaning it's enabled -
Check if NAT is set via iptables:
iptables -L -t nat
The output looks similar to (the
POSTROUTINGchain should print theMASQUERADEtarget for your VPC subnet):... Chain POSTROUTING (policy ACCEPT) target prot opt source destination MASQUERADE all -- 10.116.0.0/20 anywhere
If all checks pass, you configured the egress gateway Droplet successfully. Next, you will learn how to define and create static routes for single as well as multiple destinations, using the egress gateway created earlier via Crossplane.
The sample CRDs provided in this blueprint create a static route to two different websites which report back your public IP - ifconfig.me/ip, and ipinfo.io/ip.
To test the setup, download each sample manifest:
# Example for ifconfig.me
curl -O https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/static-routes/ifconfig-me-example.yaml
# Example for ipinfo.io
curl -O https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/static-routes/ipinfo-io-example.yamlAfter downloading the manifests, make sure to replace the <> placeholders in each manifest file. To find out the private IPv4 address of your egress gateway droplet, please run below command (assuming the Droplet name is egress-gw-nyc1):
kubectl get droplet egress-gw-nyc1 -o jsonpath="{.status.atProvider.privateIPv4}"Next, save changes, and apply each manifest using kubectl:
# Example for ifconfig.me
kubectl apply -f ifconfig-me-example.yaml
# Example for ipinfo.io
kubectl apply -f ipinfo-io-example.yamlHint:
Above command will create the static route custom resources in the default namespace. In production environments (and not only), it's best to have a dedicated namespace with RBAC policies set.
Next, check if the static route resources were created:
kubectl get staticroutes -o wideThe output looks similar to (egress gateway has private IP 10.116.0.5 in below example):
NAME DESTINATIONS GATEWAY AGE
static-route-ifconfig.me ["34.160.111.145"] 10.116.0.5 7m2s
static-route-ipinfo.io ["34.117.59.81"] 10.116.0.5 4s
Next, check if the static route resources were created:
kubectl get staticroutes -o wideThe output looks similar to:
NAME DESTINATIONS GATEWAY AGE
static-route-ifconfig.me ["34.160.111.145"] 10.116.0.5 7m2s
static-route-ipinfo.io ["34.117.59.81"] 10.116.0.5 4s
Finally, check if the custom static routes were created on each worker node, after SSH-ing:
route -nThe output looks similar to (the irrelevant lines were omitted from the output for better visibility):
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 206.81.0.1 0.0.0.0 UG 0 0 0 eth0
...
34.117.59.81 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
34.160.111.145 10.116.0.5 255.255.255.255 UGH 0 0 0 eth1
...
Hints:
-
You can check each static route status by describing the resource. Taking
static-route-ifconfig.meas an example:kubectl describe staticroute static-route-ifconfig.me
The output looks similar to (only last relevant lines are shown for simplicity):
... Spec: Destinations: 34.160.111.145 Gateway: 10.116.0.4 Status: create_fn: Destination: 34.160.111.145 Gateway: 10.116.0.4 Status: Ready Events: <none>Looking at the above output, you can see each route details including status (
ReadyorNotReady), in theStatusfield. -
To see all events emitted by the static routes controller, you can use below command:
kubectl get events -n static-routes
-
For further troubleshooting, you can check the static routes controller logs:
# Fetch all logs from the static routes controller and redirect to a file kubectl logs -n static-routes ds/k8s-staticroute-operator > k8s-staticroute-operator.log # Use vistual studio code, or your favourite text editor to inspect the logs code k8s-staticroute-operator.log
To test your egress setup, you need to check if all the requests originating from your DOKS cluster travel via a single node - the Egress Gateway. The simplest way to test, is by making a curl request to ifconfig.me/ip, and see if the response contains your Egress Gateway public IP address.
First, you need to create the curl-test pod in your DOKS cluster (the default namespace is used):
kubectl apply -f https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/curl-test.yamlVerify that the pod is up and running (in the default namespace):
kubectl get podsThe output looks similar to (curl-test pod Status should be Running):
NAME READY STATUS RESTARTS AGE
curl-test 1/1 Running 0 7s
Then, perform a HTTP request to ifconfig.me, from the curl-test pod:
kubectl exec -it curl-test -- curl ifconfig.meThe output looks similar to:
192.81.213.125
The resulting IP address that gets printed should be the public IP address that was assigned to your egress gateway droplet. You can check your NAt gateway Droplet public IPv4 address via kubectl:
kubectl get droplet egress-gw-nyc1 -o jsonpath="{.status.atProvider.publicIPv4}"If you want to clean up all the resources associated with this guide, you can do so for each major component, as follows.
To clean up the operator and associated resources, please run the following kubectl command (make sure you're using the same release version as in the install step):
kubectl delete -f https://raw.githubusercontent.com/digitalocean/k8s-staticroute-operator/main/releases/v1/k8s-staticroute-operator-v1.0.0.yamlNote:
Above command will delete the associated namespace as well (static-routes). Make sure to backup your CRDs first, if needed later.
The output looks similar to:
customresourcedefinition.apiextensions.k8s.io "staticroutes.networking.digitalocean.com" deleted
serviceaccount "k8s-staticroute-operator" deleted
clusterrole.rbac.authorization.k8s.io "k8s-staticroute-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "k8s-staticroute-operator" deleted
daemonset.apps "k8s-staticroute-operator" deleted
Check the routes on each worker node, after SSH-ing:
route -nThe custom static routes should not be present in the routing table output.
Finally, the curl-test pod should report back the public IP of the worker node where it runs:
# Inspect the node where the curl-test Pod runs:
kubectl get pod curl-test -o wideThe output looks similar to (write down the node name from the NODE column):
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
curl-test 1/1 Running 2 (45m ago) 165m 10.244.0.140 basicnp-7micg <none> <none>Above example reports - basicnp-7micg.
Check the worker node public IP:
kubectl get nodes -o wideThe output looks similar to (note the public IP of the associated node where the curl-test Pod runs):
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
basicnp-7micg Ready <none> 3h20m v1.23.9 10.116.0.2 206.189.231.90 Debian GNU/Linux 10 (buster) 5.10.0-0.bpo.15-amd64 containerd://1.4.13
basicnp-7micw Ready <none> 3h20m v1.23.9 10.116.0.3 206.81.2.154 Debian GNU/Linux 10 (buster) 5.10.0-0.bpo.15-amd64 containerd://1.4.13
Above example reports - 206.189.231.90.
Exec the ifconfig.me/ip curl:
kubectl exec -it curl-test -- curl ifconfig.me/ipThe output looks similar to:
206.189.231.90
The response should include the original public IP of the worker node where the curl-test Pod runs.
Removing the egress gateway droplet is just a matter of deleting the associated CRD (please bear in mind that this process is destructive, and you cannot revert back):
kubectl delete -f https://raw.githubusercontent.com/digitalocean/container-blueprints/main/DOKS-Egress-Gateway/assets/manifests/crossplane/egress-gw-droplet.yamlAfter running above command, the associated Droplet resource should be destroyed and removed from your DigitalOcean account.
-
Q: What Linux table ID does the static routes operator use?
A: The static routes operator acts on the main table (ID 254).
-
Q: What happens if I define same routes or overlapping subnets via separate StaticRoute CRDs?
A: The static routes operator doesn't perform checks across CRDs (tracked via #13 on operator's GitHub repo). So, it is possible to override static routes via different CRDs.
-
Q: Is it possible to override routes created by the system (e.g. Cilium)?
A: Yes, it is if updating existing CRDs. This doesn't happen when the CRD is created for the first time though. In the end, it's more safe to have a list of protected IPs/subnets defined in a ConfigMap. The operator will read the list at startup (or whenever it changes), and will not let users apply those routes (tracked via #11 on operator's GitHub repo).
-
Q: Am I allowed to change the default gateway?
A: No. It is considered an unsafe operation which can cause trouble (tracked via #6 on operator's GitHub repo).
Whenever you delete a resource in Kubernetes, the associated controller puts a finalizer on the respective object in the metadata field. Bellow snippet shows the Finalizers field in the Static Route CRD:
Name: staticroutes.networking.digitalocean.com
Namespace:
Labels: <none>
Annotations: provider: digitalocean
API Version: apiextensions.k8s.io/v1
Kind: CustomResourceDefinition
Metadata:
Creation Timestamp: 2022-09-19T15:05:34Z
Deletion Timestamp: 2022-09-27T14:51:10Z
Finalizers:
customresourcecleanup.apiextensions.k8s.io
...Kubernetes checks for the finalizer field first and if found, it will not delete the resource from the cluster until the associated controller finishes its job internally. If everything goes as planned, the controller removes the finalizer field from the object, and then and only then, Kubernetes moves forward with the actual deletion of the resource.
The main role of a finalizer is to allow the associated controller finish its job internally before actual object deletion.
If for some reason controller logic is unable to process the request and remove the finalizer field, the resource hangs in a Terminating state. Then, kubectl freezes because the Kubernetes API is not able to process the request.
As a side effect, this warning is reported in the static routes controller logs:
...
[2022-09-27 14:58:11,405] kopf._core.reactor.o [WARNING ] Non-patchable resources will not be served: {staticroutes.v1.networking.digitalocean.com}
...
Why this happens and how do I recover?
It can happen for various reasons - one such example is when the controller is down because of a upgrade, hence it is unable to process requests. In this case the CRD remains in an inconsistent state, and the only way to recover is via:
kubectl patch staticroute <YOUR_STATIC_ROUTE_RESOURCE_NAME_HERE> -p '{"metadata": {"finalizers": []}}' --type mergeAbove command removes the finalizer field from the resource, thus allowing Kubernetes proceed with actual object deletion. As a consequence, the Linux routing table still holds the old route entries. You have several options here:
- Manually delete the old entries. This can be a daunting task if many routes and nodes are implied.
- Restart the operator, and allow it to remove the finalizers. Issue the following command to perform a rollout for the static routes operator DaemonSet -
kubectl rollout restart -n static-routes ds/k8s-staticroute-operator. - As a last resort, re-deploy the static routes controller, and re-apply the original static route CRDs. Then, delete the CRDs. If everything goes as planned, the static routes controller takes care of the rest.
In this tutorial you learned how to use Crossplane to create and manage an egress Gateway resource for your DOKS cluster. This way, external services (e.g. database), will see a single source IP in the packets coming from your DOKS cluster, thus making firewall rules management easier on the other end. Also, you learned how to use the static routes operator, to manage specific routes for the egress functionality.
Additional resources:
- Static Routes Operator project page.
- Crossplane Docs page.
- DigitalOcean Droplets as NAT Gateways tutorial.