proxy-servers.md

AKS Engine and proxy servers

Using AKS Engine on Azure Stack in connected and disconnected environments often times requires the use of a non-transparent proxy server. Non-transparent means that they're not part of the default route and have to be configured to be used.

Azure Stack Hub itself supports only transparent proxy server setups today. This isn't relevant for the workload, as long as the workload (and it's components) can access the proxy server.

This document guides you through the different components and layers where you need to configure the proxy server. Considerations include:

• AKS-engine
• Cluster Nodes
• Pods

What kind of egress traffic you can expect is documented here: Control egress traffic for cluster nodes in Azure Kubernetes Service (AKS). Even though the document is referring to AKS (in Azure) it gives you an idea what kind of traffic and requests you can expect.

AKS Engine

AKS Engine means in this context the VM (Windows or Linux) that is used to run AKS Engine to deploy, scale and upgrade your Kubernetes cluster.

Linux

On a Linux system you've to make sure that you export the proxy server configuration via environment variables. Most commandline tools will leverage these environment variables automatically w/o additional configuration.

You can either set these environment variables in your current session, persistent for a specific user in ~/.bashrc or permanently for the whole system in /etc/profile (or in /etc/environment).

export HTTP_PROXY=http://proxy:8888
export HTTPS_PROXY=http://proxy:8888

In case your proxy servers require authentication, you've to set a username and a password like this:

export HTTP_PROXY="http://usrname:passwrd@host:port"
export HTTPS_PROXY="http://usrname:passwrd@host:port"

Important! Please keep in mind that these credentials are stored in plaintext and accessible for everyone on the system. Use either individual credentials per system or use a service-side authorization (e.g. for specific IPs) instead.

In case you want to exclude specific URLs and IP-addresses that do not not need a proxy you can use:

export NO_PROXY=master.hostname.example.com,<docker_registry_ip>,docker-registry.default.svc.cluster.local

Setting these environment variables will automatically enable tools like wget, curl etc. to send their traffic through the configured proxy server.

Some tools and services might require a specific, dedicated configuration.

To set the proxy server configuration permanently on a system it's recommended to write the configuration into a separate file in /etc/profile.d.

Here's an example how this can be achieved:

echo "export http_proxy=http://host:port/" > /etc/profile.d/http_proxy.sh

This will make the configuration persistent. It will now survive a reboot of the os/machine.

Windows

On Windows-based systems you have to use netsh to configure the proxy server:

netsh winhttp set proxy <proxy>:<port>

Cluster Nodes

In your cluster you've to configure a proxy server on both, your worker nodes as well as your master nodes. This is required for example to use apt and also for docker pull to download container images.

You can use the same manual configuration for the proxy servers as described in the AKS-engine section above. But that's a very static way to achieve that. A better way to dynamically configure your cluster nodes is using a Kubernetes DaemonSet this will also take care of newly added nodes while scaling out your cluster.

DaemonSet
Here's an example how you can set the proxy server configuration to all nodes in your cluster via a Kubernetes DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: proxy-configuration
spec:
  selector:
    matchLabels:
      name: proxy-configuration
  template:
    metadata:
      labels:
        name: proxy-configuration
    spec:
      volumes:
        - name: hostfs
          hostPath:
            path: /
      initContainers:
        - name: init
          image: alpine
          command:
            - /bin/sh
            - -xc
            - |
              # Write Proxy Config to /etc/profile.d/
              echo "export http_proxy=http://host:port/" > /etc/profile.d/http_proxy.sh
          volumeMounts:
            - name: hostfs
              mountPath: /host
      containers:
        - name: sleep
          image: alpine
          command:
            - /bin/sh
            - -xc
            - |
              while ($true) do { sleep 60; } done;

Linux

export HTTP_PROXY=http://proxy:8888
export HTTPS_PROXY=http://proxy:8888

Besides the system itself you might need to configure the proxy server for apt (aptitude) directly:

• Setting up apt-get to use a http-proxy

Windows

netsh winhttp set proxy <proxy>:<port>

Pods

Even your Pods/Container Instances need in some cases to go through a proxy server to access services outside the cluster or outside your network. You can inject the proxy configuration directly within your YAML manifest:

containers:
- env:
  - name: "HTTP_PROXY"
    value: "http://USER:PASSWORD@IPADDR:PORT"

External Resources

• Configure machine proxy and Internet connectivity settings (Windows)
• Configure Docker to use a proxy server (Docker)
• Setting up apt-get to use a http-proxy (Ubuntu Linux)
• How to set up proxy using http_proxy & https_proxy environment variable in Linux? (Linux)
• Proxies and Gatewaying (GNU docs)