Skip to content

Commit

Permalink
Add a section about routing errors
Browse files Browse the repository at this point in the history
A common source of beffudlement when using local hypervisors or cloud
providers with peculiar interface setups, IP adressing, or network
policies for which kubelet cannot guess the right IP to use.

`awk` is being used so that the example works in CoreOS Container Linux
too.

Requested in kubernetes/kubeadm#203.
  • Loading branch information
lloeki committed Jan 24, 2018
1 parent b960552 commit f70ffd8
Showing 1 changed file with 34 additions and 0 deletions.
34 changes: 34 additions & 0 deletions docs/setup/independent/troubleshooting-kubeadm.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,3 +181,37 @@ If you're using flannel as the pod network inside vagrant, then you will have to
Vagrant typically assigns two interfaces to all VMs. The first, for which all hosts are assigned the IP address `10.0.2.15`, is for external traffic that gets NATed.
This may lead to problems with flannel. By default, flannel selects the first interface on a host. This leads to all hosts thinking they have the same public IP address. To prevent this issue, pass the `--iface eth1` flag to flannel so that the second interface is chosen.
### Routing errors
In some situations `kubectl logs` and `kubectl run` commands may return with the following errors despite an otherwise apparently correctly working cluster:
```
Error from server: Get https://10.19.0.41:10250/containerLogs/default/mysql-ddc65b868-glc5m/mysql: dial tcp 10.19.0.41:10250: getsockopt: no route to host
```
This is due to Kubernetes using an IP that can not communicate with other IPs on the seemigly same subnet, possibly by policy of the machine provider. As an example, Digital Ocean assigns a public IP to `eth0` as well as a private one to be used internally as anchor for their floating IP feature, yet `kubelet` will pick the latter as the node's `InternalIP` instead of the public one.
To check for such a scenario, using `ip addr show` is advised instead of `ifconfig` to check assigned IPs against the one in the error, as the latter will not display the offending alias IP address. Alternatively an API endpoint specific to Digital Ocean allows to query for the anchor IP from the droplet:
```
curl http://169.254.169.254/metadata/v1/interfaces/public/0/anchor_ipv4/address
```
The workaround is to tell `kubelet` which IP to use using `--node-ip`. When using Digital Ocean, it can be the public one (assigned to `eth0`) or the private one (assigned to `eth1`) should you want to use the optional private network. For example:
```
IFACE=eth0 # change to eth1 for DO's private network
DROPLET_IP_ADDRESS=$(ip addr show dev $IFACE | awk 'match($0,/inet (([0-9]|\.)+).* scope global/,a) { print a[1]; exit }')
echo $DROPLET_IP_ADDRESS # check this, just in case
echo "Environment=\"KUBELET_EXTRA_ARGS=--node-ip=$DROPLET_IP_ADDRESS\"" >> /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
```
Please note that this assumes `KUBELET_EXTRA_ARGS` hasn't already been set in the unit file.
Then restart `kubelet`:
```
systemctl daemon-reload
systemctl restart kubelet
```

0 comments on commit f70ffd8

Please sign in to comment.