Kubernetes deployed containers are not able to reach internet

[sowmiya99]

Spawned a Jenkins container thorugh Kubernetes. Jenkisn job had git checkout task configgured. But getting the following error since the container is not able to reach internet.

ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Command "git fetch --tags --progress URL +refs/heads/:refs/remotes/origin/" returned status code 128:
stdout:
stderr: fatal: unable to access URL: Could not resolve host: github.com

On logging into the container , couldn't ping github.com or ping google.com

Kindly help!

[gtirloni]

What version of minikube (minikube version) and Kubernetes (kubectl version) are you using? What's the hypervisor?

If Kubernetes is able to fetch the container image and start the container, there's a good chance connectivity is working as expected in the minikube VM.

Try to run a generic container and see if it has network connectivity: kubectl run busybox --image=busybox --rm -ti --restart=Never --command -- ping -c 5 google-public-dns-a.google.com

If that works, it means Kubernetes is working correctly (fetching image, running container) and that the container itself can access the Internet and also that DNS is working.

You can also test the network from inside the minikube VM: minikube ssh

This is what I see when I try to use the Jenkins image with minikube 0.17.1 and Kubernetes 1.6.0:

$ kubectl run jenkins --image=jenkins --rm -ti --restart=Never --command -- /bin/sh
If you don't see a command prompt, try pressing enter.

$ cat /etc/resolv.conf
nameserver 10.0.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

$ ip route   
default via 172.17.0.1 dev eth0 
172.17.0.0/16 dev eth0  proto kernel  scope link  src 172.17.0.3 

$ ping -c 4 google.com
PING google.com (172.217.29.110): 56 data bytes
64 bytes from 172.217.29.110: icmp_seq=0 ttl=61 time=20.127 ms
64 bytes from 172.217.29.110: icmp_seq=1 ttl=61 time=19.237 ms
64 bytes from 172.217.29.110: icmp_seq=2 ttl=61 time=19.942 ms
64 bytes from 172.217.29.110: icmp_seq=3 ttl=61 time=19.923 ms
--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max/stddev = 19.237/19.807/20.127/0.339 ms

And from the minikube VM:

$ minikube ssh
$ cat /etc/resolv.conf | egrep -v '^#'
nameserver 10.0.2.3

$ ip route
default via 10.0.2.2 dev eth0  proto dhcp  src 10.0.2.15  metric 1024 
10.0.2.0/24 dev eth0  proto kernel  scope link  src 10.0.2.15 
10.0.2.2 dev eth0  proto dhcp  scope link  src 10.0.2.15  metric 1024 
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.0.1 
192.168.99.0/24 dev eth1  proto kernel  scope link  src 192.168.99.100 

$ ping -c 4 google.com
PING google.com (172.217.29.110): 56 data bytes
64 bytes from 172.217.29.110: seq=0 ttl=63 time=20.012 ms
64 bytes from 172.217.29.110: seq=1 ttl=63 time=19.372 ms
64 bytes from 172.217.29.110: seq=2 ttl=63 time=19.813 ms
64 bytes from 172.217.29.110: seq=3 ttl=63 time=20.500 ms

--- google.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 19.372/19.924/20.500 ms

[basservice]

Thanks for the reply.

I'm not using minikube. I'm usign Kubernetes 1.6 cluster.
One master and 2 slave nodes attached to it.

On executing , kubectl run busybox --image=busybox --rm -ti --restart=Never --command -- ping -c 5 google-public-dns-a.google.com

I get bad address google.com

[r2d4]

This is a repository for minikube issues. You might want to reach out in the kubernetes slack channel under #kubernetes-users for help debugging your cluster.

[sowmiya99]

Sure. Thank you!

…

On Tue, Apr 11, 2017 at 8:16 PM, Matt Rickard ***@***.***> wrote: Closed #1340 <#1340>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1340 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AKsQZHwWp5jC483ACuxcTemsCzw4su-qks5ru5JQgaJpZM4M2vci> .

[mkuzmentsov]

@sowmiya99 were you able to resolve this issue? I got exactly the same problem with k8s 1.7.

[mattdodge]

What happened here? From what I can tell this is an issue within minkube (at least I'm experiencing it too). Some random person (not OP) came on out of nowhere and said they weren't using minikube so it got closed? I think this probably should be re-opened...

[sowmiya99]

@BattleBeaver

Sorry for late response.
I followed the below process and issue has been resolved.

Get cluster IP of Kube
#kubectl get svc --all-namespaces
NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes xx.xx.x.xx 443/TCP 24d
kube-system kube-dns xx.xx.x.xx 53/UDP,53/TCP 24d

2. Edit the following file in the Slave nodes -
   #cat /etc/resolv.conf
   ##############################################################
   #Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
   #DO NOT EDIT THIS FILE WITH HAND -- YOUR CHANGES WILL BE OVERWRITTEN
   #nameserver xx.xx.xx.xx ##comment the default nameserver
   nameserver xx.xx.xx.xx ##add the cluster ip obtained above
   search ec2.internal
   ##############################################################

This will help the container to gain internet access.

Regards,
Sowmiya

[6graNik]

Adding cube dns worked for me

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  upstreamNameservers: |
    ["8.8.8.8"]

more information https://blog.yaakov.online/kubernetes-getting-pods-to-talk-to-the-internet/

[Nurlan199206]

@6graNik didn't help for me. i'm using minikube

[6graNik]

Me too

[pritidesai]

yup, I am using minikube and running into similar issue:

minikube version
minikube version: v0.34.1

ping -c 4 google.com
PING google.com (172.217.6.78): 56 data bytes

--- google.com ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss

[nomansadiq11]

me too also facing same issue I also added dns info but still same issue.

services showing gray instead green

[akashclose2u]

adding "nameserver 8.8.8.8" in /etc/resolve.conf, helped me reach internet.

[ikurtz]

It's DNS, It's not DNS... it's fucking DNS.

I'm just attempting to install a barebones Jenkins instance on minikube (1.17.1) running on MacOS Big Sur 11.2.1. The DNS resolver for minikube (coredns) seems to employ an endless loop of DNS requests that never end up leaving the pod. I've followed the instructions here --> https://www.jenkins.io/doc/book/installing/kubernetes/

I'm getting a healthy DNS log output that indicates a running MD5 checksum config and a healthy pod log output for my Jenkins app indicating it's fully up and running. However, when I attempt to access the instance via the minikube ip and the specified service port it just times out. Has ANYONE found a workaround for getting Jenkins installed on a single minikube node? Plsss share.

1. Jenkins deployment was successful
2. Both Jenkins pod and coredns are RUNNING with healthy log outputs
3. /etc/resolv.conf in Jenkins pod lists the DNS server for the kube-dns service (10.96.0.10)

Thanks!

[Type1J]

If anybody gets here because networking isn't working for a container started outside of Kubernetes 2 things:

• AWS Removed the bridge network interface unless you add an option to bootstrap.sh. See: Docker in Docker no longer works without docker0 bridge awslabs/amazon-eks-ami#183
• If you want a quick fix (that may or may not be a good solution), you can use --network=host when you invoke docker run ... or docker build ....

[dnguyenzd]

I solved this issue with my minikube by editing coredns configuration:

• Run kubectl edit configmap coredns -n kube-system
• My original coredns configmap has following section:

        forward . /etc/resolv.conf {
           max_concurrent 1000
        }

• Change it to: forward . 8.8.8.8 8.8.4.4, wait for coredns pod to be restarted and pods in minikube are able to reach internet

[brainthinks]

UPDATE:

Turns out something with my minikube VM got corrupted or improperly configured. I ran

minikube delete

which destroys the VM and all minikube containers. After this, I re-deployed, and my deployment worked as expected with no DNS issues, even with Alpine / MUSL.

Everything below was my original, unstable solution to the problem:

For anyone still having this issue, here is what I was able to determine.

This only happens (for me) when using alpine docker images. Through some investigation, it seems that because alpine uses musl rather than glibc (which is a source of some other headaches, such as AWS CLI not working), something about the DNS resolution is different, and this makes resolving "normal" DNS names, such as "google.com" not work.

You can test this yourself by doing the following inside of the container:

# this failed for me
ping google.com 
# this worked for me (notice the dot at the end, explicitly signifying it's an FQDN)
ping google.com.

I don't know the true underlying cause of the issue, but I was able to fix it by doing 2 things:

First, create an empty file at the following location on your host machine: ~/.minikube/files/etc/resolv.conf - see the relevant documentation here: https://minikube.sigs.k8s.io/docs/handbook/filesync/

touch ~/.minikube/files/etc/resolv.conf

Second, install bind-tools in your alpine container:

# from any alpine image...
FROM nginx:1-alpine

# be sure to install `bind-tools`
RUN apk update && apk add bind-tools

# ...

With those 2 steps, I was able to get "proper" DNS resolution working inside of my alpine containers running in minikube. I don't fully know why it works.

Additionally, you can observe this with stock busybox images. @gtirloni graciously gave us the command that will allow us to test whether or not DNS works properly using default minikube settings:

minikube kubectl -- run busybox-test --image=busybox --rm -ti --restart=Never --command -- ping google.com

If you run this same command (before implementing the fixes above), but use the musl version of busybox, that ping will fail:

minikube kubectl -- run busybox-musl-test --image=busybox:musl --rm -ti --restart=Never --command -- ping google.com

Hopefully this helps someone.