Kubeadm creating multi master cluster DNS Resolution not working

[abizake]

Is this a request for help?

If no, delete this section and continue on.

What keywords did you search in kubeadm issues before filing this one?

If you have found any duplicates, you should instead reply there and close this page.

If you have not found any duplicates, delete this section and continue on.

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT
Choose one: BUG REPORT or FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):
1.12.1
Environment:

• Kubernetes version (use kubectl version): 1.12.1
• Cloud provider or hardware configuration: On Prem
• OS (e.g. from /etc/os-release): CentOS Linux 7 (Core)
• Kernel (e.g. uname -a):3.10.0-957.5.1.el7.x86_64
• Others:
  Docker Version : 18.6.2

What happened?

When trying to test the dns resolution working or not on a newly set up k8s cluster , when I try :

kubectl exec -ti busybox -- nslookup kubernetes.default

it gives the following ouput

Server:    10.0.0.10
Address 1: 10.0.0.10

nslookup: can't resolve 'kubernetes.default

What you expected to happen?

kubectl exec -ti busybox -- nslookup kubernetes.default

should give the following

Server:    10.0.0.10
Address 1: 10.0.0.10

Name:      kubernetes.default
Address 1: 10.0.0.1

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Kube Proxy Logs (also on master nodes) :

0219 10:56:34.395729 1 server_others.go:189] Using ipvs Proxier.
W0219 10:56:34.396299 1 proxier.go:343] IPVS scheduler not specified, use rr by default
I0219 10:56:34.396803 1 server_others.go:216] Tearing down inactive rules.
E0219 10:56:34.514331 1 proxier.go:430] Failed to execute iptables-restore for nat: exit status 1 (iptables-restore: line 7 failed
)
I0219 10:56:34.523628 1 server.go:447] Version: v1.12.1
I0219 10:56:34.546089 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0219 10:56:34.546246 1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0219 10:56:34.546396 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0219 10:56:34.546478 1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0219 10:56:34.547102 1 config.go:102] Starting endpoints config controller
I0219 10:56:34.547123 1 controller_utils.go:1027] Waiting for caches to sync for endpoints config controller
I0219 10:56:34.547165 1 config.go:202] Starting service config controller
I0219 10:56:34.547179 1 controller_utils.go:1027] Waiting for caches to sync for service config controller
I0219 10:56:34.647319 1 controller_utils.go:1034] Caches are synced for endpoints config controller
I0219 10:56:34.647324 1 controller_utils.go:1034] Caches are synced for service config controller

Kube adm Config File :-

apiVersion: kubeadm.k8s.io/v1alpha3
kind: InitConfiguration
nodeRegistration:
  name: "172.16.11.86"
  kubeletExtraArgs:
    cgroup-driver: "systemd"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
CgroupDriver: "systemd"

---
apiVersion: kubeadm.k8s.io/v1alpha3
kind: ClusterConfiguration
kubernetesVersion: v1.12.1
apiServerCertSANs:
- "172.16.11.86"
controlPlaneEndpoint: "172.16.11.86:6443"
apiServerExtraArgs:
  apiserver-count: "1"
  service-node-port-range: 380-32767
api:
  advertiseAddress: 172.16.11.86
  bindPort: 6443
etcd:
  local:
    extraArgs:
      listen-client-urls: "https://127.0.0.1:2379,https://172.16.11.86:2379"
      advertise-client-urls: "https://172.16.11.86:2379"
      listen-peer-urls: "https://172.16.11.86:2380"
      initial-advertise-peer-urls: "https://172.16.11.86:2380"
      initial-cluster: "172.16.11.86=https://172.16.11.86:2380"
      initial-cluster-state: new
    serverCertSANs:
      - 172.16.11.86
    peerCertSANs:
      - 172.16.11.86
networking:
    # This CIDR is a Calico default. Substitute or remove for your CNI provider.
    podSubnet: "192.168.0.0/16"
    serviceSubnet: 10.96.0.0/12
    dnsDomain: cluster.local
clusterName: "k8-cluster-test"
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
hostnameOverride: "172.16.11.86"
clusterCIDR : "192.168.0.0/16"

kube-dns logs

 for p in $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name); do kubectl logs --namespace=kube-system $p; done

.:53
2019/02/19 10:54:09 [INFO] CoreDNS-1.2.2
2019/02/19 10:54:09 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2019/02/19 10:54:09 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
2019/02/19 11:08:31 [INFO] Reloading
2019/02/19 11:08:31 [INFO] plugin/reload: Running configuration MD5 = c55db9af7a6a362f860f9ea9036f4edb
2019/02/19 11:08:31 [INFO] Reloading complete
127.0.0.1:59151 - [19/Feb/2019:11:08:31 +0000] 6109 "HINFO IN 2411300716778801859.8013831051134805532. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.066969186s
.:53
2019/02/19 10:54:04 [INFO] CoreDNS-1.2.2
2019/02/19 10:54:04 [INFO] linux/amd64, go1.11, eb51e8b
CoreDNS-1.2.2
linux/amd64, go1.11, eb51e8b
2019/02/19 10:54:04 [INFO] plugin/reload: Running configuration MD5 = f65c4821c8a9b7b5eb30fa4fbc167769
2019/02/19 11:07:51 [INFO] Reloading
2019/02/19 11:07:51 [INFO] plugin/reload: Running configuration MD5 = c55db9af7a6a362f860f9ea9036f4edb
2019/02/19 11:07:51 [INFO] Reloading complete
127.0.0.1:52806 - [19/Feb/2019:11:07:52 +0000] 33651 "HINFO IN 2914971660933733705.1837770882138718035. udp 57 false 512" NXDOMAIN qr,rd,ra 57 0.069761877s

Please advise . Seems I am missing some important configuration while setting up the cluster.

[fabriziopandini]

@abizake
Could you kindly check with other is (not busy box)
I have faced the same problem in the past and it turned out it was a busy box issue (docker-library/busybox#48)

[fabriziopandini]

And also kelseyhightower/kubernetes-the-hard-way#356

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.