Service Name Resolution is not working

[sacashgit]

Hi There

I have a ubuntu multi-node cluster set up - running. I am able to create pods, controllers and services at will. But I am facing the issue where service name resolution is not working. As part of the guestbook example, I have created redis-slave, redis-master and frontend services. But when containers are trying to talk to each other through this service interface, it is not working.

E.g

1. When PHP application try and talk to redis-slave:6379 - it fails (This call dont get routed any where)
2. When redis slave try to synch with master using redis-master , it doesnt work.
3. When I try and ping redis-master from inside the container, it comes back with unknown host error.

Some logs to show you the problem

[8] 10 Apr 00:03:57.902 # Unable to connect to MASTER: Connection timed out
[8] 10 Apr 00:03:58.905 * Connecting to MASTER redis-master:6379
[8] 10 Apr 00:03:58.909 # Unable to connect to MASTER: Connection timed out
[8] 10 Apr 00:03:59.913 * Connecting to MASTER redis-master:6379

Temporary failure in name resolution [tcp://redis-slave:6379]' in /vendor/predis/predis/lib/Predis/Connection/AbstractConnection.php:141

I do see Kube-Proxy is making the right iptable entries.
I0409 13:04:47.690367 20353 proxier.go:556] Opened iptables from-containers portal for service "redis-master" on TCP 11.1.1.67:6379
I0409 13:04:47.696276 20353 proxier.go:567] Opened iptables from-host portal for service "redis-master" on TCP 11.1.1.67:6379
I0409 13:11:50.223702 20353 proxier.go:556] Opened iptables from-containers portal for service "redis-slave" on TCP 11.1.1.55:6379
I0409 13:11:50.252093 20353 proxier.go:567] Opened iptables from-host portal for service "redis-slave" on TCP 11.1.1.55:6379
I0409 13:14:17.024773 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 11.1.1.58:8000
I0409 13:14:17.034296 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 11.1.1.58:8000
I0409 13:14:17.046845 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 10.64.80.83:8000
I0409 13:14:17.057679 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 10.64.80.83:8000
I0409 13:14:17.067562 20353 proxier.go:556] Opened iptables from-containers portal for service "frontend" on TCP 10.64.80.84:8000
I0409 13:14:17.077085 20353 proxier.go:567] Opened iptables from-host portal for service "frontend" on TCP 10.64.80.84:8000

Do you know what is missing in the set up which is causing this problem ? I do not have any DNS (skydns) running here and think that it is desirable but not needed.

What might be going wrong here ?

[lavalamp]

It sounds like something about your network is not configured correctly. I think you have to tell every node what range of IP addresses it should answer to.

@thockin is it possible to make kube-proxy configure that? Seeing as it's already doing IPTables magic.

[sacashgit]

@lavalamp - I am trying to figure out what. To work around this, in my code, when I refer to 'Services' using the IP address 11.1.X.X range, it worked just fine. That confirms that Service names are not getting resolved at all. I don't know much about where these services names are registered and resolved. None of the logs that I have access to shows why the name resolution failed or any issues while creating the new service. I am clueless. Are there any specific logs or commands I must run to get some insights here ? Appreciate your help.

[thockin]

A few things.

First, 11.* is a real IP range - you probably should not be using it unless
you own it :)

Second, you actually do have to run the DNS server to get DNS resolution.

On Fri, Apr 10, 2015 at 12:36 PM, sacashgit [email protected]
wrote:

@lavalamp https://github.com/lavalamp - I am trying to figure out what.
To work around this, in my code, when I refer to 'Services' using the IP
address 11.1.X.X range, it worked just fine. That confirms that Service
names are not getting resolved at all. I don't know much about where these
services names are registered and resolved. None of the logs that I have
access to shows why the name resolution failed or any issues while creating
the new service. I am clueless. Are there any specific logs or commands I
must run to get some insights here ? Appreciate your help.

—
Reply to this email directly or view it on GitHub
#6667 (comment)
.

[sacashgit]

@thockin - Thanks for the quick note.

IP Range : I dont own it :). I just didnt change any of that when I built the binaries and kept the config files as is to large extent. I can take care of that.

Name resolution: So I do need to run the DNS even for the intra-cluster (I just need this to work between PODS since it gives a nice abstraction) ? If So, any idea on whats recommended for Kubernetes on Ubuntu Cluster that is not on GCE ?

[thockin]

The IP range isn't likely to be a real problem, but it is a routable
range, so if you ever find a real site with an 11. prefix that you want to
hit, you'll be debugging for weeks :)

If any of our docs say to use 11. they are wrong and should be fixed.

For DNS: Our DNS service is exactly for intracluster use.

https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/dns

Start with the README - it covers what to run and how, and has template pod
and replication controller configs for you.

On Fri, Apr 10, 2015 at 1:00 PM, sacashgit [email protected] wrote:

@thockin https://github.com/thockin - Thanks for the quick note.

IP Range : I dont own it :). I just didnt change any of that when I built
the binaries and kept the config files as is to large extent. I can take
care of that.

Name resolution: So I do need to run the DNS even for the intra-cluster (I
just need this to work between PODS since it gives a nice abstraction) ? If
So, any idea on whats recommended for Kubernetes on Ubuntu Cluster that is
not on GCE ?

—
Reply to this email directly or view it on GitHub
#6667 (comment)
.

[sacashgit]

@thockin - Thanks..

DNS : I read quite a bit from last 2 days on the above link but what I am suspecting is, this is not supported on Ubuntu cluster. The reason being, I dont see this configuration option on the bits I have for ubunut-cluster.

am I missing some thing ? { I hope to hear Yes and able to use it to resolve the issue}. will look deeper in the mean time

On the Portal IP Base address (11.*) , I think it is part of the GuestBook example.

[thockin]

It looks like the ubuntu-cluster does not start DNS. You can start it
manually.

Follow the "If you are not using a supported cluster setup" part of
https://github.com/GoogleCloudPlatform/kubernetes/tree/master/cluster/addons/dns/README.md

I don't find any examples that use an 11.* for portals?

On Fri, Apr 10, 2015 at 2:17 PM, sacashgit [email protected] wrote:

@thockin https://github.com/thockin - Thanks..

DNS : I read quite a bit from last 2 days on the above link but what I am
suspecting is, this is not supported on Ubuntu cluster. The reason being, I
dont see this configuration option on the bits I have for ubunut-cluster.
[image: screen shot 2015-04-10 at 2 05 31 pm]
https://cloud.githubusercontent.com/assets/6346188/7097509/fd7c4a04-df8a-11e4-953a-8bb8f971870d.png

am I missing some thing ? { I hope to hear Yes and able to use it to
resolve the issue}. will look deeper in the mean time

On the Portal IP Base address (11.*) , I think it is part of the GuestBook
example.

—
Reply to this email directly or view it on GitHub
#6667 (comment)
.

[sacashgit]

@thockin : Thanks for pointing the process around launching DNS in Ubuntu Cluster. I will try the manual steps and report back the findings..

Regarding the Portal IP (11.1.*) : I think I found some thing that should help :). As I mentioned, I did not add any thing new on my own but just followed the steps to get the cluster up and running. I indeed find a reference to portal IP 11.1 series in default_scripts directory for API_SERVER. Check it out. Hope this helps.

https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/ubuntu-cluster/default_scripts/kube-apiserver

[WIZARD-CXY]

@sacashgit I am the author of this deployment approach.Sorry for the late. I learned this portal net configuration from the single-node ubuntu deployment and it works fine in my lab.I will make it configurable.In fact I am working on it #5498. Once this pr is merged you can configure it as you want

[thockin]

This is fixed now, yes? Please reopen if not.

[medhedifour]

@sacashgit
Is this problem fixed or not ?
I have the same problem in my ubuntu cluster,
services are not working.

[sacashgit]

Yes - problem fixed, after installing the SkyDNS service on Ubuntu Cluster. All Good now.