Openshift: delete dnsmasq procces [THREESCALE-1555]

[eloycoto]

DNSmasq process was not handled by Apicast at all, dnsmasq process can
die and
will never be up, so dns queries will start to fail.

The reasons to have APICast were:

Openshift cluster provided multiple DNS servers in the resolv.conf:
not anymore, currently is using a Kubernetes service with static IP, so
all connections goes to the same service.
DNS service was block during pod updates, no longer a case, it's
working correctly, added more pods in to a service and do not have the
case.
Cache was not used, no longer needed due to resolver.search_dns is
looking for the full query in the cache before make the dns request to
the server. [0]

[0]

APIcast/gateway/src/resty/resolver.lua

Lines 282 to 302 in 41ad2c2

	local function search_dns(self, qname, stale)
	local search = self.search
	local dns = self.dns
	local options = self.options
	local cache = self.cache
	local answers, err
	for i=1, #search do
	local query = qname .. '.' .. search[i]
	ngx.log(ngx.DEBUG, 'resolver query: ', qname, ' search: ', search[i], ' query: ', query)
	answers, err = cache:get(query, stale)
	if valid_answers(answers) then break end
	answers, err = dns:query(query, options)
	if valid_answers(answers) then
	cache:save(answers)
	break
	end
	end

Fix THREESCALE-1555

Signed-off-by: Eloy Coto [email protected]

[eloycoto]

Consider this work WIP:

1. Need to test this on a real cluster: OP 4 and OP 3.11
2. Need to be checked if dnsmasq is still needed on OP4.

[eloycoto]

Here are the result on OP 4 series:

Traffic flow:
apicast-client -> ApicastService(172.30.136.[123]0)-> Apicast Pod -> Apicast-server

Using local address: server-service.3scalegateway.svc.cluster.local
Using external address: test-wrk.acalustra.com

DNS server as a sidecar container in the POD:

Using local address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.10:8080/?user_key=123";
Running 2m test @ http://172.30.136.10:8080/?user_key=123
  20 threads and 100 connections

  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    99.41ms   29.22ms 364.87ms   82.76%
    Req/Sec    50.76     14.83   101.00     72.14%
  101108 requests in 1.67m, 13.59MB read
Requests/sec:   1010.14
Transfer/sec:    139.04KB

Using external address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.10:8080/?user_key=123";
Running 2m test @ http://172.30.136.10:8080/?user_key=123
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   104.11ms   28.50ms 367.03ms   81.01%
    Req/Sec    48.37     13.77   101.00     75.56%
  96345 requests in 1.67m, 12.95MB read
Requests/sec:    962.53
Transfer/sec:    132.49KB

DNS server inside APICast container

Using local DNS address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.20:8080/?user_key=123";
Running 2m test @ http://172.30.136.20:8080/?user_key=123
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   107.50ms   35.60ms 513.84ms   85.82%
    Req/Sec    47.31     14.61   101.00     73.35%
  93862 requests in 1.67m, 12.62MB read
Requests/sec:    937.65
Transfer/sec:    129.07KB

Using external address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.20:8080/?user_key=123";
Running 2m test @ http://172.30.136.20:8080/?user_key=123
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   120.59ms   41.67ms 648.06ms   84.06%
    Req/Sec    42.35     13.03   100.00     72.82%
  83826 requests in 1.67m, 11.27MB read
Requests/sec:    837.31
Transfer/sec:    115.26KB

Without DNS Server and without RESOLVER env variable.

Using local DNS address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.30:8080/?user_key=123";
Running 2m test @ http://172.30.136.30:8080/?user_key=123
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   117.59ms   35.15ms 546.93ms   79.84%
    Req/Sec    42.99     12.37    99.00     68.72%
  85537 requests in 1.67m, 11.50MB read
Requests/sec:    854.52
Transfer/sec:    117.62KB

Using external address:

oc exec -ti apicast-client -- wrk -d 100 -c 100 -t 20  -H "Host: one" "http://172.30.136.30:8080/?user_key=123";
Running 2m test @ http://172.30.136.30:8080/?user_key=123
  20 threads and 100 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   103.65ms   31.32ms 375.85ms   84.15%
    Req/Sec    48.83     14.70   110.00     73.11%
  97039 requests in 1.67m, 13.04MB read
Requests/sec:    969.43
Transfer/sec:    133.44KB

Openshift deploy config:

kind: ConfigMap
apiVersion: v1
metadata:
  name: apicast-config
data:
  # BACKEND_ENDPOINT_OVERRIDE: https://echo-api.3scale.net:443
  BACKEND_ENDPOINT_OVERRIDE: http://172.30.136.11/
  THREESCALE_CONFIG_FILE: /gateway/test.json
  RESOLVER: 127.0.0.1#5300
  test.json: |-
    {
      "services": [
        {
          "id": 200,
          "backend_version":  1,
          "backend_authentication_type": "service_token",
          "backend_authentication_value": "token-value",
          "proxy": {
            "hosts": [
              "one"
            ],
            "api_backend": "http://test-wrk.acalustra.com",
            "proxy_rules": [
              {
                "pattern": "/",
                "http_method": "GET",
                "metric_system_name": "hits",
                "delta": 1
              }
            ],
            "policy_chain": [
              {
                "name": "apicast.policy.apicast"
              }
            ]
          }
        }
      ]
    }
---
apiVersion: v1
kind: Service
metadata:
  name: apicast-service
spec:
  selector:
    app: apicast
  clusterIP: 172.30.136.10
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
---
kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
  name: "apicast"
spec:
  template:
    metadata:
      labels:
        name: "apicast"
        app: apicast
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: gateway
          # image: quay.io/3scale/apicast:master
          image: eloycoto/apicast
          imagePullPolicy: Always
          ports:
            - name: proxy
              port: 8080
              protocol: TCP
              containerPort: 8080
            - name: management
              port: 8090
              protocol: TCP
              containerPort: 8090
          env:
            - name: BACKEND_ENDPOINT_OVERRIDE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: BACKEND_ENDPOINT_OVERRIDE
            - name: THREESCALE_CONFIG_FILE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: THREESCALE_CONFIG_FILE
            - name: RESOLVER
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: RESOLVER
          volumeMounts:
            - name: config-volume
              mountPath: /gateway
        - name: dns
          image: quay.io/3scale/apicast:master
          imagePullPolicy: Always
          ports:
            - name: dns
              port: 5300
              containerPort: 5300
          # command: [ "sleep" ]
          # args:
          #   - "1000h"
          command: [ "dnsmasq" ]
          args:
          - "-d"
          - "--listen-address=127.0.0.1"
          - "--port=5300"
          - "--all-servers"
          - "--no-host"
          - "--no-hosts"
          - "--cache-size=1000"
          - "--no-negcache"
          - "--domain-needed"
      volumes:
        - name: config-volume
          configMap:
            name: apicast-config
  replicas: 1
  strategy:
    type: "Rolling"
---
apiVersion: v1
kind: Pod
metadata:
  name: apicast-client
  labels:
    app: apicast-client
spec:
  terminationGracePeriodSeconds: 0
  containers:
  - name: client
    image: williamyeh/wrk
    # image: docker.io/cilium/demo-client
    imagePullPolicy: IfNotPresent
    command: [ "sleep" ]
    args:
    - "1000h"
---
apiVersion: v1
kind: Pod
metadata:
  name: apicast-api-server
  labels:
    app: apicast-api-server
spec:
  terminationGracePeriodSeconds: 0
  containers:
  - name: server
    image: eexit/mirror-http-server
    imagePullPolicy: IfNotPresent
    ports:
    - name: dns
      port: 80
      containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: server-service
spec:
  selector:
    app: apicast-api-server
  clusterIP: 172.30.136.11
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: apicast-service-local-resolver
spec:
  selector:
    app: apicast-local-resolver
  clusterIP: 172.30.136.20
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
---
kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
  name: "apicast-local-resolver"
spec:
  template:
    metadata:
      labels:
        name: "apicast-local-resolver"
        app: apicast-local-resolver
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: gateway
          image: quay.io/3scale/apicast:master
          imagePullPolicy: Always
          ports:
            - name: proxy
              port: 8080
              protocol: TCP
              containerPort: 8080
            - name: management
              port: 8090
              protocol: TCP
              containerPort: 8090
          env:
            - name: BACKEND_ENDPOINT_OVERRIDE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: BACKEND_ENDPOINT_OVERRIDE
            - name: THREESCALE_CONFIG_FILE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: THREESCALE_CONFIG_FILE
          volumeMounts:
            - name: config-volume
              mountPath: /gateway
      volumes:
        - name: config-volume
          configMap:
            name: apicast-config
  replicas: 1
  strategy:
    type: "Rolling"
---
kind: "DeploymentConfig"
apiVersion: "v1"
metadata:
  name: "apicast-no-dnsserver"
spec:
  template:
    metadata:
      labels:
        name: "apicast-no-dnsserver"
        app: apicast-no-dnsserver
    spec:
      terminationGracePeriodSeconds: 0
      containers:
        - name: gateway
          # image: quay.io/3scale/apicast:master
          image: eloycoto/apicast
          imagePullPolicy: Always
          ports:
            - name: proxy
              port: 8080
              protocol: TCP
              containerPort: 8080
            - name: management
              port: 8090
              protocol: TCP
              containerPort: 8090
          env:
            - name: BACKEND_ENDPOINT_OVERRIDE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: BACKEND_ENDPOINT_OVERRIDE
            - name: THREESCALE_CONFIG_FILE
              valueFrom:
                configMapKeyRef:
                  name: apicast-config
                  key: THREESCALE_CONFIG_FILE
          volumeMounts:
            - name: config-volume
              mountPath: /gateway
      volumes:
        - name: config-volume
          configMap:
            name: apicast-config
  replicas: 1
  strategy:
    type: "Rolling"
---
apiVersion: v1
kind: Service
metadata:
  name: apicast-service-no-dnsserver
spec:
  selector:
    app: apicast-no-dnsserver
  clusterIP: 172.30.136.30
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080

[mikz]

The main reason we had dnsmasq was to cope with unreliable DNS server in the cluster. That happend on deploys or node reallocations when services / pods moved around the cluster. Also the time (20s) might not be enough to do a second resolve during the test. It is necessary to have several resolvs in one test to see if there is any impact.

[eloycoto]

Ok,

The main reason we had dnsmasq was to cope with unreliable DNS server in the cluster. That happend on deploys or node reallocations when services / pods moved around the cluster.

What is a good test then? running wrk and kill all DNS servers in the cluster? Is that something that Apicast should be aware of?

Also the time (20s) might not be enough to do a second resolve during the test.

The -t option in the wrk is the number of threads, not the timeout.

It is necessary to have several resolvs in one test to see if there is any impact.

Kubernetes already fix this using services, no? Resolv.conf is always pointing to the right DNS server:

localhost.localdomain ➜  APIcast git:(master) ✗ oc exec -ti apicast-local-resolver-1-tscw7 -- cat /etc/resolv.conf
search eloy-apicast-test.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal
nameserver 172.30.0.10
options ndots:5
localhost.localdomain ➜  APIcast git:(master) ✗ oc get services --all-namespaces | grep 172.30.0.10
openshift-dns                                           dns-default                               ClusterIP      172.30.0.10      <none>                                                                       53/UDP,53/TCP,9153/TCP       42d
localhost.localdomain ➜  APIcast git:(master) ✗ oc get service -n openshift-dns -o yaml
apiVersion: v1
items:
- apiVersion: v1
  kind: Service
  metadata:
    creationTimestamp: 2019-06-12T09:44:59Z
    labels:
      dns.operator.openshift.io/owning-dns: default
    name: dns-default
    namespace: openshift-dns
    ownerReferences:
    - apiVersion: operator.openshift.io/v1
      controller: true
      kind: DNS
      name: default
      uid: be3202a5-8cf6-11e9-92ef-02df15259600
    resourceVersion: "2016"
    selfLink: /api/v1/namespaces/openshift-dns/services/dns-default
    uid: be667aaf-8cf6-11e9-92ef-02df15259600
  spec:
    clusterIP: 172.30.0.10
    ports:
    - name: dns
      port: 53
      protocol: UDP
      targetPort: dns
    - name: dns-tcp
      port: 53
      protocol: TCP
      targetPort: dns-tcp
    - name: metrics
      port: 9153
      protocol: TCP
      targetPort: metrics
    selector:
      dns.operator.openshift.io/daemonset-dns: default
    sessionAffinity: None
    type: ClusterIP
  status:
    loadBalancer: {}
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
localhost.localdomain ➜  APIcast git:(master) ✗

[mikz]

A good test is running some deployment in the cluster. This used to cause timeouts on one of the two resolvers. But if /etc/resolv.conf has just one server now, then it is not necessary. It used to have two.

I see. The test runs for 100 seconds. What is the DNS reply TTL then?

I meant "resolvs" in a sense of making multiple DNS queries over the course of the test.

[eloycoto]

Hi,

For the track, I had a chat with Michal, and some issues are still present, the
big one that it's to have multiple resolver entries in the resolv.conf is not
longer a issues, (Validated on OP 3.11 and 4.X)

There are still two pending issues:

Resty cache queries:

If the API backend is set as the apicast-api-server the cache will never save
the value at all, and the second request will always ask to the upstream
resolver, relevant logs:

2019/07/24 14:57:02 [debug] 28#28: *9 [lua] resolver.lua:294: search_dns(): resolver query: apicast-api-server search: 3scalegateway.svc.cluster.local query: apicast-api-server.3scalegateway.svc.cluster.local
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] cache.lua:122: fetch_answers(): resolver cache miss apicast-api-server.3scalegateway.svc.cluster.local
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] cache.lua:188: get(): resolver cache miss: apicast-api-server.3scalegateway.svc.cluster.local
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] dns_client.lua:68: query(): resolver query: apicast-api-server.3scalegateway.svc.cluster.local nameserver: 127.0.0.1:5353
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] cache.lua:74: store(): resolver cache write apicast-api-server.3scalegateway.svc.cluster.local with TLL 30
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] resolver.lua:348: lookup(): resolver query: apicast-api-server finished with 1 answers
2019/07/24 14:57:02 [debug] 28#28: *9 [lua] resolver.lua:388: get_servers(): query for apicast-api-server finished with 1 answers

This can be fixed without too much trouble, we have the query and the search, so
we can fetch that info from the cache.

DNS server hangs during service updates:

I tested that, increase the deploy to more replicas and no timeout at all, so I think that is already fixed.

[eloycoto]

Hi @mikz

I've just checked this, the query can be to apicast-api-server and the DNS cache will be used.

The query will hit the search_dns as apicast-api-server

APIcast/gateway/src/resty/resolver.lua

Lines 323 to 344 in 41ad2c2

	function _M.lookup(self, qname, stale)
	local cache = self.cache
	ngx.log(ngx.DEBUG, 'resolver query: ', qname)
	local answers, err
	if is_ip(qname) then
	ngx.log(ngx.DEBUG, 'host is ip address: ', qname)
	answers = { new_answer(qname) }
	else
	if is_fqdn(qname) then
	answers, err = cache:get(qname, stale)
	else
	answers, err = resolve_upstream(qname)
	end
	if not valid_answers(answers) then
	answers, err = search_dns(self, qname, stale)
	end
	end

And in the search_dns will loop over the query strings, and if the full query
(qname + search) is in there, the cache key will be used if it's in there:

APIcast/gateway/src/resty/resolver.lua

Lines 282 to 302 in 41ad2c2

	local function search_dns(self, qname, stale)
	local search = self.search
	local dns = self.dns
	local options = self.options
	local cache = self.cache
	local answers, err
	for i=1, #search do
	local query = qname .. '.' .. search[i]
	ngx.log(ngx.DEBUG, 'resolver query: ', qname, ' search: ', search[i], ' query: ', query)
	answers, err = cache:get(query, stale)
	if valid_answers(answers) then break end
	answers, err = dns:query(query, options)
	if valid_answers(answers) then
	cache:save(answers)
	break
	end
	end

As example:

First request

2019/07/26 13:39:05 [debug] 25#25: *10 [lua] resolver.lua:294: search_dns(): resolver query: apicast-api-server search: 3scalegateway.svc.cluster.local query: apicast-api-server.3scalegateway.svc.cluster.local
2019/07/26 13:39:05 [debug] 25#25: *10 [lua] cache.lua:122: fetch_answers(): resolver cache miss apicast-api-server.3scalegateway.svc.cluster.local
2019/07/26 13:39:05 [debug] 25#25: *10 [lua] cache.lua:188: get(): resolver cache miss: apicast-api-server.3scalegateway.svc.cluster.local
2019/07/26 13:39:05 [debug] 25#25: *10 [lua] dns_client.lua:68: query(): resolver query: apicast-api-server.3scalegateway.svc.cluster.local nameserver: 172.30.0.2:53
2019/07/26 13:39:05 [debug] 25#25: *10 [lua] cache.lua:74: store(): resolver cache write apicast-api-server.3scalegateway.svc.cluster.local with TLL 30

Second request:

2019/07/26 13:39:06 [debug] 25#25: *15 [lua] resolver.lua:294: search_dns(): resolver query: apicast-api-server search: 3scalegateway.svc.cluster.local query: apicast-api-server.3scalegateway.svc.cluster.local
2019/07/26 13:39:06 [debug] 25#25: *15 [lua] cache.lua:115: fetch_answers(): resolver cache read apicast-api-server.3scalegateway.svc.cluster.local 1 entries
2019/07/26 13:39:06 [debug] 25#25: *15 [lua] cache.lua:190: get(): resolver cache hit: apicast-api-server.3scalegateway.svc.cluster.local 172.30.136.11

This is working as expected.

To summarize:

• Issue with multiple resolv.conf is not an issue, due to OP 3.11 and 4.X are now using the DNS service IP.
• Cache will be used, it'll hit twice in the cache, but as happens using dnsmasq.
• Deployments and updates in the API are not longer hold the DNS server, so no timeouts in there.

I do not think that are any other issue, so I think that we can remove dnsmasq
from the container and all will work as expected.

Best regards.

[davidor]

Nice work 🏅

We should change the title and the changelog to avoid confusion, right? Because this PR does not move dnsmasq to another container it just removes it from the APIcast image.