Wrong NodePort when restoring Service

[domcar]

What steps did you take and what happened:
I created a NodePort service (notice the NodePort: 30001)

spec:
  clusterIP: 10.100.171.67
  externalTrafficPolicy: Cluster
  ports:
  - name: api-server
    nodePort: 30001
    port: 30001
    protocol: TCP
    targetPort: 30001
  selector:
    component: kube-apiserver
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

After I did a backup with ark and the .json file looks correct:

{
    "spec": {
    "clusterIP": "10.100.171.67",
    "externalTrafficPolicy": "Cluster",
    "ports": [
      {
        "name": "api-server",
        "nodePort": 30001,
        "port": 30001,
        "protocol": "TCP",
        "targetPort": 30001
      }
    ],
    "selector": {
      "component": "kube-apiserver"
    },
    "sessionAffinity": "None",
    "type": "NodePort"
  },
  "status": {
    "loadBalancer": {}
  }
}

I then deleted the service and restore it with ark: the NodePort was random:

spec:
  clusterIP: 10.100.171.67
  externalTrafficPolicy: Cluster
  ports:
  - name: api-server
    nodePort: 30878
    port: 30001
    protocol: TCP
    targetPort: 30001
  selector:
    component: kube-apiserver
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

I then deleted again the service and restored usign

kubectl apply -f ~/Downloads/resources\ 2/services/namespaces/cke-wunder/external-api-service.json

The port was correct: 30001

What did you expect to happen:
The NodePort should have been the same as when I took the backup

The output of the following commands will help us better understand what's going on:

• kubectl logs deployment/ark -n heptio-ark

time="2018-10-24T07:03:36Z" level=info msg="starting restore" backup=external-svc logSource="pkg/controller/restore_controller.go:430" restore=heptio-ark/external-svc-20181024090335
time="2018-10-24T07:03:36Z" level=info msg="restore completed" backup=external-svc logSource="pkg/controller/restore_controller.go:432" restore=heptio-ark/external-svc-20181024090335

• ark backup logs <backupname>
  No errors nor warnings;
  Phase completed;
• ark restore describe <restorename> or kubectl get restore/<restorename> -n heptio-ark -o yaml
  No errors nor warnings;
  Phase completed;
• ark restore logs <restorename>
  Strange error:

An error occurred: Get http://cke-seed-minio.kube-system.svc.cluster.local:9000/ark/external-svc/restore-external-svc-20181024090335-logs.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=adasdwgrrt%!F(MISSING)20181024%!F(MISSING)us-east-1%!F(MISSING)s3%!F(MISSING)aws4_request&X-Amz-Date=20181024T071348Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=0263c9b468b727aea67629be5d6eaad5006a998b59f961d0df1e1570bf37228d: dial tcp: lookup cke-seed-minio.kube-system.svc.cluster.local on 8.8.8.8:53: no such host

This error is strange, if the minio was not reachable ark could have not restored the service at all.

Environment:

• Ark version (use ark version):

Version: v0.9.8
Git commit: v0.9.8
Git tree state: clean

• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-10T11:44:36Z", GoVersion:"go1.11", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.3", GitCommit:"a4529464e4629c21224b3d52edfe0ea91b072862", GitTreeState:"clean", BuildDate:"2018-09-09T17:53:03Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version:
  Kubeadm
• Cloud provider or hardware configuration:
  kubeadm on vsphere

[ncdc]

@domcar we currently only support preserving node ports at restore time if your service has the annotation kubectl.kubernetes.io/last-applied-configuration filled in (this comes from kubectl apply). We added this logic in #712. The thinking is that if you choose the node port yourself, we'll preserve it. If you didn't pick a node port, and instead let Kubernetes assign a random one, we don't preserve it. This is something that's hard to get right for every use case. If we can improve upon this without breaking other use cases, we'd be happy to do so.

cc @rosskukulinski

[ncdc]

@domcar I meant to also respond to this comment:

This error is strange, if the minio was not reachable ark could have not restored the service at all.

When you run ark backup logs, ark restore logs, and 1 or 2 of the describe commands, the ark command line client creates a DownloadRequest custom resource and sends it to the Kubernetes apiserver. When the ark server sees this request, it generates a pre-signed URL using its access credentials to object storage. The server updates the DownloadRequest with the pre-signed URL. The ark command line client is waiting for the server to make this update, and once it occurs, the client makes an http(s) request to retrieve the pre-signed URL.

If you are using Minio, and you've followed our example deployment and configuration, the s3Url that ark uses to talk to Minio is http://minio.heptio-ark.svc:9000. This is the URL that the ark server uses to communicate with Minio. It is also the URL that is returned to the client for the pre-signed logs URL. If you are running the ark command line client on your laptop/desktop, and you are not able to resolve the "in-cluster" DNS (minio.heptio-ark.svc), you will get the error that you saw (... lookup cke-seed-minio.kube-system.svc.cluster.local on 8.8.8.8:53: no such host).

We'll see if we can add some functionality to allow you to specify a "public" URL that the ark server can use for pre-signed URLs. You would need to set up an ingress or use a NodePort for this, but it would allow the ark server to use in-cluster DNS while allowing the client to use a resolvable URL.

[ncdc]

@domcar I put together a quick PR to try to add this public URL feature - #1006.

[domcar]

@ncdc Thank you a lot for you answers, they are as always very helpful! And thank for the PR, i will try it out.

we currently only support preserving node ports at restore time if your service has the annotation kubectl.kubernetes.io/last-applied-configuration filled in (this comes from kubectl apply). We added this logic in #712. The thinking is that if you choose the node port yourself, we'll preserve it.

Ok, good to know, i'll test it

[ncdc]

@domcar would you consider this issue resolved?

[domcar]

@ncdc Yes, i'll close it now