Wait For Dataplane Ready takes too long and times out at 3 minutes #7849
Labels
Comments
cbugneac-nex
added
kind/bug
|
jakubdyszkiewicz
added
triage/needs-information
|
The control plane is up and running: $ kubectl -n kuma-system get pods
NAME READY STATUS RESTARTS AGE
kuma-control-plane-6899546ff5-7k95r 1/1 Running 0 101m
kuma-control-plane-6899546ff5-wj9mj 1/1 Running 0 101m
$ kubectl -n kuma-system get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kuma-control-plane ClusterIP 172.20.171.160 <none> 5680/TCP,5681/TCP,5682/TCP,443/TCP,5676/TCP,5678/TCP 74dWell, I cannot get the logs as container is not started: Waiting a bit and then I get: and after a while: |
|
Here is the logs of |
|
Hi @jakubdyszkiewicz here is the log of DP side-car for a test pod: From the logs seems that DP cannot fetch the config from CP. |
|
As we confirmed in Slack the issue here is using |
lobkovilya
added
triage/accepted
|
Thanks @lobkovilya for help. Can I suggest to add a proper logging message as at the moment from the above logs it's not clear and difficult to guess that the root cause is IPv6 loopback [::1] address ? |
|
Hi @lobkovilya some thoughts:
Does the above make sense ? |
6 tasks
|
@lobkovilya Please can this issue be re-opened as even after update to 2.4.2 and setting in Helm values this configuration it still doesn't work: envVars:
KUMA_BOOTSTRAP_SERVER_PARAMS_ADMIN_ADDRESS: "::1"
KUMA_RUNTIME_KUBERNETES_INJECTOR_SIDECAR_CONTAINER_WAIT_FOR_DATAPLANE_READY: true |
|
Removing closed state labels due to the issue being reopened. |
|
Hi @lobkovilya , I've tried to disable IPv6 on pods as AWS VPC CNI Add-on in version 1.15.0 has added support for it via setting $ kubectl describe pod ...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 5m46s default-scheduler Successfully assigned default/service-****-dbb8999bd-tv2mq to ip-**-***-***-**.eu-west-1.compute.internal
Warning FailedCreatePodSandBox 5m45s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "c8fb6332ddd05b7bd76fb445eddae4778165d6a3ec0b42754b9dc941281a2cff": plugin type="kuma-cni" name="kuma-cni" failed (add): pod excluded - could not inject rules into namespace: cannot restore ipv6 iptable rules: failed to add IPv6 inbound address: permission denied
Warning FailedCreatePodSandBox 5m44s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "a513afa6ec36d0fc201d436739b5f242f7bcdfbac46c5491770ff050f564488d": plugin type="kuma-cni" name="kuma-cni" failed (add): pod excluded - could not inject rules into namespace: cannot restore ipv6 iptable rules: failed to add IPv6 inbound address: permission deniedInside the kuma-sidecar Pod I see that IPv6 loopback exists: Looks like kuma-sidecar doesn't bind to |
github-actions
bot
added
the
triage/stale
|
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. |
michaelbeaumont
removed
the
triage/stale
github-actions
bot
added
the
triage/stale
|
This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed. |
lukidzi
added
triage/pending
jakubdyszkiewicz
added
triage/needs-reproducing
|
@cbugneac-nex I believe this was causing it #9431 (since you are using CNI). It has been fixed lately and should be released in 2.7. Feel free to reopen if still happens in 2.7 |
lukidzi
added
triage/accepted
lahabana
added
triage/duplicated
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened?
I have enabled the
waitForDataplaneReadyfunctionality and now starting a pod takes ~ 3 minutes due to postStart lifecycle hook timing out.It does restart the pod eventually but waiting 3 minutes is not acceptable. Are there any other changes needed to be done apart for setting in Helm chart ?
The config is pretty generic, just couple for
externalServices.This is the config I have added in Helm chart values file:
Here are the pod events after enabling this feature:
Control plane Helm values file configuration:
It looks strange it does takes so long and eventually times out. Please could someone shed some light why this happens ?
Kuma Mesh version: v2.4.1
AWS EKS cluster version: 1.25.12
Networking: VPC CNI Add-on
The text was updated successfully, but these errors were encountered: