K8s probes connection refused

[silversoul93]

Hi guys,
often the pods start Crash Looping for a strange error:

Liveness probe failed: Get "http://{{IP_ADDRESS}}:3000/-/healthz": dial tcp {{IP_ADDRESS}}:3000: connect: connection refused

It seems like k8s is not able to reach the service to check the readyness or liveness probe.
The only way to fix it is by restarting the pod from the Runtime section of the Mia-Platform console.

Can you explain me the reason for this behavior?

[davidebianchi]

The server inside the service is not ready to response to the probe, so the connection fails. Maybe the service is under pression, and/or has not many resources available?

[silversoul93]

I don't think this is the problem, it happens with services with very low traffic too, and with the correct amount of resources, enough to handle the high resources consumption at startup time.

And after a restart usually the service starts without problems.

[davidebianchi]

The issue is from the service side, maybe it has some high intensive cpu tasks?

[silversoul93]

Sorry, I forgot this thread 😂
It happens on services without high intensive cpu tasks at startup.
The example I pasted above was about the Mia-Platform authorization-service that doesn't have intensive operations during the startup.

Moreover, the service usually keeps crashlooping until manual refresh. The service restarts lot of times.
It seems like k8s is not able to reach the probes routes, like a network problem.

[fredmaggiowski]

Hey, can you please provide the liveness probe configuration? it seems that the server is taking some time to boot and maybe it's just a matter of better tuning the probe startup delay or the interval between between invocations

[silversoul93]

Yes, it's the default one.

But I'm convinced that this is not the problem.
Every time it happens we solve just pressing the Restart Pod button. We had problems related to the probes configurations too, and a restart never fixed them.

[fredmaggiowski]

To sum it up, this is a problem you randomly face at service startup (particularly with the Authorization Service) and using the default probes settings, right?

I'm still thinking that the problem might be this; consider that the authorization service does a few things at startup and they depend on the size of the project (specifically by the amount of endpoints and variety of user group permissions configured). So with low resources it may take a while; we have never noticed such latencies during boot but this might be the case.

[silversoul93]

To sum it up, this is a problem you randomly face at service startup (particularly with the Authorization Service) and using the default probes settings, right?

Not "particularly", this example is about the Authorization Service, but it happened on other services and we fixed it with a restart.

I understand what you are talking about, but it's not clear to why only a restart fixes the problem.
I'd agree with you if, after some restart, the service started; but in this case only a restart, deleting the pod, fixes the problem.

I'll try to get some other example.

[fredmaggiowski]

I'll try to get some other example.

They would be helpful, thank you

[theliuk]

Have you tried starting the service locally? Does it answer immediately to some calls to GET /-/healthz?

[silversoul93]

Hi, the problem never happened again, if it happens again I'll reopen this issue.
Thanks for the suggestions guys.