The Hydra can't recovery the readiness status in k8s

[yxzhm]

Describe the bug

The Hydra can't recovery the readiness status in k8s.
It looks like Hydra can't connect the DB again, after it meet the bad connection status.

Reproducing the bug

Deploy the 5 Hydra Pods in Kubernetes. They are connect one same Azure Mysql DB.
The readiness probe is "/health/ready".
After waiting for a while(every few minutes or one night), the status of some Pods is not ready.
We find that it can't be recovered automatically.
The DB is available, since other Pods work fine. Not all Pods down at same time. You can refer to the below illustration.

The K8S readiness response is
{"errors":{"database":"bad connection"}}

Expected behavior

The Hydra Pod can be ready status again, due to the DB is available.

Environment

Hydra: v1.0.0-rc15
Azure DB version: 5.7

Add any other context about the problem here.

[aeneasr]

Thank you for creating an issue in this repository. Unfortunately, your issue does not comply with community guidelines by not using one of the provided issue templates. Those templates help us better understand your issue and reduce latency as well as communication overhead. Please update your issue in accordance with the issue template that fits best or this issue will be auto-closed in a 3 days.

[aeneasr]

Your issue is a duplicate of #1460 and #1463 which have been resolved on master but not yet released.

[leonfancy]

Your issue is a duplicate of #1460 and #1463 which have been resolved on master but not yet released.

I don't think it's the same issue as #1460. This problem is that application starts correctly and /health/ready does response 200 at the begining, but it starts return that error after a random time (maybe few minutes or hours).

Even weired thing is that only /health/ready returns error, however other API endpoints is still working. Here is some tests on the APIs:

root@localhost# curl  http://10.244.72.160:4444/.well-known/jwks.json
{"keys":[{"use":"sig","kty":"RSA","kid":"public:b76e6ae7-dc0d-470e-9ecf-9ee03e0d61c3","alg":"RS256","n":"zk5JxI95KPdt1LJar4J7j673wVmprQ5XT37u0wD25EapJJp25I1A_xu8kS17NsMDpIX9IFfPgCMYNL0jaHmY9AWk3lUpAPMaRcAaCd7jBTQ6KUWYD5Wfi_BHtnSfmdRZ12UxnfBGZ-fHaFpgxU9MRlWSu3C-07R0RBERyd5teg2SwVErDkPQwSSgaKVywrAcED0528S4kh12qYcIc9wJbuxVTWyYzWgVUpTqqy__6rSmcXZrgRHKxpLcne8EQmTKKl3yLMY10K7QcpFUsEyHoGhAT4XdJP5EjCYBl5qxefpc4lGltMY6YdN_xmtyLVgOPsifAw7HmMXoUKHVCqBUH9SiifKGs7ii7yYDYTCbwu-c1zytqb_JMCYaDi1ISwTT09ZK4iVvO0DOg6JGGeQTNzGEeBcTPQA79gRopS9ba3x_pcjHKXsfcKqrrWp4EB3M6HRT70WshTVVdhhscy0DszKgmPKe6GtBkxpk8P9evZHrMt2wbJrRR39-TZ456VwQPtQEoN0wyrOjAOLRkOEokDZDd65U52wztJH9Ewh6wxNPikV2pJ4_jsORgryQ7GrseT0HE7XhkTxg2N6j74ek2MdfZN1r9bAsE0aaaF-uUJkK8qIPtjJmkBmj-3uTT5Ac2JmxHAvc1ZV5RTYDwL-6dqweYEPMKK1_I0UpSrzNTGU","e":"AQAB"},{"use":"sig","kty":"RSA","kid":"public:a78a0754-7134-4b46-add1-a83a936070e5","alg":"RS256","n":"ziuAn5lSBQQJkwtBDpgdhomcvkAEbeIfCk5YmD51hZrZ5ksOajxSPQn7HhVfT3kAV3zLwR9gmLYMpWSbDRXeYZS6d-3ES9yCfjc8OKHDeIw6r5anow4tGiYAhVG5NbbXIByAkp_fDHt-qzkqtJ499aschR8tcAHMj3wizq-9wJExGM0k8cIExBZMeN1M7Raa3gie9MNklCuLzbME0iu5dJtkrIzyHyQSH-_TEZHBoDzaBjzAT3d-lAlR7cnFlMJbs4_FnNF6f6E6KJq2NSVC03y9tX4YaRgI3U_2Y_2PCnMWU3fvKhBCoNrL4ufS9N4OCHCEBHrcioBdesxQp948VQdMvUFDDcXU4_10vB19-eIikNuBAebBhjWnLWwEU2k_SJhwqeVwmiKwupskbTqu8myBvdTaLWlWqCb5TH6mU75KcLKyclu3t4Ke65N0HkfXg7XdLYVf92OjRP8P2mFXG-q7PTZ8X_30PLcHPhetUICI-JmS0ceSuM7LZ1-LN1hAAN0eNlDgizL9wY1rp-T83qic7TxZBXGNC8p1_pOYzEhP2PrO_Rq2dlxyQ2M8fCfJezCsP0dmZ4yunDfwpuPBmK87IZnHMGzkaFOI4D3FuOiE7EmGoE5l7iNZLkluNjqMR0EkJj-qSGaUrS_JPZ5Qg__B8PFr2xhfmp1_9YR4KTU","e":"AQAB"}]}
root@localhost# curl  http://10.244.72.160:4445/health/ready
{"errors":{"database":"bad connection"}}

[Renkas]

Having the same issue with MariaDB.

I dont get it why would you close the issue with mentions of issues that are not relateted to this one?

I'm using Hydra v1.0.0

[aeneasr]

Please provide more context, at bare minimum the configuration. It's best to create a new issue for that. Please follow the issue template.

Please also note that MariaDB is not supported.