[ISSUE] Pods Randomly Going Down During Test Runs

[slolatte]

Describe the issue:

The component pods are going down randomly because of the nature of the GKE nodes. It is causing most of the UI tests to fail during the test run of the cross component test suite. This is causing the test suite to become unreliable and flaky which is a problem because this test suite is used for nightly regression testing of the Helm Chart with SNAPSHOT component versions and as well used in release testing.

Actual behavior:

The pods randomly go down which result in a '503 Service Temporarily Unavailable' error appearing in the UI.

Expected behavior:

The pods should not go down and the UI should appear as normal.

How to reproduce:

Run the nightly SM Helm cross-component test suite test run

Logs:

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform:
• Helm CLI version:
• Chart version:
• Values file:

(for environment see details above)

[markfarkas-camunda]

After the mentioned 503 errors, when the login screen is available again providing the correct credentials results in showing an error message for a brief moment saying: "your session expired", After that half of a second it redirects back to the login page again.

[slolatte]

I am also seeing this error randomly now too

[slolatte]

@aabouzaid Is there any update on when this might be fixed?

[aabouzaid]

This will be fixed in: https://github.com/camunda/distribution/issues/293

[slolatte]

@aabouzaid When performing release testing I encountered 404 errors sometimes...could this be related?

See the following Slack thread for more details.