[UI Settings] Regularly Failing Jest Integration Config

[jbudz]

https://buildkite.com/elastic/kibana-on-merge/builds/23746#01848784-7e14-4800-b63a-bfc5905864d7/3656-4619 https://buildkite.com/elastic/kibana-on-merge/builds/23745#01848782-d6ff-42f5-a111-6f2608db2c47/2443-3188

This test config has recently semi-frequently started failing. Tests are passing, but server cleanup is continuing to run after:

Cannot log after tests are done. Did you forget to wait for something async in your test?
Attempted to log "[2022-11-17T16:56:38.105-05:00][ERROR][plugins.fleet] uninstalling synthetics-0.11.3 after error installing: [NoLivingConnectionsError: There are no living connections]".

Related to #141477

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[jbudz]

Skipped

main: 64bae2e
8.6: 9bb90c3

[pgayvallet]

@elastic/fleet This is not the first time some of our tests are forced to be skipped because the cleanup logic of the fleet plugin is flawed. Is there any way we find a stable solution for these problems?

[jbudz]

This is spreading to other test suites - https://buildkite.com/elastic/kibana-on-merge/builds/23795#01848b07-cbfb-40aa-9090-8664fbbe8e7f. We'll leave this for tracking the ui settings skip and open separate issues accordingly, but want to emphasize the priority in the interim.

[nchaulet]

Looks like this is happening because the synthetics plugin launch the synthetics installation from their plugin start method, I can try to take a look how we can fix/change that.

[nchaulet]

After some investigations, it seems the way synthetics install the integration introduce some flakyness.

The issue is happening because the synthetics plugin launch the synthetics integration during their plugin start, because we remotely fetch the integration there is a chance this break especially in a CI environment.

First to be able to choose the best solutions here @shahzad31 @dominiqueclarke maybe you can help me understanding why the synthetics plugin need to be installed during the plugin start?

Potential way to resolve this

Bundle the package in CI

the synthetics integration is a bundled package, this mean if the package is present in the Kibana directory we will not fetch it from the registry but use the local package instead, this will reduce the flakyness the test.

It is possible to bundle package during CI? this step could probably be flaky as we will have to remotely fetch packages.

Lazy install the synthetics package

Synthetics is the only package installed during Kibana startup, I am wondering if there is way to change that, to like it's done in other plugin (endpoint for example) and install the package when the user visit the synthetics plugin, or the first time you create a package policy.

This is in my opinion the best solution as we will remove the flakyness for all the tests that do not need the package installed.

How to reproduce/simulate the issue

It's easy to reproduce the problem by introducing some delay in the ensureInstalledPackage function

like this for example

diff --git a/x-pack/plugins/fleet/server/services/epm/packages/install.ts b/x-pack/plugins/fleet/server/services/epm/packages/install.ts
index d7f67ca1d2a..9c0eaa60407 100644
--- a/x-pack/plugins/fleet/server/services/epm/packages/install.ts
+++ b/x-pack/plugins/fleet/server/services/epm/packages/install.ts
@@ -109,7 +109,7 @@ export async function ensureInstalledPackage(options: {
     force = false,
     spaceId = DEFAULT_SPACE_ID,
   } = options;
-
+  await require('util').promisify(require('child_process').exec)('sleep 120');
   // If pkgVersion isn't specified, find the latest package version
   const pkgKeyProps = pkgVersion
     ? { name: pkgName, version: pkgVersion }

Then run the tests

node ./scripts/jest  --config="src/core/server/integration_tests/ui_settings/jest.integration.config.js" --runInBand --coverage=false --passWithNoTests  src/core/server/integration_tests/ui_settings/index.test.ts

[elasticmachine]

Pinging @elastic/uptime (Team:uptime)

[shahzad31]

We do need synthetics package templates on kibana start. So i think it can't be lazy loaded. We will have to figure out something else.

[shahzad31]

@jbudz @nchaulet is this still an issue? is there a way to check in which builds this is failing?

[jbudz]

It is, the config was skipped in main and 8.6 - #145652 (comment) . There's no recent builds, the links on the top comment are dated but the closest I can think of.

[elasticmachine]

Pinging @elastic/appex-sharedux (Team:SharedUX)

[shahzad31]

I have a plan to fix this. Basically I will disable installation of synthetic package on kibana start in ci generally. And for our E2e tests . We will enable installation via flag.

But this won't make 8.7 so I am pushing this to 8.8.