Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Jest Integration Tests.src/core/server/elasticsearch/integration_tests - fake elasticsearch should return unknown product when it cannot perform the Product check (503 response) #129754

Closed
kibanamachine opened this issue Apr 7, 2022 · 8 comments · Fixed by #148616 or #190158
Closed

Failing test: Jest Integration Tests.src/core/server/elasticsearch/integration_tests - fake elasticsearch should return unknown product when it cannot perform the Product check (503 response) #129754

kibanamachine opened this issue Apr 7, 2022 · 8 comments · Fixed by #148616 or #190158
Assignees
@gsoldevila
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Apr 7, 2022 •
edited
Loading

A test failed on a tracked branch

Error: expect(received).toBe(expected) // Object.is equality

Expected: "critical"
Received: "unavailable"
    at Object.<anonymous> (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/src/core/server/elasticsearch/integration_tests/client.test.ts:94:44)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at _callCircusTest (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/run.js:212:5)
    at _runTest (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/var/lib/buildkite-agent/builds/kb-n2-4-spot-bc5f5d6543db3d55/elastic/kibana-on-merge/kibana/node_modules/jest-runner/build/runTest.js:472:34)

First failure: CI Build - main
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Apr 7, 2022
@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 7, 2022
@kibanamachine kibanamachine added the Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc label Apr 7, 2022
@botelastic botelastic bot removed the needs-team Issues missing a team label label Apr 7, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-core (Team:Core)

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 8.2

@spalger
Copy link
Contributor

spalger commented Apr 15, 2022 •
edited
Loading

Failed 5 PRs this week

image

8.3/main: fc7d385
8.2: 72501b9

spalger pushed a commit that referenced this issue Apr 15, 2022
skip flaky suite (#129754)
fc7d385
spalger pushed a commit that referenced this issue Apr 15, 2022
skip flaky suite (#129754)
72501b9 
(cherry picked from commit fc7d385)
@pgayvallet
Copy link
Contributor

This unavailable status is coming from here:

kibana/src/core/server/elasticsearch/status.ts

Lines 20 to 27 in 5edf7e2

of({
level: ServiceStatusLevels.unavailable,
summary: `Waiting for Elasticsearch`,
meta: {
warningNodes: [],
incompatibleNodes: [],
},
}),

Which means that for some reason, esNodesCompatibility$ did not emit yet, or, more likely, the emission was not propagated to the observable used by the /api/status endpoint when we're reaching this line:

kibana/src/core/server/elasticsearch/integration_tests/client.test.ts

Line 94 in fc7d385

const resp = await supertest(kibanaHttpServer).get('/api/status').expect(503);

Looking at the code, we have some debounces in the observable chain, e.g this one:

kibana/src/core/server/status/status_service.ts

Lines 72 to 74 in 9d5aca5

this.overall$ = combineLatest([core$, this.pluginsStatus.getAll$()]).pipe(
// Prevent many emissions at once from dependency status resolution from making this too noisy
debounceTime(80),

and we're using a combineLatest operator for the /status/api route:

kibana/src/core/server/status/routes/status.ts

Lines 64 to 66 in 9d5aca5

combineLatest([status.overall$, status.coreOverall$, status.core$, status.plugins$]).subscribe(
combinedStatus$
);

Given we're using a local 'fake' service and the overall quick time it takes for core to complete the setup phase, this looks like a simple race issue.

kertal pushed a commit to kertal/kibana that referenced this issue May 24, 2022
@kertal
skip flaky suite (elastic#129754)
f6ffc19
@gsoldevila gsoldevila mentioned this issue Jan 10, 2023
Merged
gsoldevila added a commit that referenced this issue Jan 11, 2023
@gsoldevila @kibanamachine
Fix unknown product test flaky behavior (#148616)
9f39b78 
Fixes #129754

The failing integration test is querying the `/api/status` endpoint
before Kibana and ES are completely initialized, and thus, it is getting
a _unavailable: Waiting for Elasticsearch_ instead of the expected
_critical_.

The proposed fix consists in awaiting a few milliseconds, in order to
give the `debounceTime` operators enough time to propagate the right
status.

Co-authored-by: kibanamachine <[email protected]>
jennypavlova pushed a commit to jennypavlova/kibana that referenced this issue Jan 13, 2023
@gsoldevila @kibanamachine @jennypavlova
Fix unknown product test flaky behavior (elastic#148616)
948be95 
Fixes elastic#129754

The failing integration test is querying the `/api/status` endpoint
before Kibana and ES are completely initialized, and thus, it is getting
a _unavailable: Waiting for Elasticsearch_ instead of the expected
_critical_.

The proposed fix consists in awaiting a few milliseconds, in order to
give the `debounceTime` operators enough time to propagate the right
status.

Co-authored-by: kibanamachine <[email protected]>
@kibanamachine
Copy link
Contributor Author

New failure: kibana-7-dot-17-es-8-dot-14-forward-compatibility - 7.17

@kibanamachine kibanamachine reopened this May 11, 2024
@afharo
Copy link
Member

afharo commented Aug 8, 2024

@gsoldevila, do you think that we can backport #148616 to 7.17? It looks like it's failing in that branch.

gsoldevila added a commit to gsoldevila/kibana that referenced this issue Aug 8, 2024
@gsoldevila
Fix unknown product test flaky behavior (elastic#148616)
ece815c 
Fixes elastic#129754

The failing integration test is querying the `/api/status` endpoint
before Kibana and ES are completely initialized, and thus, it is getting
a _unavailable: Waiting for Elasticsearch_ instead of the expected
_critical_.

The proposed fix consists in awaiting a few milliseconds, in order to
give the `debounceTime` operators enough time to propagate the right
status.

Co-authored-by: kibanamachine <[email protected]>
(cherry picked from commit 9f39b78)
@gsoldevila
Copy link
Contributor

In progress #190158

@afharo afharo linked a pull request Aug 8, 2024 that will close this issue
[7.17] Fix unknown product test flaky behavior (#148616) #190158
Merged
gsoldevila referenced this issue Aug 8, 2024
@gsoldevila
[7.17] Fix unknown product test flaky behavior (#148616) (#190158)
68492d5 
# Backport

This will backport the following commits from `main` to `7.17`:
- [Fix unknown product test flaky behavior
(#148616)](#148616)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Gerard
Soldevila","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-01-11T14:20:37Z","message":"Fix
unknown product test flaky behavior (#148616)\n\nFixes
https://github.com/elastic/kibana/issues/129754\r\n\r\nThe failing
integration test is querying the `/api/status` endpoint\r\nbefore Kibana
and ES are completely initialized, and thus, it is getting\r\na
_unavailable: Waiting for Elasticsearch_ instead of the
expected\r\n_critical_.\r\n\r\nThe proposed fix consists in awaiting a
few milliseconds, in order to\r\ngive the `debounceTime` operators
enough time to propagate the right\r\nstatus.\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"9f39b785620fbc42e30aea774e875bb3048eb2ae","branchLabelMapping":{"^v8.7.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Team:Core","release_note:skip","test-failure-flaky","backport:skip","v8.7.0"],"number":148616,"url":"https://github.com/elastic/kibana/pull/148616","mergeCommit":{"message":"Fix
unknown product test flaky behavior (#148616)\n\nFixes
https://github.com/elastic/kibana/issues/129754\r\n\r\nThe failing
integration test is querying the `/api/status` endpoint\r\nbefore Kibana
and ES are completely initialized, and thus, it is getting\r\na
_unavailable: Waiting for Elasticsearch_ instead of the
expected\r\n_critical_.\r\n\r\nThe proposed fix consists in awaiting a
few milliseconds, in order to\r\ngive the `debounceTime` operators
enough time to propagate the right\r\nstatus.\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"9f39b785620fbc42e30aea774e875bb3048eb2ae"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.7.0","labelRegex":"^v8.7.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/148616","number":148616,"mergeCommit":{"message":"Fix
unknown product test flaky behavior (#148616)\n\nFixes
https://github.com/elastic/kibana/issues/129754\r\n\r\nThe failing
integration test is querying the `/api/status` endpoint\r\nbefore Kibana
and ES are completely initialized, and thus, it is getting\r\na
_unavailable: Waiting for Elasticsearch_ instead of the
expected\r\n_critical_.\r\n\r\nThe proposed fix consists in awaiting a
few milliseconds, in order to\r\ngive the `debounceTime` operators
enough time to propagate the right\r\nstatus.\r\n\r\nCo-authored-by:
kibanamachine
<[email protected]>","sha":"9f39b785620fbc42e30aea774e875bb3048eb2ae"}}]}]
BACKPORT-->
@afharo
Copy link
Member

afharo commented Aug 8, 2024

Backport merged (thanks @gsoldevila)! Closing this issue.

@afharo afharo closed this as completed Aug 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@gsoldevila gsoldevila

Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team:Core Core services & architecture: plugins, logging, config, saved objects, http, ES client, i18n, etc
Projects
None yet
Milestone
No milestone
6 participants
@spalger @pgayvallet @afharo @elasticmachine @gsoldevila @kibanamachine