Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Functional tests - refactor response code checks #127875

Merged
merged 5 commits into from
Mar 17, 2022
Merged

[ML] Functional tests - refactor response code checks #127875

merged 5 commits into from
Mar 17, 2022

Conversation

pheyos
Copy link
Member

@pheyos pheyos commented Mar 16, 2022 •
edited
Loading

Summary

This PR refactors the way how we assert response codes from API requests in our functional tests.

Details

  • Before, a response code check failure would log something like the following: 
    Error: expected 400 "Bad Request", got 500 "Internal Server Error"
  • This makes it hard to debug as no details are printed out. If the failure is coming from CI and doesn't reproduce locally, it can be tricky to find out the root cause.
  • This PR
    • makes sure we're fetching the status code and the response body from every supertest request
    • replaces all chained .expect(someStatusCode) occurrances in supertest requests with a separate call of assertResponseStatusCode from the ml.api or transform.api service, which makes sure to log the response body in case of an assertion failure.
  • The new failure message looks like this: 
    Error: Expected status code 200, got 400 with body '{"statusCode":400,"error":"Bad Request","message":"[request body.jobIds]: expected value of type [array] but got [undefined]"}'
+ expected - actual

-400
+200

@pheyos pheyos added :ml test_ui_functional release_note:skip Skip the PR/issue when compiling release notes test-api-integration auto-backport Deprecated - use backport:version if exact versions are needed v8.2.0 v8.1.1 v7.17.2 labels Mar 16, 2022
@pheyos pheyos self-assigned this Mar 16, 2022
@pheyos pheyos requested a review from a team as a code owner March 16, 2022 13:58
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

peteharverson
peteharverson approved these changes Mar 17, 2022
View reviewed changes
Copy link
Contributor

@peteharverson peteharverson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Could you add an example of the new failure response output in the PR description?

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @pheyos

jgowdyelastic
jgowdyelastic approved these changes Mar 17, 2022
View reviewed changes
Copy link
Member

@jgowdyelastic jgowdyelastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great improvement. LGTM

@pheyos pheyos merged commit f4c4fd6 into elastic:main Mar 17, 2022
@pheyos pheyos deleted the refactor_response_code_checks branch March 17, 2022 16:48
@pheyos pheyos added auto-backport Deprecated - use backport:version if exact versions are needed and removed auto-backport Deprecated - use backport:version if exact versions are needed labels Mar 17, 2022
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
❌ 8.1 Backport failed because of merge conflicts
❌ 7.17 Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 127875

Questions ?

Please refer to the Backport tool documentation

@pheyos pheyos mentioned this pull request Mar 18, 2022
Merged
pheyos added a commit to pheyos/kibana that referenced this pull request Mar 18, 2022
@pheyos
[ML] Functional tests - refactor response code checks (elastic#127875)
3eaccfd 
This PR refactors the way how we assert response codes from API requests in our functional tests.

(cherry picked from commit f4c4fd6)

# Conflicts:
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/jobs_exist_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/new_job_caps.ts
#	x-pack/test/functional/services/ml/api.ts
pheyos added a commit to pheyos/kibana that referenced this pull request Mar 18, 2022
@pheyos
[ML] Functional tests - refactor response code checks (elastic#127875)
b80facf 
This PR refactors the way how we assert response codes from API requests in our functional tests.

(cherry picked from commit f4c4fd6)

# Conflicts:
#	x-pack/test/api_integration/apis/ml/anomaly_detectors/forecast_with_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/get.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/jobs_exist_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/new_job_caps.ts
#	x-pack/test/api_integration/apis/ml/datafeeds/update.ts
#	x-pack/test/api_integration/apis/ml/results/get_category_definition.ts
#	x-pack/test/api_integration/apis/ml/results/get_category_examples.ts
#	x-pack/test/api_integration/apis/transform/reset_transforms.ts
#	x-pack/test/functional/services/ml/api.ts
#	x-pack/test/functional/services/ml/test_resources.ts
@pheyos pheyos mentioned this pull request Mar 18, 2022
Merged
@pheyos
Copy link
Member Author

pheyos commented Mar 18, 2022

💚 All backports created successfully

Status Branch Result
✅ 8.1
✅ 7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

pheyos added a commit that referenced this pull request Mar 18, 2022
@pheyos
[ML] Functional tests - refactor response code checks (#127875) (#128044
26a40dc 
)

This PR refactors the way how we assert response codes from API requests in our functional tests.

(cherry picked from commit f4c4fd6)

# Conflicts:
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/jobs_exist_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/new_job_caps.ts
#	x-pack/test/functional/services/ml/api.ts
pheyos added a commit that referenced this pull request Mar 18, 2022
@pheyos
[ML] Functional tests - refactor response code checks (#127875) (#128046
3197085 
)

This PR refactors the way how we assert response codes from API requests in our functional tests.

(cherry picked from commit f4c4fd6)

# Conflicts:
#	x-pack/test/api_integration/apis/ml/anomaly_detectors/forecast_with_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/get.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/jobs_exist_spaces.ts
#	x-pack/test/api_integration/apis/ml/data_frame_analytics/new_job_caps.ts
#	x-pack/test/api_integration/apis/ml/datafeeds/update.ts
#	x-pack/test/api_integration/apis/ml/results/get_category_definition.ts
#	x-pack/test/api_integration/apis/ml/results/get_category_examples.ts
#	x-pack/test/api_integration/apis/transform/reset_transforms.ts
#	x-pack/test/functional/services/ml/api.ts
#	x-pack/test/functional/services/ml/test_resources.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@peteharverson peteharverson peteharverson approved these changes

@jgowdyelastic jgowdyelastic jgowdyelastic approved these changes

Assignees

@pheyos pheyos

Labels
auto-backport Deprecated - use backport:version if exact versions are needed :ml release_note:skip Skip the PR/issue when compiling release notes test_ui_functional test-api-integration v7.17.2 v8.1.1 v8.2.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@pheyos @elasticmachine @kibana-ci @kibanamachine @peteharverson @jgowdyelastic