Update Model Serving tests for 2.5 #1044

bdattoma · 2023-11-27T16:32:37Z

Tests for CustomServingRuntime feature must be updated to account for serving platform selection, introduced with 2.5.

This PR includes:

fix broken CustomServingRuntime tests
add one new CustomServingRuntime test (ODS-2542)
fix keywords for model mesh serving UI
fix test teardown for KServe tests
fix Resource-GPU tag for KServe test (it should be Resources-GPU)
(tentative) try to mitigate timing issue in KServe metrics test

ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHDashboardSettingsRuntimes.resource

...ci/tests/Tests/400__ods_dashboard/420__model_serving/423__model_serving_customruntimes.robot

github-actions · 2023-11-27T16:35:35Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %
373	0	0	373	100

ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHDashboardSettingsRuntimes.resource

Signed-off-by: bdattoma <[email protected]>

github-advanced-security

Robocop found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

Signed-off-by: bdattoma <[email protected]>

bdattoma · 2023-12-05T09:44:44Z

PR Validation

CustomServingRuntime suite: rhods-ci-pr-test/2165 PASS - re-run after new changes: rhods-ci-pr-test/2171 PASS
Model (mesh) Serving Smoke suite: rhods-ci-pr-test/2172 PASS
KServe metrics tests: rhods-ci-pr-test/2174 PASS
Model (mesh) Serving Sanity suite: rhods-ci-pr-test/2177 PASS

Signed-off-by: bdattoma <[email protected]>

jstourac

Thank you Berto.

I put some comments but overall LGTM.

Note: we should mark the new test-case as automated in Polarion once this is merged, correct?

ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource

jstourac · 2023-12-05T15:38:58Z

...ci/tests/Tests/400__ods_dashboard/420__model_serving/423__model_serving_customruntimes.robot

    Verify Model Status    ${model_name}    success
    Verify Model Inference    ${model_name}    ${inference_input}    ${exp_inference_output}    token_auth=${TRUE}
    ...    project_title=${PRJ_TITLE}
-
+    


please remove

why? according to the linter, there must be 2 lines btw sections

I meant to remove the trailing spaces 🙂

oh, sorry. I'll remove it in some next PRs (I have in my backlog new tests in this suite after 2.5 release)

jstourac · 2023-12-05T15:49:50Z

...ci/tests/Tests/400__ods_dashboard/420__model_serving/423__model_serving_customruntimes.robot

+    ...                supported serving platform (single model, multi model, both)
+    Set Suite Variable    ${RUNTIME_BOTH_FILEPATH}    ${RESOURCES_DIRPATH}/csr_both_model.yaml
+    Set Suite Variable    ${RUNTIME_SINGLE_FILEPATH}    ${RESOURCES_DIRPATH}/csr_single_model.yaml
+    Set Suite Variable    ${RUNTIME_MULTI_FILEPATH}    ${RESOURCES_DIRPATH}/csr_multi_model.yaml


These three ${RESOURCES_DIRPATH}/csr_* are used on two places. Maybe it would make sense to create a global variables in this file for them?

these will be used only by one test, hence I made these suite variable only when running that particular test

yeah, I didn't mean to make it global as that much global, just to define it only once in the file, but I don't insist, that's a small thing 🙂

Signed-off-by: bdattoma <[email protected]>

sonarqubecloud · 2023-12-05T16:16:02Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

* add platform selection and check Signed-off-by: bdattoma <[email protected]> * replace rhods ns with global variable Signed-off-by: bdattoma <[email protected]> * fix ods-2276 Signed-off-by: bdattoma <[email protected]> * fix runtime creation steps Signed-off-by: bdattoma <[email protected]> * fix model deployment step Signed-off-by: bdattoma <[email protected]> * increase model ready timeout in the test Signed-off-by: bdattoma <[email protected]> * fix alerts Signed-off-by: bdattoma <[email protected]> * add ODS-2542 Signed-off-by: bdattoma <[email protected]> * fix one alert Signed-off-by: bdattoma <[email protected]> * remove unused line Signed-off-by: bdattoma <[email protected]> * fix teardown for kserve tests Signed-off-by: bdattoma <[email protected]> * fix modelmesh teardown Signed-off-by: bdattoma <[email protected]> * adapt serve model kw for 2.5 Signed-off-by: bdattoma <[email protected]> * fix gpu tag for kserve test Signed-off-by: bdattoma <[email protected]> * invert order uwm conf with enablement Signed-off-by: bdattoma <[email protected]> * change modelmesh token enablement logic for 2.5 Signed-off-by: bdattoma <[email protected]> * minor change in kserve clean up Signed-off-by: bdattoma <[email protected]> * fix auto fetch of project in mm sanity test Signed-off-by: bdattoma <[email protected]> * remove sleep Signed-off-by: bdattoma <[email protected]> --------- Signed-off-by: bdattoma <[email protected]>

bdattoma added needs testing Needs to be tested in Jenkins enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes) labels Nov 27, 2023

github-advanced-security bot found potential problems Nov 27, 2023

View reviewed changes

ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHDashboardSettingsRuntimes.resource Fixed Show fixed Hide fixed

bdattoma force-pushed the f/customruntimes-2 branch from a8cde03 to 6ff5b06 Compare November 30, 2023 12:42

bdattoma marked this pull request as ready for review December 4, 2023 16:55

bdattoma added 6 commits December 4, 2023 17:56

add platform selection and check

f57235e

Signed-off-by: bdattoma <[email protected]>

replace rhods ns with global variable

aa3a196

Signed-off-by: bdattoma <[email protected]>

fix ods-2276

77bf392

Signed-off-by: bdattoma <[email protected]>

fix runtime creation steps

35f08de

Signed-off-by: bdattoma <[email protected]>

fix model deployment step

e67026c

Signed-off-by: bdattoma <[email protected]>

increase model ready timeout in the test

38f08d5

Signed-off-by: bdattoma <[email protected]>

bdattoma force-pushed the f/customruntimes-2 branch from 7481107 to 38f08d5 Compare December 4, 2023 16:56

bdattoma added 2 commits December 4, 2023 18:00

fix alerts

3a01db0

Signed-off-by: bdattoma <[email protected]>

add ODS-2542

b0d33f8

Signed-off-by: bdattoma <[email protected]>

github-advanced-security bot found potential problems Dec 4, 2023

View reviewed changes

fix one alert

7239d52

Signed-off-by: bdattoma <[email protected]>

bdattoma requested review from lugi0, jgarciao, jstourac, FedeAlonso and tarukumar December 4, 2023 17:55

bdattoma self-assigned this Dec 4, 2023

remove unused line

d17ff21

Signed-off-by: bdattoma <[email protected]>

bdattoma changed the title ~~Update Custom Serving Runtime tests for 2.5~~ Update Custom Serving Runtime + Model Mesh tests for 2.5 Dec 5, 2023

bdattoma changed the title ~~Update Custom Serving Runtime + Model Mesh tests for 2.5~~ Update Model Serving tests for 2.5 Dec 5, 2023

bdattoma added 2 commits December 5, 2023 11:16

fix teardown for kserve tests

bc78dae

Signed-off-by: bdattoma <[email protected]>

fix modelmesh teardown

96d33ec

Signed-off-by: bdattoma <[email protected]>

bdattoma added 5 commits December 5, 2023 11:29

adapt serve model kw for 2.5

c614cdb

Signed-off-by: bdattoma <[email protected]>

fix gpu tag for kserve test

3d151fa

Signed-off-by: bdattoma <[email protected]>

invert order uwm conf with enablement

089573a

Signed-off-by: bdattoma <[email protected]>

change modelmesh token enablement logic for 2.5

74e1026

Signed-off-by: bdattoma <[email protected]>

minor change in kserve clean up

3b0ab18

Signed-off-by: bdattoma <[email protected]>

bdattoma added verified This PR has been tested with Jenkins needs testing Needs to be tested in Jenkins and removed needs testing Needs to be tested in Jenkins verified This PR has been tested with Jenkins labels Dec 5, 2023

fix auto fetch of project in mm sanity test

8f02960

Signed-off-by: bdattoma <[email protected]>

bdattoma added verified This PR has been tested with Jenkins and removed needs testing Needs to be tested in Jenkins labels Dec 5, 2023

lugi0 previously approved these changes Dec 5, 2023

View reviewed changes

jstourac reviewed Dec 5, 2023

View reviewed changes

jgarciao previously approved these changes Dec 5, 2023

View reviewed changes

bdattoma dismissed stale reviews from jgarciao and lugi0 via a7972c9 December 5, 2023 16:15

remove sleep

a7972c9

Signed-off-by: bdattoma <[email protected]>

bdattoma requested review from jstourac, lugi0 and jgarciao December 5, 2023 16:18

lugi0 approved these changes Dec 5, 2023

View reviewed changes

jgarciao approved these changes Dec 5, 2023

View reviewed changes

jstourac approved these changes Dec 5, 2023

View reviewed changes

bdattoma merged commit e2c9711 into red-hat-data-services:master Dec 5, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Model Serving tests for 2.5 #1044

Update Model Serving tests for 2.5 #1044

bdattoma commented Nov 27, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023 •

edited

Loading

github-advanced-security bot left a comment

bdattoma commented Dec 5, 2023 •

edited

Loading

jstourac left a comment

jstourac Dec 5, 2023

bdattoma Dec 5, 2023

jstourac Dec 5, 2023

bdattoma Dec 5, 2023

jstourac Dec 5, 2023

bdattoma Dec 5, 2023

jstourac Dec 5, 2023

sonarqubecloud bot commented Dec 5, 2023

Update Model Serving tests for 2.5 #1044

Update Model Serving tests for 2.5 #1044

Conversation

bdattoma commented Nov 27, 2023 • edited Loading

github-actions bot commented Nov 27, 2023 • edited Loading

Robot Results

github-advanced-security bot left a comment

Choose a reason for hiding this comment

bdattoma commented Dec 5, 2023 • edited Loading

jstourac left a comment

Choose a reason for hiding this comment

jstourac Dec 5, 2023

Choose a reason for hiding this comment

bdattoma Dec 5, 2023

Choose a reason for hiding this comment

jstourac Dec 5, 2023

Choose a reason for hiding this comment

bdattoma Dec 5, 2023

Choose a reason for hiding this comment

jstourac Dec 5, 2023

Choose a reason for hiding this comment

bdattoma Dec 5, 2023

Choose a reason for hiding this comment

jstourac Dec 5, 2023

Choose a reason for hiding this comment

sonarqubecloud bot commented Dec 5, 2023

bdattoma commented Nov 27, 2023 •

edited

Loading

github-actions bot commented Nov 27, 2023 •

edited

Loading

bdattoma commented Dec 5, 2023 •

edited

Loading