Update Model Serving test tags and skip if component is enabled #1010

bdattoma · 2023-11-07T17:36:40Z

Since KServe and ModelMesh are becoming full part of the product, we should have easier ways to select the tests when we runt them.
In order to achieve that, this PR is adding KServe and ModelMesh, besides adding Quality Gates tags in the KServe suite.

In this way we could include and exclude the tests based on the components we are going to deploy in the test cluster. Moreover, it automatically skips the corresponding tests if Model Mesh or KServe or both are disabled.

Small Extra: replacing the WatsonX references with KServe

FedeAlonso

LGTM

ods_ci/tests/Tests/100__deploy/100__installation/110__model_serving_watsonx.robot

github-actions · 2023-11-14T11:32:27Z

Robot Results

✅ Passed	❌ Failed	⏭️ Skipped	Total	Pass %
373	0	0	373	100

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/420__model_serving.robot

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

bdattoma · 2023-11-14T16:31:28Z

Smoke for PR test executed on rhods-smoke/3900: SUCCESS (2 failures unrelated to this PR)

jgarciao

Added some comments, but they can be addressed in a future PR if needed

jgarciao · 2023-11-15T09:24:02Z

ods_ci/tests/Tests/100__deploy/100__installation/110__model_serving_watsonx.robot

@@ -15,8 +15,8 @@ ${KSERVE_NS}=    ${APPLICATIONS_NAMESPACE}
 *** Test Cases ***
 Verify KServe Is Shipped
    [Documentation]    Verify KServe Is Shipped And Enabled Within ODS
-    [Tags]    WatsonX
-    ...       ODS-2325
+    [Tags]    ODS-2325


Missing TierX tag

Optional: Usage of Run Keyword And Continue On Failure can be replaced by adding [Tags] robot:recursive-continue-on-failure. This is documented here https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#continue-on-failure

Missing TierX tag

on purpose for this one. I'm not sure about keeping it because it requires the pre-req operators to be installed and it's not done in this suite. Will need to re-assess it's utility

ods_ci/tests/Tests/100__deploy/100__installation/110__model_serving_watsonx.robot

jgarciao · 2023-11-15T09:27:40Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

@@ -65,15 +66,15 @@ ${SCRIPT_BREW_TAG}=    ${EMPTY}    # ^[0-9]+$
 Verify External Dependency Operators Can Be Deployed
    [Documentation]    Checks the pre-required Operators can be installed
    ...                and configured
-    [Tags]    ODS-2326    WatsonX
+    [Tags]    ODS-2326


Missing TierX tag

@bdattoma What is the status of this? Is it okay or shall it be added?

just updated the PR

jgarciao · 2023-11-15T09:27:52Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

-    [Tags]    ODS-2341    WatsonX
-    [Setup]    Set Project And Runtime    namespace=berto
-    ${test_namespace}=    Set Variable     berto
+    [Tags]    Smoke    ODS-2341


Missing TierX tag

jgarciao · 2023-11-15T09:28:27Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

@@ -384,7 +385,7 @@ Verify Model Can Be Served And Query On A GPU Node
 Verify Non Admin Can Serve And Query A Model
    [Documentation]    Basic tests leveraging on a non-admin user for preparing, deploying and querying a LLM model
    ...                using Kserve and Caikit+TGIS runtime
-    [Tags]    ODS-2326    WatsonX
+    [Tags]    Smoke    ODS-2326


Missing TierX tag

I thought we don't want to set Tier tag on Smoke after the discussion on redefine quality gates, am I wrong?

I believe TierX has to be always included, independently if the test is in Smoke or Sanity or any other special purpose quality gate

jgarciao · 2023-11-15T09:28:41Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

-    [Tags]    ODS-2401    WatsonX
-    [Setup]    Set Project And Runtime    namespace=watsonx-metrics    enable_metrics=${TRUE}
-    ${test_namespace}=    Set Variable     watsonx-metrics
+    [Tags]    Smoke    ODS-2401


Missing TierX tag

jgarciao · 2023-11-15T09:28:58Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

@@ -535,7 +536,7 @@ Verify User Can Access Model Metrics From UWM
 Verify User Can Query A Model Using HTTP Calls
    [Documentation]    From RHOAI 2.5 HTTP is allowed and default querying protocol.
    ...                This tests deploys the runtime enabling HTTP port and send queries to the model
-    [Tags]    ODS-2501    WatsonX
+    [Tags]    ODS-2501    Smoke


Missing TierX tag

jgarciao · 2023-11-21T10:56:24Z

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot

@@ -384,7 +385,7 @@ Verify Model Can Be Served And Query On A GPU Node
 Verify Non Admin Can Serve And Query A Model
    [Documentation]    Basic tests leveraging on a non-admin user for preparing, deploying and querying a LLM model
    ...                using Kserve and Caikit+TGIS runtime
-    [Tags]    ODS-2326    WatsonX
+    [Tags]    Smoke    ODS-2326


I believe TierX has to be always included, independently if the test is in Smoke or Sanity or any other special purpose quality gate

Signed-off-by: bdattoma <[email protected]>

sonarqubecloud · 2023-11-21T11:14:11Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

bdattoma requested review from tarukumar, jstourac, jgarciao, aloganat, FedeAlonso, mwaykole and lugi0 November 7, 2023 17:36

bdattoma self-assigned this Nov 7, 2023

bdattoma added the enhancements Bugfixes, enhancements, refactoring, ... in tests or libraries (PR will be listed in release-notes) label Nov 7, 2023

FedeAlonso previously approved these changes Nov 8, 2023

View reviewed changes

jgarciao previously approved these changes Nov 8, 2023

View reviewed changes

ods_ci/tests/Tests/100__deploy/100__installation/110__model_serving_watsonx.robot Show resolved Hide resolved

bdattoma added the do not merge Do not merge this yet please label Nov 8, 2023

bdattoma dismissed stale reviews from jgarciao and FedeAlonso via 3047289 November 9, 2023 09:08

bdattoma force-pushed the update_kserve_tags branch from 3047289 to 79e6867 Compare November 14, 2023 11:29

github-advanced-security bot found potential problems Nov 14, 2023

View reviewed changes

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Nov 14, 2023

View reviewed changes

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/420__model_serving.robot Fixed Show fixed Hide fixed

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/420__model_serving.robot Fixed Show fixed Hide fixed

jgarciao reviewed Nov 14, 2023

View reviewed changes

ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot Outdated Show resolved Hide resolved

bdattoma requested review from jgarciao and FedeAlonso November 14, 2023 16:03

bdattoma removed the do not merge Do not merge this yet please label Nov 14, 2023

bdattoma added the verified This PR has been tested with Jenkins label Nov 15, 2023

bdattoma changed the title ~~Update Model Serving test tags~~ Update Model Serving test tags and skip if component is enabled Nov 15, 2023

jgarciao previously approved these changes Nov 15, 2023

View reviewed changes

bdattoma dismissed jgarciao’s stale review via 09ed515 November 15, 2023 11:39

bdattoma requested a review from jgarciao November 15, 2023 11:42

jgarciao mentioned this pull request Nov 21, 2023

Add tag with Polarion test case to DW test #1024

Merged

jgarciao previously approved these changes Nov 21, 2023

View reviewed changes

bdattoma added 12 commits November 21, 2023 12:12

add quality gates tags + remove watsonx references

931a118

Signed-off-by: bdattoma <[email protected]>

add tier1 tags

80938f4

Signed-off-by: bdattoma <[email protected]>

add modelmesh tag

20a92eb

Signed-off-by: bdattoma <[email protected]>

replace KServe tag with Test Tags setting

a78156b

Signed-off-by: bdattoma <[email protected]>

add smoke and kserve tag to ODS-2325

332d349

Signed-off-by: bdattoma <[email protected]>

remove temporarily ODS-2325 from Smoke

9e611a5

Signed-off-by: bdattoma <[email protected]>

add http test to smoke

c3344c8

Signed-off-by: bdattoma <[email protected]>

add skip kserve suite if not enabled

a290c1d

Signed-off-by: bdattoma <[email protected]>

add skip mm suite is disabled

93f8823

Signed-off-by: bdattoma <[email protected]>

move kw to skip based on component to Common.robot

64bf59b

Signed-off-by: bdattoma <[email protected]>

add test skip for missing mm

75e4c67

Signed-off-by: bdattoma <[email protected]>

rename kserve post install test

02f5c0e

Signed-off-by: bdattoma <[email protected]>

bdattoma force-pushed the update_kserve_tags branch from 830b5e1 to 02f5c0e Compare November 21, 2023 11:12

add Tier1 on Smoke

8bfbc52

Signed-off-by: bdattoma <[email protected]>

bdattoma dismissed jgarciao’s stale review via 8bfbc52 November 21, 2023 11:13

jgarciao self-requested a review November 21, 2023 11:45

jstourac approved these changes Nov 21, 2023

View reviewed changes

jgarciao approved these changes Nov 21, 2023

View reviewed changes

jgarciao merged commit f24f800 into red-hat-data-services:master Nov 21, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Model Serving test tags and skip if component is enabled #1010

Update Model Serving test tags and skip if component is enabled #1010

bdattoma commented Nov 7, 2023 •

edited

Loading

FedeAlonso left a comment

github-actions bot commented Nov 14, 2023

bdattoma commented Nov 14, 2023 •

edited

Loading

jgarciao left a comment

jgarciao Nov 15, 2023

jgarciao Nov 15, 2023

bdattoma Nov 15, 2023

jgarciao Nov 15, 2023

jstourac Nov 21, 2023

bdattoma Nov 21, 2023

jgarciao Nov 15, 2023

jgarciao Nov 15, 2023

bdattoma Nov 15, 2023 •

edited

Loading

jgarciao Nov 21, 2023

jgarciao Nov 15, 2023

jgarciao Nov 15, 2023

jgarciao Nov 21, 2023

sonarqubecloud bot commented Nov 21, 2023

Update Model Serving test tags and skip if component is enabled #1010

Update Model Serving test tags and skip if component is enabled #1010

Conversation

bdattoma commented Nov 7, 2023 • edited Loading

FedeAlonso left a comment

Choose a reason for hiding this comment

github-actions bot commented Nov 14, 2023

Robot Results

bdattoma commented Nov 14, 2023 • edited Loading

jgarciao left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bdattoma Nov 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Nov 21, 2023

bdattoma commented Nov 7, 2023 •

edited

Loading

bdattoma commented Nov 14, 2023 •

edited

Loading

bdattoma Nov 15, 2023 •

edited

Loading