Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Model Serving test tags and skip if component is enabled #1010

Merged
merged 13 commits into from
Nov 21, 2023
Merged

Update Model Serving test tags and skip if component is enabled #1010

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
931a118
add quality gates tags + remove watsonx references
bdattoma Nov 7, 2023
80938f4
add tier1 tags
bdattoma Nov 7, 2023
20a92eb
add modelmesh tag
bdattoma Nov 7, 2023
a78156b
replace KServe tag with Test Tags setting
bdattoma Nov 7, 2023
332d349
add smoke and kserve tag to ODS-2325
bdattoma Nov 7, 2023
9e611a5
remove temporarily ODS-2325 from Smoke
bdattoma Nov 9, 2023
c3344c8
add http test to smoke
bdattoma Nov 14, 2023
a290c1d
add skip kserve suite if not enabled
bdattoma Nov 14, 2023
93f8823
add skip mm suite is disabled
bdattoma Nov 14, 2023
64bf59b
move kw to skip based on component to Common.robot
bdattoma Nov 14, 2023
75e4c67
add test skip for missing mm
bdattoma Nov 14, 2023
02f5c0e
rename kserve post install test
bdattoma Nov 15, 2023
8bfbc52
add Tier1 on Smoke
bdattoma Nov 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions ods_ci/tests/Resources/Common.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Library RequestsLibrary
Library ../../libs/Helpers.py
Resource Page/ODH/JupyterHub/JupyterLabLauncher.robot
Resource Page/ODH/JupyterHub/JupyterHubSpawner.robot
Resource ../../tasks/Resources/RHODS_OLM/install/oc_install.robot
Resource RHOSi.resource


Expand Down Expand Up @@ -406,3 +407,9 @@ Escape String Chars
${str}= Replace String ${str} ${char} \\${char}
END
RETURN ${str}

Skip If Component Is Not Enabled
[Documentation] Skips test if ${component_name} is not enabled in DataScienceCluster
[Arguments] ${component_name}
${enabled}= Is Component Enabled ${component_name}
Skip If "${enabled}" == "false"
4 changes: 2 additions & 2 deletions ...allation/110__model_serving_watsonx.robot → ...tallation/110__model_serving_kserve.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,8 @@ ${KSERVE_NS}= ${APPLICATIONS_NAMESPACE}
*** Test Cases ***
Verify KServe Is Shipped
[Documentation] Verify KServe Is Shipped And Enabled Within ODS
[Tags] WatsonX
... ODS-2325
[Tags] ODS-2325
... KServe
@{kserve_pods_info} = Fetch KServe Pods
@{kserve_services_info} = Fetch KServe Controller Services
@{kserve_wh_services_info} = Fetch KServe Webhook Services
Expand Down
3 changes: 2 additions & 1 deletion ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/420__model_serving.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ Resource ../../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject
Resource ../../../Resources/OCP.resource
Suite Setup Model Serving Suite Setup
Suite Teardown Model Serving Suite Teardown

Test Tags ModelMesh

*** Variables ***
${INFERENCE_INPUT}= @ods_ci/tests/Resources/Files/modelmesh-mnist-input.json
Expand Down Expand Up @@ -132,6 +132,7 @@ Model Serving Suite Setup
[Documentation] Suite setup steps for testing DSG. It creates some test variables
... and runs RHOSi setup
Set Library Search Order SeleniumLibrary
Skip If Component Is Not Enabled modelmeshserving
RHOSi Setup
Launch Dashboard ${TEST_USER.USERNAME} ${TEST_USER.PASSWORD} ${TEST_USER.AUTH_TYPE}
... ${ODH_DASHBOARD_URL} ${BROWSER.NAME} ${BROWSER.OPTIONS}
Expand Down
2 changes: 1 addition & 1 deletion ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/421__model_serving_gpu.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Resource ../../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject
Resource ../../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/ModelServer.resource
Suite Setup Model Serving Suite Setup
Suite Teardown Model Serving Suite Teardown

Test Tags ModelMesh

*** Variables ***
${RHODS_NAMESPACE}= ${APPLICATIONS_NAMESPACE}
Expand Down
70 changes: 36 additions & 34 deletions ods_ci/tests/Tests/400__ods_dashboard/420__model_serving/422__model_serving_llm.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ Resource ../../../Resources/Page/ODH/ODHDashboard/ODHDashboardAPI.resou
Library OpenShiftLibrary
Suite Setup Install Model Serving Stack Dependencies
Suite Teardown RHOSi Teardown
Test Tags KServe


*** Variables ***
Expand All @@ -31,7 +32,7 @@ ${JAEGER_OP_NAME}= jaeger-product
${JAEGER_SUB_NAME}= jaeger-product
${KSERVE_NS}= ${APPLICATIONS_NAMESPACE} # NS is "kserve" for ODH
${CAIKIT_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/caikit_servingruntime_{{protocol}}.yaml
${TEST_NS}= watsonx
${TEST_NS}= singlemodel
${BUCKET_SECRET_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/bucket_secret.yaml
${BUCKET_SA_FILEPATH}= ${LLM_RESOURCES_DIRPATH}/bucket_sa.yaml
${USE_BUCKET_HTTPS}= "1"
Expand Down Expand Up @@ -65,15 +66,15 @@ ${SCRIPT_BREW_TAG}= ${EMPTY} # ^[0-9]+$
Verify External Dependency Operators Can Be Deployed
[Documentation] Checks the pre-required Operators can be installed
... and configured
[Tags] ODS-2326 WatsonX
[Tags] ODS-2326
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing TierX tag

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bdattoma What is the status of this? Is it okay or shall it be added?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just updated the PR

Pass Execution message=Installation done as part of Suite Setup.

Verify User Can Serve And Query A Model
[Documentation] Basic tests for preparing, deploying and querying a LLM model
... using Kserve and Caikit+TGIS runtime
[Tags] ODS-2341 WatsonX
[Setup] Set Project And Runtime namespace=berto
${test_namespace}= Set Variable berto
[Tags] Smoke Tier1 ODS-2341
[Setup] Set Project And Runtime namespace=${TEST_NS}
${test_namespace}= Set Variable ${TEST_NS}
${flan_model_name}= Set Variable flan-t5-small-caikit
${models_names}= Create List ${flan_model_name}
Compile Inference Service YAML isvc_name=${flan_model_name}
Expand All @@ -94,7 +95,7 @@ Verify User Can Serve And Query A Model

Verify User Can Deploy Multiple Models In The Same Namespace
[Documentation] Checks if user can deploy and query multiple models in the same namespace
[Tags] ODS-2371 WatsonX
[Tags] Sanity Tier1 ODS-2371
[Setup] Set Project And Runtime namespace=${TEST_NS}-multisame
${test_namespace}= Set Variable ${TEST_NS}-multisame
${model_one_name}= Set Variable bloom-560m-caikit
Expand Down Expand Up @@ -127,10 +128,10 @@ Verify User Can Deploy Multiple Models In The Same Namespace

Verify User Can Deploy Multiple Models In Different Namespaces
[Documentation] Checks if user can deploy and query multiple models in the different namespaces
[Tags] ODS-2378 WatsonX
[Setup] Run Keywords Set Project And Runtime namespace=watsonx-multi1
[Tags] Sanity Tier1 ODS-2378
[Setup] Run Keywords Set Project And Runtime namespace=singlemodel-multi1
... AND
... Set Project And Runtime namespace=watsonx-multi2
... Set Project And Runtime namespace=singlemodel-multi2
${model_one_name}= Set Variable bloom-560m-caikit
${model_two_name}= Set Variable flan-t5-small-caikit
${models_names_ns_1}= Create List ${model_one_name}
Expand All @@ -139,27 +140,27 @@ Verify User Can Deploy Multiple Models In Different Namespaces
... sa_name=${DEFAULT_BUCKET_SA_NAME}
... model_storage_uri=${BLOOM_STORAGE_URI}
Deploy Model Via CLI isvc_filepath=${LLM_RESOURCES_DIRPATH}/caikit_isvc_filled.yaml
... namespace=watsonx-multi1
... namespace=singlemodel-multi1
Compile Inference Service YAML isvc_name=${model_two_name}
... sa_name=${DEFAULT_BUCKET_SA_NAME}
... model_storage_uri=${FLAN_STORAGE_URI}
Deploy Model Via CLI isvc_filepath=${LLM_RESOURCES_DIRPATH}/caikit_isvc_filled.yaml
... namespace=watsonx-multi2
... namespace=singlemodel-multi2
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${model_one_name}
... namespace=watsonx-multi1
... namespace=singlemodel-multi1
Wait For Pods To Be Ready label_selector=serving.kserve.io/inferenceservice=${model_two_name}
... namespace=watsonx-multi2
... namespace=singlemodel-multi2
Query Model Multiple Times model_name=${model_one_name} n_times=2
... namespace=watsonx-multi1
... namespace=singlemodel-multi1
Query Model Multiple Times model_name=${model_two_name} n_times=2
... namespace=watsonx-multi2
[Teardown] Run Keywords Clean Up Test Project test_ns=watsonx-multi1 isvc_names=${models_names_ns_1}
... namespace=singlemodel-multi2
[Teardown] Run Keywords Clean Up Test Project test_ns=singlemodel-multi1 isvc_names=${models_names_ns_1}
... AND
... Clean Up Test Project test_ns=watsonx-multi2 isvc_names=${models_names_ns_2}
... Clean Up Test Project test_ns=singlemodel-multi2 isvc_names=${models_names_ns_2}

Verify Model Upgrade Using Canaray Rollout
[Documentation] Checks if user can apply Canary Rollout as deployment strategy
[Tags] ODS-2372 WatsonX
[Tags] Sanity Tier1 ODS-2372
[Setup] Set Project And Runtime namespace=canary-model-upgrade
${test_namespace}= Set Variable canary-model-upgrade
${isvc_name}= Set Variable canary-caikit
Expand Down Expand Up @@ -198,7 +199,7 @@ Verify Model Upgrade Using Canaray Rollout
Verify Model Pods Are Deleted When No Inference Service Is Present
[Documentation] Checks if model pods gets successfully deleted after
... deleting the KServe InferenceService object
[Tags] ODS-2373 WatsonX
[Tags] Tier2 ODS-2373
[Setup] Set Project And Runtime namespace=no-infer-kserve
${flan_isvc_name}= Set Variable flan-t5-small-caikit
${model_name}= Set Variable flan-t5-small-caikit
Expand All @@ -217,7 +218,7 @@ Verify Model Pods Are Deleted When No Inference Service Is Present
Verify User Can Change The Minimum Number Of Replicas For A Model
[Documentation] Checks if user can change the minimum number of replicas
... of a deployed model
[Tags] ODS-2376 WatsonX
[Tags] Sanity Tier1 ODS-2376
[Setup] Set Project And Runtime namespace=${TEST_NS}-reps
${test_namespace}= Set Variable ${TEST_NS}-reps
${model_name}= Set Variable flan-t5-small-caikit
Expand Down Expand Up @@ -253,7 +254,7 @@ Verify User Can Change The Minimum Number Of Replicas For A Model

Verify User Can Autoscale Using Concurrency
[Documentation] Checks if model successfully scale up based on concurrency metrics (KPA)
[Tags] ODS-2377 WatsonX
[Tags] Sanity Tier1 ODS-2377
[Setup] Set Project And Runtime namespace=autoscale-con
${test_namespace}= Set Variable autoscale-con
${flan_model_name}= Set Variable flan-t5-small-caikit
Expand All @@ -278,7 +279,7 @@ Verify User Can Autoscale Using Concurrency

Verify User Can Validate Scale To Zero
[Documentation] Checks if model successfully scale down to 0 if there's no traffic
[Tags] ODS-2379 WatsonX
[Tags] Sanity Tier1 ODS-2379
[Setup] Set Project And Runtime namespace=autoscale-zero
${flan_model_name}= Set Variable flan-t5-small-caikit
${model_name}= Create List ${flan_model_name}
Expand Down Expand Up @@ -315,7 +316,7 @@ Verify User Can Validate Scale To Zero

Verify User Can Set Requests And Limits For A Model
[Documentation] Checks if user can set HW request and limits on their inference service object
[Tags] ODS-2380 WatsonX
[Tags] Sanity Tier1 ODS-2380
[Setup] Set Project And Runtime namespace=hw-res
${test_namespace}= Set Variable hw-res
${flan_model_name}= Set Variable flan-t5-small-caikit
Expand Down Expand Up @@ -353,9 +354,9 @@ Verify User Can Set Requests And Limits For A Model
Verify Model Can Be Served And Query On A GPU Node
[Documentation] Basic tests for preparing, deploying and querying a LLM model on GPU node
... using Kserve and Caikit+TGIS runtime
[Tags] ODS-2381 WatsonX Resource-GPU
[Setup] Set Project And Runtime namespace=watsonx-gpu
${test_namespace}= Set Variable watsonx-gpu
[Tags] Sanity Tier1 ODS-2381 Resource-GPU
[Setup] Set Project And Runtime namespace=singlemodel-gpu
${test_namespace}= Set Variable singlemodel-gpu
${model_name}= Set Variable flan-t5-small-caikit
${models_names}= Create List ${model_name}
${requests}= Create Dictionary nvidia.com/gpu=1
Expand Down Expand Up @@ -384,7 +385,7 @@ Verify Model Can Be Served And Query On A GPU Node
Verify Non Admin Can Serve And Query A Model
[Documentation] Basic tests leveraging on a non-admin user for preparing, deploying and querying a LLM model
... using Kserve and Caikit+TGIS runtime
[Tags] ODS-2326 WatsonX
[Tags] Smoke Tier1 ODS-2326
[Setup] Run Keywords Login To OCP Using API ${TEST_USER_3.USERNAME} ${TEST_USER_3.PASSWORD} AND
... Set Project And Runtime namespace=non-admin-test
${test_namespace}= Set Variable non-admin-test
Expand All @@ -411,7 +412,7 @@ Verify Non Admin Can Serve And Query A Model

Verify User Can Serve And Query Flan-t5 Grammar Syntax Corrector
[Documentation] Deploys and queries flan-t5-large-grammar-synthesis model
[Tags] ODS-2441 WatsonX
[Tags] Tier2 ODS-2441
[Setup] Set Project And Runtime namespace=grammar-model
${test_namespace}= Set Variable grammar-model
${flan_model_name}= Set Variable flan-t5-large-grammar-synthesis-caikit
Expand All @@ -434,7 +435,7 @@ Verify User Can Serve And Query Flan-t5 Grammar Syntax Corrector

Verify User Can Serve And Query Flan-t5 Large
[Documentation] Deploys and queries flan-t5-large model
[Tags] ODS-2434 WatsonX
[Tags] Tier2 ODS-2434
[Setup] Set Project And Runtime namespace=flan-t5-large3
${test_namespace}= Set Variable flan-t5-large3
${flan_model_name}= Set Variable flan-t5-large
Expand All @@ -461,7 +462,7 @@ Verify Runtime Upgrade Does Not Affect Deployed Models
... must remain unchanged after the runtime upgrade.
... ATTENTION: this is an approximation of the runtime upgrade scenario, however
... the real case scenario will be defined once RHODS actually ships the Caikit runtime.
[Tags] ODS-2404 WatsonX
[Tags] Sanity Tier1 ODS-2404
[Setup] Set Project And Runtime namespace=${TEST_NS}
${test_namespace}= Set Variable ${TEST_NS}
${flan_model_name}= Set Variable flan-t5-small-caikit
Expand Down Expand Up @@ -496,9 +497,9 @@ Verify User Can Access Model Metrics From UWM
... PARTIALLY DONE: it is checking number of requests, number of successful requests
... and model pod cpu usage. Waiting for a complete list of expected metrics and
... derived metrics.
[Tags] ODS-2401 WatsonX
[Setup] Set Project And Runtime namespace=watsonx-metrics enable_metrics=${TRUE}
${test_namespace}= Set Variable watsonx-metrics
[Tags] Smoke Tier1 ODS-2401
[Setup] Set Project And Runtime namespace=singlemodel-metrics enable_metrics=${TRUE}
${test_namespace}= Set Variable singlemodel-metrics
${flan_model_name}= Set Variable flan-t5-small-caikit
${models_names}= Create List ${flan_model_name}
${thanos_url}= Get OpenShift Thanos URL
Expand Down Expand Up @@ -535,7 +536,7 @@ Verify User Can Access Model Metrics From UWM
Verify User Can Query A Model Using HTTP Calls
[Documentation] From RHOAI 2.5 HTTP is allowed and default querying protocol.
... This tests deploys the runtime enabling HTTP port and send queries to the model
[Tags] ODS-2501 WatsonX
[Tags] ODS-2501 Smoke Tier1
[Setup] Set Project And Runtime namespace=kserve-http protocol=http
${test_namespace}= Set Variable kserve-http
${model_name}= Set Variable flan-t5-small-caikit
Expand Down Expand Up @@ -565,6 +566,7 @@ Install Model Serving Stack Dependencies
... This is likely going to change in the future and it will include a way to skip installation.
... Caikit runtime will be shipped Out-of-the-box and will be removed from here.
RHOSi Setup
Skip If Component Is Not Enabled kserve
IF ${SKIP_PREREQS_INSTALL} == ${FALSE}
IF ${SCRIPT_BASED_INSTALL} == ${FALSE}
Install Service Mesh Stack
Expand Down
4 changes: 3 additions & 1 deletion ...tests/Tests/400__ods_dashboard/420__model_serving/423__model_serving_customruntimes.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,10 @@ Verify RHODS Admins Can Delete A Custom Serving Runtime Template
Verify RHODS Users Can Deploy A Model Using A Custom Serving Runtime
[Documentation] Verifies that a model can be deployed using only the UI.
... At the end of the process, verifies the correct resources have been deployed.
[Tags] Sanity ODS-2281
[Tags] Sanity Tier1 ODS-2281 ModelMesh
[Setup] Run Keywords
... Skip If Component Is Not Enabled modelmeshserving
... AND
... Create Test Serving Runtime Template If Not Exists
... AND
... Create Data Science Project If Not Exists project_title=${PRJ_TITLE} username=${TEST_USER_3.USERNAME}
Expand Down
Loading