Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Model Serving tests for 2.5 #1044

Merged
merged 19 commits into from
Dec 5, 2023
Merged

Update Model Serving tests for 2.5 #1044

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
f57235e
add platform selection and check
bdattoma Nov 27, 2023
aa3a196
replace rhods ns with global variable
bdattoma Nov 27, 2023
77bf392
fix ods-2276
bdattoma Nov 30, 2023
35f08de
fix runtime creation steps
bdattoma Dec 4, 2023
e67026c
fix model deployment step
bdattoma Dec 4, 2023
38f08d5
increase model ready timeout in the test
bdattoma Dec 4, 2023
3a01db0
fix alerts
bdattoma Dec 4, 2023
b0d33f8
add ODS-2542
bdattoma Dec 4, 2023
7239d52
fix one alert
bdattoma Dec 4, 2023
d17ff21
remove unused line
bdattoma Dec 4, 2023
bc78dae
fix teardown for kserve tests
bdattoma Dec 5, 2023
96d33ec
fix modelmesh teardown
bdattoma Dec 5, 2023
c614cdb
adapt serve model kw for 2.5
bdattoma Dec 5, 2023
3d151fa
fix gpu tag for kserve test
bdattoma Dec 5, 2023
089573a
invert order uwm conf with enablement
bdattoma Dec 5, 2023
74e1026
change modelmesh token enablement logic for 2.5
bdattoma Dec 5, 2023
3b0ab18
minor change in kserve clean up
bdattoma Dec 5, 2023
8f02960
fix auto fetch of project in mm sanity test
bdattoma Dec 5, 2023
a7972c9
remove sleep
bdattoma Dec 5, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 31 additions & 3 deletions ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHDashboardSettingsRuntimes.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@
${UPLOAD_RUNTIME_BTN_XP}= //button[text()="Upload files"]
${SCRATCH_RUNTIME_BTN_XP}= //button[text()="Start from scratch"]
${EDITOR_RUNTIME_BTN_XP}= //div[contains(@class, "odh-dashboard__code-editor")]
&{PLATFORM_NAMES_MAPPING}= single=Single model serving platform multi=Multi-model serving platform
... both=Both single and multi-model serving platforms
&{PLATFORM_LABELS_MAPPING}= single=Single model multi=Multi-model


*** Keywords ***
Expand All @@ -23,14 +26,15 @@

Upload Serving Runtime Template
[Documentation] Uploads via UI a YAML file containing a custom Serving Runtime definition
[Arguments] ${runtime_filepath}
[Arguments] ${runtime_filepath} ${serving_platform}
Click Add Serving Runtime Template Button
SeleniumLibrary.Click Element ${UPLOAD_RUNTIME_BTN_XP}
${rc} ${pwd}= Run And Return Rc And Output echo $PWD
Should Be Equal As Integers ${rc} ${0}
SeleniumLibrary.Choose File ${EDITOR_RUNTIME_BTN_XP}//input[@type="file"] ${pwd}/${runtime_filepath}
Run Keyword And Continue On Failure SeleniumLibrary.Wait Until Page Contains Element //span[text()="kind"]
Run Keyword And Continue On Failure SeleniumLibrary.Wait Until Page Contains Element //span[text()="ServingRuntime"]
Select Model Serving Platform platform=${serving_platform}
Submit Serving Runtime Template

Submit Serving Runtime Template
Expand All @@ -39,11 +43,27 @@
... wait_for_cards=${FALSE}

Serving Runtime Template Should Be Listed
[Arguments] ${displayed_name}
[Arguments] ${displayed_name} ${serving_platform}
Run Keyword And Continue On Failure
... SeleniumLibrary.Wait Until Page Contains Element
... //table//tr/td[@data-label="Name"]//div[text()="${displayed_name}"]
... timeout=10s
Run Keyword And Continue On Failure
... SeleniumLibrary.Wait Until Page Contains Element
... xpath=//table//tr[td[@data-label="Name"]//div[text()="${displayed_name}"]]/td[@data-label="Serving platforms supported"] # robocop: disable
${actual_platform_labels_str}= SeleniumLibrary.Get Text
... xpath=//table//tr[td[@data-label="Name"]//div[text()="${displayed_name}"]]/td[@data-label="Serving platforms supported"] # robocop: disable
${actual_platform_labels}= Split To Lines ${actual_platform_labels_str}
IF "${serving_platform}" == "both"
Dismissed Show dismissed Hide dismissed
Run Keyword And Continue On Failure Length Should Be ${actual_platform_labels} ${2}
${exp_platform_labels}= Create List ${PLATFORM_LABELS_MAPPING}[single]
... ${PLATFORM_LABELS_MAPPING}[multi]
Run Keyword And Continue On Failure Lists Should Be Equal ${actual_platform_labels}
... ${exp_platform_labels}
ELSE
Run Keyword And Continue On Failure Should Be Equal ${actual_platform_labels_str}
... ${PLATFORM_LABELS_MAPPING}[${serving_platform}]
END

Delete Serving Runtime Template
[Arguments] ${displayed_name} ${press_cancel}=${FALSE}
Expand All @@ -64,7 +84,15 @@
Get OpenShift Template Resource Name By Displayed Name
[Arguments] ${displayed_name}
${rc} ${resource_name}= Run And Return Rc And Output
... oc get templates -ojson -n redhat-ods-applications | jq '.items[] | select(.objects[].metadata.annotations."openshift.io/display-name"=="${displayed_name}") | .metadata.name' | tr -d '"'
... oc get templates -ojson -n ${APPLICATIONS_NAMESPACE} | jq '.items[] | select(.objects[].metadata.annotations."openshift.io/display-name"=="${displayed_name}") | .metadata.name' | tr -d '"'
Fixed Show fixed Hide fixed
Should Be Equal As Integers ${rc} ${0}
RETURN ${resource_name}

Select Model Serving Platform
[Documentation] Selects which model serving platform the serving runtime could be executed on
[Arguments] ${platform}="both"
${platform_link_name}= Set Variable Single model serving platform
Fixed Show fixed Hide fixed
${platform_link_name}= Get From Dictionary ${PLATFORM_NAMES_MAPPING} ${platform.lower()}
SeleniumLibrary.Click Element css:div#custom-serving-runtime-selection > button
SeleniumLibrary.Wait Until Page Contains Element xpath://a[text()="${platform_link_name}"]
SeleniumLibrary.Click Link ${platform_link_name}
4 changes: 2 additions & 2 deletions ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/ModelServer.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ Create Model Server
[Arguments] ${no_replicas}=1 ${server_size}=Small ${ext_route}=${TRUE}
... ${token}=${TRUE} ${runtime}=OpenVINO Model Server ${server_name}=Model Serving Test
... ${no_gpus}=0
Click Button Add server
SeleniumLibrary.Click Button Add model server
Wait Until Page Contains Element //span[.="Add model server"]
Set Model Server Name ${server_name}
Set Replicas Number With Buttons ${no_replicas}
Expand Down Expand Up @@ -143,7 +143,7 @@ Set Model Server Runtime
[Documentation] Selects a given Runtime for the model server
[Arguments] ${runtime}
Open Serving runtime Options Menu # robocop: disable
Click Element xpath://div[@id="serving-runtime-template-selection"]//li//div[text()="${runtime}"]
Click Element xpath://div[@id="serving-runtime-template-selection"]//li//*[text()="${runtime}"]
# TODO: Implement Custom

Set Model Server Name
Expand Down
20 changes: 11 additions & 9 deletions ods_ci/tests/Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,14 @@ Serve Model
[Arguments] ${project_name} ${model_name} ${framework} ${data_connection_name} ${model_path}
... ${existing_data_connection}=${TRUE} ${model_server}=Model Serving Test
# TODO: Does not work if there's already a model deployed
Wait Until Page Contains Deploy model
Click Button Deploy model
Wait Until Page Contains Element xpath://h1[.="Deploy model"]
SeleniumLibrary.Wait Until Page Contains Element //div[@id="model-server"]//table
SeleniumLibrary.Wait Until Page Contains Deploy model
SeleniumLibrary.Click Button Deploy model
SeleniumLibrary.Wait Until Page Contains Element xpath://h1[.="Deploy model"]
Select Project ${project_name}
Set Model Name ${model_name}
Select Model Server ${model_server}
Wait Until Page Contains Element xpath://span[.="Model framework (name - version)"]
SeleniumLibrary.Wait Until Page Contains Element xpath://span[.="Model framework (name - version)"]
Select Framework ${framework}
IF ${existing_data_connection}==${TRUE}
# Select Radio Button group_name=radiogroup value=existing-data-connection-radio
Expand All @@ -63,13 +64,14 @@ Serve Model
Set Folder Path ${model_path}
ELSE
# Existing connection radio is selected by default; for now blindly click on new connection radio
Click Element //input[@id="new-data-connection-radio"]
SeleniumLibrary.Click Element //input[@id="new-data-connection-radio"]
# Select Radio Button group_name=radiogroup value=new-data-connection-radio
Set Up New Data Connection dc_name=${data_connection_name}
Set Folder Path ${model_path}
END
Click Button Deploy
Wait Until Page Does Not Contain xpath://h1[.="Deploy model"]
Sleep 5s
jstourac marked this conversation as resolved.
Show resolved Hide resolved
SeleniumLibrary.Click Button Deploy
SeleniumLibrary.Wait Until Page Does Not Contain xpath://h1[.="Deploy model"]

Select Project
[Documentation] Selects a project in the "deploy model" modal.
Expand Down Expand Up @@ -359,13 +361,13 @@ Disable Model Serving Runtime Using CLI
Should Be Equal As Integers ${rc} ${0}

Wait Until Runtime Pod Is Running
[Arguments] ${server_name} ${project_title}
[Arguments] ${server_name} ${project_title} ${timeout}=10s
${ns_name}= Get Openshift Namespace From Data Science Project project_title=${project_title}
${runtime_pod_name} = Replace String Using Regexp string=${server_name} pattern=\\s replace_with=-
${runtime_pod_name} = Convert To Lower Case ${runtime_pod_name}
Log name=modelmeshserving${runtime_pod_name}
Wait For Pods To Be Ready label_selector=name=modelmesh-serving-${runtime_pod_name}
... namespace=${ns_name} timeout=10s
... namespace=${ns_name} timeout=${timeout}

Create Custom Serving Runtime Using Template By CLI
[Documentation] Use Openshift API to create a custome runtime using template CR
Expand Down
68 changes: 65 additions & 3 deletions ...tests/Tests/400__ods_dashboard/420__model_serving/423__model_serving_customruntimes.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@
[Tags] Smoke ODS-2276
Open Dashboard Settings settings_page=Serving runtimes
Upload Serving Runtime Template runtime_filepath=${OVMS_RUNTIME_FILEPATH}
... serving_platform=multi
Serving Runtime Template Should Be Listed displayed_name=${UPLOADED_OVMS_DISPLAYED_NAME}
... serving_platform=multi

Verify RHODS Admins Can Delete A Custom Serving Runtime Template
[Tags] Smoke ODS-2279
Expand All @@ -31,6 +33,33 @@
Delete Serving Runtime Template displayed_name=${UPLOADED_OVMS_DISPLAYED_NAME}
... press_cancel=${TRUE}
Delete Serving Runtime Template displayed_name=${UPLOADED_OVMS_DISPLAYED_NAME}

Check warning

Code scanning / Robocop

Trailing whitespace at the end of line Warning test

Trailing whitespace at the end of line
Verify RHODS Admins Can Import A Custom Serving Runtime Template For Each Serving Platform

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Verify RHODS Admins Can Import A Custom Serving Runtime Template For Each Serving Platform' is too long (25/20)
[Documentation] Imports a Custom Serving Runtime for each supported serving platform
[Tags] Sanity ODS-2542
[Setup] Generate Runtime YAMLs
Open Dashboard Settings settings_page=Serving runtimes
${RUNTIME_BOTH_FILEPATH}= Set Variable ${RESOURCES_DIRPATH}/csr_both_model.yaml
${RUNTIME_SINGLE_FILEPATH}= Set Variable ${RESOURCES_DIRPATH}/csr_single_model.yaml
${RUNTIME_MULTI_FILEPATH}= Set Variable ${RESOURCES_DIRPATH}/csr_multi_model.yaml
Upload Serving Runtime Template runtime_filepath=${RUNTIME_BOTH_FILEPATH}
... serving_platform=both
Serving Runtime Template Should Be Listed displayed_name=${RUNTIME_BOTH_DISPLAYED_NAME}
... serving_platform=both
Upload Serving Runtime Template runtime_filepath=${RUNTIME_SINGLE_FILEPATH}
... serving_platform=single
Serving Runtime Template Should Be Listed displayed_name=${RUNTIME_SINGLE_DISPLAYED_NAME}
... serving_platform=single
Upload Serving Runtime Template runtime_filepath=${RUNTIME_MULTI_FILEPATH}
... serving_platform=multi
Serving Runtime Template Should Be Listed displayed_name=${RUNTIME_MULTI_DISPLAYED_NAME}
... serving_platform=multi
[Teardown] Run Keywords
... Delete Serving Runtime Template From CLI displayed_name=${RUNTIME_BOTH_DISPLAYED_NAME}
... AND
... Delete Serving Runtime Template From CLI displayed_name=${RUNTIME_SINGLE_DISPLAYED_NAME}
... AND
... Delete Serving Runtime Template From CLI displayed_name=${RUNTIME_MULTI_DISPLAYED_NAME}

Verify RHODS Users Can Deploy A Model Using A Custom Serving Runtime
[Documentation] Verifies that a model can be deployed using only the UI.
Expand All @@ -54,11 +83,11 @@
Serve Model project_name=${PRJ_TITLE} model_name=${model_name} framework=onnx existing_data_connection=${TRUE}
... data_connection_name=model-serving-connection model_path=mnist-8.onnx
Wait Until Runtime Pod Is Running server_name=${MODEL_SERVER_NAME}
... project_title=${PRJ_TITLE}
... project_title=${PRJ_TITLE} timeout=15s
Verify Model Status ${model_name} success
Verify Model Inference ${model_name} ${inference_input} ${exp_inference_output} token_auth=${TRUE}
... project_title=${PRJ_TITLE}

Check warning

Code scanning / Robocop

Trailing whitespace at the end of line Warning test

Trailing whitespace at the end of line
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remove

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why? according to the linter, there must be 2 lines btw sections

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I meant to remove the trailing spaces 🙂

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh, sorry. I'll remove it in some next PRs (I have in my backlog new tests in this suite after 2.5 release)


*** Keywords ***
Custom Serving Runtime Suite Setup
Expand All @@ -81,5 +110,38 @@
Log message=Creating the necessary Serving Runtime as part of Test Setup.
Open Dashboard Settings settings_page=Serving runtimes
Upload Serving Runtime Template runtime_filepath=${OVMS_RUNTIME_FILEPATH}
Serving Runtime Template Should Be Listed displayed_name=${UPLOADED_OVMS_DISPLAYED_NAME}
... serving_platform=multi
Serving Runtime Template Should Be Listed displayed_name=${UPLOADED_OVMS_DISPLAYED_NAME}
... serving_platform=multi
END

Generate Runtime YAMLs

Check warning

Code scanning / Robocop

Keyword '{{ keyword_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Keyword 'Generate Runtime YAMLs' has too many keywords inside (13/10)
[Documentation] Generates three different Custom Serving Runtime YAML files
... starting from OVMS one. Each YAML will be used for a different
... supported serving platform (single model, multi model, both)
Set Suite Variable ${RUNTIME_BOTH_FILEPATH} ${RESOURCES_DIRPATH}/csr_both_model.yaml
Set Suite Variable ${RUNTIME_SINGLE_FILEPATH} ${RESOURCES_DIRPATH}/csr_single_model.yaml
Set Suite Variable ${RUNTIME_MULTI_FILEPATH} ${RESOURCES_DIRPATH}/csr_multi_model.yaml
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These three ${RESOURCES_DIRPATH}/csr_* are used on two places. Maybe it would make sense to create a global variables in this file for them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these will be used only by one test, hence I made these suite variable only when running that particular test

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I didn't mean to make it global as that much global, just to define it only once in the file, but I don't insist, that's a small thing 🙂

Set Suite Variable ${RUNTIME_BOTH_DISPLAYED_NAME} ODS-CI CSR - Both Platforms
Set Suite Variable ${RUNTIME_SINGLE_DISPLAYED_NAME} ODS-CI CSR - Single model Platform
Set Suite Variable ${RUNTIME_MULTI_DISPLAYED_NAME} ODS-CI CSR - Multi models Platform
Copy File ${OVMS_RUNTIME_FILEPATH} ${RUNTIME_BOTH_FILEPATH}
Copy File ${OVMS_RUNTIME_FILEPATH} ${RUNTIME_SINGLE_FILEPATH}
Copy File ${OVMS_RUNTIME_FILEPATH} ${RUNTIME_MULTI_FILEPATH}
${rc} ${out}= Run And Return Rc And Output
... yq -i '.metadata.annotations."openshift.io/display-name" = "${RUNTIME_BOTH_DISPLAYED_NAME}"' ${RUNTIME_BOTH_FILEPATH}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (128/120)
Should Be Equal As Integers ${rc} ${0} msg=${out}
${rc} ${out}= Run And Return Rc And Output

Check warning

Code scanning / Robocop

Local variable '{{ name }}' is overwritten before usage Warning test

Local variable '${rc}' is overwritten before usage

Check warning

Code scanning / Robocop

Local variable '{{ name }}' is overwritten before usage Warning test

Local variable '${out}' is overwritten before usage
... yq -i '.metadata.name = "ods-ci-both"' ${RUNTIME_BOTH_FILEPATH}
${rc} ${out}= Run And Return Rc And Output
... yq -i '.metadata.annotations."openshift.io/display-name" = "${RUNTIME_SINGLE_DISPLAYED_NAME}"' ${RUNTIME_SINGLE_FILEPATH}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (132/120)
Should Be Equal As Integers ${rc} ${0} msg=${out}
${rc} ${out}= Run And Return Rc And Output

Check warning

Code scanning / Robocop

Local variable '{{ name }}' is overwritten before usage Warning test

Local variable '${rc}' is overwritten before usage

Check warning

Code scanning / Robocop

Local variable '{{ name }}' is overwritten before usage Warning test

Local variable '${out}' is overwritten before usage
... yq -i '.metadata.name = "ods-ci-single"' ${RUNTIME_SINGLE_FILEPATH}
${rc} ${out}= Run And Return Rc And Output

Check warning

Code scanning / Robocop

{{ bad_indent_msg }} Warning test

Line is over-indented
... yq -i '.metadata.annotations."openshift.io/display-name" = "${RUNTIME_MULTI_DISPLAYED_NAME}"' ${RUNTIME_MULTI_FILEPATH}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (130/120)

Check warning

Code scanning / Robocop

Continuation marker should be aligned with starting row Warning test

Continuation marker should be aligned with starting row
Should Be Equal As Integers ${rc} ${0} msg=${out}
${rc} ${out}= Run And Return Rc And Output
... yq -i '.metadata.name = "ods-ci-multi"' ${RUNTIME_MULTI_FILEPATH}
Should Be Equal As Integers ${rc} ${0} msg=${out}

Loading