Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Upgrade test scenarios for Distributed workloads #1491

Merged
merged 1 commit into from
Jun 25, 2024
Merged

Add Upgrade test scenarios for Distributed workloads #1491

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions ods_ci/tests/Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@
${GO_TEST_TIMEOUT} 1h
${JOB_GO_BIN} %{WORKSPACE=.}/go-bin
${GO_JUNIT_REPORT_TOOL} github.com/jstemmer/go-junit-report/v2@latest
${VIRTUAL_ENV_NAME} venv3.9
${CODEFLARE-SDK-API_URL} %{CODEFLARE-SDK-API_URL=https://api.github.com/repos/project-codeflare/codeflare-sdk/releases/latest}
${CODEFLARE-SDK_DIR} codeflare-sdk-upgrade
${CODEFLARE-SDK_REPO_URL} %{CODEFLARE-SDK_REPO_URL=https://github.com/project-codeflare/codeflare-sdk.git}


*** Keywords ***
Expand Down Expand Up @@ -53,3 +57,53 @@
IF ${result.rc} != 0
FAIL Failed to convert Go test results to Junit
END

Run Codeflare Upgrade Tests
Srihari1192 marked this conversation as resolved.
Show resolved Hide resolved
[Documentation] Run codeflare upgrade tests by cloning codeflare-sdk repo
[Arguments] ${TEST_NAME}
Dismissed Show dismissed Hide dismissed
${latest_tag} = Run Process curl -s "${CODEFLARE-SDK-API_URL}" | grep '"tag_name":' | cut -d '"' -f 4
... shell=True stderr=STDOUT
Log To Console codeflare-sdk latest tag is : ${latest_tag.stdout}
IF ${latest_tag.rc} != 0
Dismissed Show dismissed Hide dismissed
FAIL Unable to fetch codeflare-sdk latest tag
END

Clone Git Repository ${CODEFLARE-SDK_REPO_URL} ${latest_tag.stdout} ${CODEFLARE-SDK_DIR}

${result} = Run Process virtualenv -p python3.9 ${VIRTUAL_ENV_NAME}
... shell=true stderr=STDOUT
Log To Console ${result.stdout}
IF ${result.rc} != 0
Dismissed Show dismissed Hide dismissed
FAIL Unable to setup Python virtual environment
END

${result} = Run Process source ${VIRTUAL_ENV_NAME}/bin/activate && cd ${CODEFLARE-SDK_DIR} && poetry env use 3.9 && poetry install --with test,docs && poetry run pytest -v -s ./tests/upgrade/raycluster_sdk_upgrade_test.py::${TEST_NAME} --timeout\=300 && deactivate

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (272/120)
... shell=true
... stderr=STDOUT
Log To Console ${result.stdout}
IF ${result.rc} != 0
Dismissed Show dismissed Hide dismissed
Dismissed Show dismissed Hide dismissed
FAIL Running test ${TEST_NAME} failed
END

Codeflare Upgrade Tests Teardown
[Documentation] Run codeflare upgrade tests by cloning codeflare-sdk repo
[Arguments] ${project_name} ${project_created}
${result} = Run Process rm -rf ${VIRTUAL_ENV_NAME}
... shell=true stderr=STDOUT
Log To Console ${result.stdout}
IF ${result.rc} != 0
Dismissed Show dismissed Hide dismissed
FAIL Unable to cleanup Python virtual environment
END

${result} = Run Process rm -rf ${CODEFLARE-SDK_DIR}
... shell=true stderr=STDOUT
Log To Console ${result.stdout}
IF ${result.rc} != 0

Check notice

Code scanning / Robocop

'{{ block_name }}' condition can be simplified Note test

'IF' condition can be simplified
FAIL Unable to cleanup directory ${CODEFLARE-SDK_DIR}
END

IF ${project_created} == True Run Keywords

Check notice

Code scanning / Robocop

'{{ block_name }}' condition can be simplified Note test

'INLINE IF' condition can be simplified
... Run oc delete project ${project_name} AND
... Run Process oc delete LocalQueue local-queue-mnist -n ${project_name} &
... oc delete ClusterQueue cluster-queue-mnist &
... oc delete ResourceFlavor default-flavor-mnist shell=True
Comment on lines +105 to +109

Check warning

Code scanning / Robocop

Avoid splitting inline IF to multiple lines Warning test

Avoid splitting inline IF to multiple lines
5 changes: 4 additions & 1 deletion ods_ci/tests/Resources/Page/DistributedWorkloads/WorkloadMetricsUI.resource
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,10 +125,13 @@ Get Memory Requested
${memory_requested}= Replace String ${output} Mi ${EMPTY}
${memory_requested_num}= Convert To Number ${memory_requested}
${memory_requested_gb_value}= Evaluate round( ${memory_requested_num} / 1024, 3)
ELSE
ELSE IF "${workload_type}" == "RayCluster"
${memory_requested}= Replace String ${output} Ki ${EMPTY}
${memory_requested_num}= Convert To Number ${memory_requested}
${memory_requested_gb_value}= Evaluate round( ${memory_requested_num} / (1024*1024), 3)
ELSE
${memory_requested_num}= Convert To Number ${output}
${memory_requested_gb_value}= Evaluate round( ${memory_requested_num} / (1024*1024*1024), 3)
END
RETURN ${memory_requested_gb_value}

Expand Down
37 changes: 37 additions & 0 deletions ods_ci/tests/Tests/100__deploy/120__upgrades/120__pre_upgrades.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@
Resource ../../../Resources/Page/OCPDashboard/Builds/Builds.robot
Resource ../../../Resources/Page/HybridCloudConsole/OCM.robot
Resource ../../../Resources/CLI/ModelServing/modelmesh.resource
Resource ../../../Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
Resource ../../../Resources/Page/DistributedWorkloads/WorkloadMetricsUI.resource
Suite Setup Dashboard Suite Setup
Suite Teardown RHOSi Teardown

Expand All @@ -34,6 +36,8 @@
${MODEL_NAME}= test-model
${MODEL_CREATED}= ${FALSE}
${RUNTIME_NAME}= Model Serving Test
${DW_PROJECT_CREATED}= False


*** Test Cases ***
Set PVC Size Via UI
Expand Down Expand Up @@ -119,6 +123,39 @@
Page Should Contain Element //tr[@id='caikit-runtime']
[Teardown] Dashboard Test Teardown

Verify Distributed Workload Metrics Resources By Creating Ray Cluster Workload

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Verify Distributed Workload Metrics Resources By Creating Ray Cluster Workload' is too long (33/20)

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Verify Distributed Workload Metrics Resources By Creating Ray Cluster Workload' has too many keywords inside (20/10)
[Documentation] Creates the Ray Cluster and verify resource usage
[Tags] Upgrade
${PRJ_UPGRADE} Set Variable test-ns-rayupgrade

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '' instead
${JOB_NAME} Set Variable mnist

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '' instead
Run Codeflare Upgrade Tests TestMNISTRayClusterUp
Launch Dashboard ${TEST_USER.USERNAME} ${TEST_USER.PASSWORD} ${TEST_USER.AUTH_TYPE}
... ${ODH_DASHBOARD_URL} ${BROWSER.NAME} ${BROWSER.OPTIONS}
Open Distributed Workload Metrics Home Page
Select Distributed Workload Project By Name ${PRJ_UPGRADE}
Set Global Variable ${DW_PROJECT_CREATED} True

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Global Variable can be replaced with VAR
Select Refresh Interval 15 seconds
Wait Until Element Is Visible ${DISTRIBUITED_WORKLOAD_RESOURCE_METRICS_TITLE_XP} timeout=20
Wait Until Element Is Visible xpath=//*[text()="Running"] timeout=30

${cpu_requested} = Get CPU Requested ${PRJ_UPGRADE} local-queue-mnist
${memory_requested} = Get Memory Requested ${PRJ_UPGRADE} local-queue-mnist Upgrade
Check Requested Resources Chart ${PRJ_UPGRADE} ${cpu_requested} ${memory_requested}
Check Requested Resources ${PRJ_UPGRADE} ${CPU_SHARED_QUOTA}
... ${MEMEORY_SHARED_QUOTA} ${cpu_requested} ${memory_requested} RayCluster

Check Distributed Workload Resource Metrics Status ${JOB_NAME} Running
Check Distributed Worklaod Status Overview ${JOB_NAME} Running
... All pods were ready or succeeded since the workload admission

Click Button ${PROJECT_METRICS_TAB_XP}
Check Distributed Workload Resource Metrics Chart ${PRJ_UPGRADE} ${cpu_requested}
... ${memory_requested} RayCluster ${JOB_NAME}

[Teardown] Run Keyword If Test Failed
... Codeflare Upgrade Tests Teardown ${PRJ_UPGRADE} ${DW_PROJECT_CREATED}


*** Keywords ***
Dashboard Suite Setup
[Documentation] Basic suite setup
Expand Down
37 changes: 36 additions & 1 deletion ods_ci/tests/Tests/100__deploy/120__upgrades/122__post_ugrade.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,8 @@
Resource ../../../Resources/Page/OCPDashboard/Pods/Pods.robot
Resource ../../../Resources/Page/OCPDashboard/Builds/Builds.robot
Resource ../../../Resources/Page/HybridCloudConsole/OCM.robot

Resource ../../../Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
Resource ../../../Resources/Page/DistributedWorkloads/WorkloadMetricsUI.resource

*** Variables ***
${S_SIZE} 25
Expand All @@ -30,6 +31,7 @@
${MODEL_NAME}= test-model
${MODEL_CREATED}= ${FALSE}
${RUNTIME_NAME}= Model Serving Test
${DW_PROJECT_CREATED}= False


*** Test Cases ***
Expand Down Expand Up @@ -145,6 +147,39 @@
Delete Serving Runtime Template From CLI By Runtime Name OR Display Name runtime_name=caikit-runtime
[Teardown] Dashboard Test Teardown

Verify Ray Cluster Exists And Monitor Workload Metrics By Submitting Ray Job After Upgrade

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' is too long ({{ test_length }}/{{ allowed_length }}) Warning test

Test case 'Verify Ray Cluster Exists And Monitor Workload Metrics By Submitting Ray Job After Upgrade' is too long (34/20)

Check warning

Code scanning / Robocop

Test case '{{ test_name }}' has too many keywords inside ({{ keyword_count }}/{{ max_allowed_count }}) Warning test

Test case 'Verify Ray Cluster Exists And Monitor Workload Metrics By Submitting Ray Job After Upgrade' has too many keywords inside (21/10)
[Documentation] check the Ray Cluster exists , submit ray job and verify resource usage after upgrade
[Tags] Upgrade
${PRJ_UPGRADE} Set Variable test-ns-rayupgrade
${LOCAL_QUEUE} Set Variable local-queue-mnist
${JOB_NAME} Set Variable mnist
Run Codeflare Upgrade Tests TestMnistJobSubmit
Set Global Variable ${DW_PROJECT_CREATED} True

Check notice

Code scanning / Robocop

{{ set_variable_keyword }} can be replaced with VAR Note test

Set Global Variable can be replaced with VAR
Launch Dashboard ${TEST_USER.USERNAME} ${TEST_USER.PASSWORD} ${TEST_USER.AUTH_TYPE}
... ${ODH_DASHBOARD_URL} ${BROWSER.NAME} ${BROWSER.OPTIONS}
Open Distributed Workload Metrics Home Page
Select Distributed Workload Project By Name ${PRJ_UPGRADE}
Select Refresh Interval 15 seconds
Wait Until Element Is Visible ${DISTRIBUITED_WORKLOAD_RESOURCE_METRICS_TITLE_XP} timeout=20
Wait Until Element Is Visible xpath=//*[text()="Running"] timeout=30

${cpu_requested} = Get CPU Requested ${PRJ_UPGRADE} ${LOCAL_QUEUE}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got ' =' instead
${memory_requested} = Get Memory Requested ${PRJ_UPGRADE} ${LOCAL_QUEUE} Upgrade

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got ' =' instead
Check Requested Resources Chart ${PRJ_UPGRADE} ${cpu_requested} ${memory_requested}
Check Requested Resources ${PRJ_UPGRADE} ${CPU_SHARED_QUOTA}
... ${MEMEORY_SHARED_QUOTA} ${cpu_requested} ${memory_requested} RayCluster

Check Distributed Workload Resource Metrics Status ${JOB_NAME} Running
Check Distributed Worklaod Status Overview ${JOB_NAME} Running
... All pods were ready or succeeded since the workload admission

Click Button ${PROJECT_METRICS_TAB_XP}
Check Distributed Workload Resource Metrics Chart ${PRJ_UPGRADE} ${cpu_requested}
... ${memory_requested} RayCluster ${JOB_NAME}

[Teardown] Codeflare Upgrade Tests Teardown ${PRJ_UPGRADE} ${DW_PROJECT_CREATED}


*** Keywords ***
Dashboard Suite Setup
[Documentation] Basic suite setup
Expand Down
Loading