Verify GPU AcceleratorProfile is created after restarting Dashboard #1103
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add to gpu_deploy.sh script two verification steps after restarting rhods-dashboard deployment: - Wait for up to 3 minutes until the deployment is rolled out - Verify that an AcceleratorProfiles resource is created Signed-off-by: manosnoam <[email protected]>
Robot Results
|
FedeAlonso
requested changes
Jan 9, 2024
| oc rollout status deployment.apps/rhods-dashboard -n redhat-ods-applications --watch --timeout 3m | ||
|
|
||
| echo "Verifying that an AcceleratorProfiles resource was created in redhat-ods-applications" | ||
| oc describe AcceleratorProfiles -n redhat-ods-applications |
There was a problem hiding this comment.
would it be better to check the instance was created?
There was a problem hiding this comment.
Describe checks if the resource was created + describes it, for example:
$▶ oc describe AcceleratorProfiles -n redhat-ods-applications
Name: migrated-gpu
Namespace: redhat-ods-applications
Labels: <none>
Annotations: <none>
API Version: dashboard.opendatahub.io/v1
Kind: AcceleratorProfile
Metadata:
Creation Timestamp: 2024-01-09T15:46:11Z
Generation: 1
Managed Fields:
API Version: dashboard.opendatahub.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
.:
f:displayName:
f:enabled:
f:identifier:
f:tolerations:
Manager: unknown
Operation: Update
Time: 2024-01-09T15:46:11Z
Resource Version: 2416221
UID: ce19e4c7-97eb-46b6-b2f4-cc719e0d8e1b
Spec:
Display Name: NVIDIA GPU
Enabled: true
Identifier: nvidia.com/gpu
Tolerations:
Effect: NoSchedule
Key: nvidia.com/gpu
Operator: Exists
Events: <none>
If the resource is missing it will fail with:
error: the server doesn't have a resource type "AcceleratorProfiles"
The previous implementation of the function did not handle completed pods, so the script hanged for ~45 minutes waiting for completed pods (e.g. nvidia-cuda-validator pods) to become running: ``` 19:24:53 GPU installation seems to be still running ... 20:13:48 nvidia-cuda-validator-2jnxd 0/1 Completed 0 5h50m 20:13:48 nvidia-cuda-validator-nsf98 0/1 Completed 0 5h51m ... 21:11:34 ERROR: Timeout reached while waiting for gpu operator ``` A simple call to `oc wait --for=condition=ready pod -l app=$pod_label` within the function should resolve it. Signed-off-by: manosnoam <[email protected]>
manosnoam
added
verified
|
This has been tested on smoke test 4285. |
|
|
I approve this since this PR doesn't break anything and doesn't add extra additional time. It might be useful for debugging. |
apodhrad
approved these changes
Jan 10, 2024
FedeAlonso
approved these changes
Jan 10, 2024
bdattoma
approved these changes
Jan 10, 2024
|
Note: bunch of those shellcheck warnings could be easily fixed. |
@jstourac oh, I missed your suggestion, since the spell check warnings did not break the Quality Gate... |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add to gpu_deploy.sh script two verification steps after restarting rhods-dashboard deployment.
Notice that the wait for pods running happens before the deployment is restarted.
That is why an additional wait for rollout is necessary to be called after restarting the deployment.