HOTFIX: make the default number of devices be all the devices seen by… #613
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… cudaGetDeviceCount/hipGetDeviceCount/zeDeviceGet Current code is testing if parsec_device_cuda_enabled_index < 0 (or similar counter for LZ), but parsec_device_cuda_enabled_index is the value returned by parsec_mca_param_reg_int_name() which is always positive, as it returns where that parameter is located in the parameter array. The code wanted to check if testing if parsec_device_cuda_enabled < 0, so that if the user passes -1 or if the user omits to define the MCA parameter (default value is -1), then parsec takes the number of devices discovered by CUDA/HIP/LZ, as is claimed in the documentation of the MCA parameter.
|
Looking at the failures, they occur because now we try to initialize CUDA for all tests, if there is a GPU available. On 'mp' runs, or more generally on all runs that run more than 1 rank, when they are run on 'guyot', we actually oversubscribe the node, so of course all processes try to grab all the memory on the same GPU, and some fail... I can add And I guess we should deactivate the multi-rank GPU test when there is only one GPU to share... Is there a way to detect that we only have 1 node to do the test at the beginning of the CI, and to propagate this information to the testers so they don't try something that cannot succeed? |
abouteiller
approved these changes
Jan 19, 2024
bosilca
approved these changes
Jan 19, 2024
therault
added a commit
to therault/parsec
that referenced
this pull request
Jan 22, 2024
PR ICLDisco#613 made all CI tests initialize the GPU if there is a GPU available. When running in oversubscribe mode, this can lead to falsely failing tests, that fail not because of a software issue, but because of a deployment issue (multiple processes trying to allocate 90% of the GPU memory at the same time). In general, since we don't know if the GPU will be used or not, we should not preemptively allocate all the memory on it. This PR makes memory allocation lazy: it is delayed until we do try to use some GPU memory. The drawback is that the first GPU task will also pay the cost of a large cuda_malloc / zmalloc etc...
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
… cudaGetDeviceCount/hipGetDeviceCount/zeDeviceGet
Current code is testing if parsec_device_cuda_enabled_index < 0 (or similar counter for LZ), but parsec_device_cuda_enabled_index is the value returned by parsec_mca_param_reg_int_name() which is always positive, as it returns where that parameter is located in the parameter array.
The code wanted to check if testing if parsec_device_cuda_enabled < 0, so that if the user passes -1 or if the user omits to define the MCA parameter (default value is -1), then parsec takes the number of devices discovered by CUDA/HIP/LZ, as is claimed in the documentation of the MCA parameter.