Update A3 mega blueprint to use Slurm-GCP 6.5.12 #2763
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tpdownes
added
the
release-improvements
tpdownes
force-pushed
the
a3mega_slurm_update
branch
from
ce83dfb to
7b286e7
Compare
Fix the versions for local google guest VM services so that they do not upgrade to versions that are known to have boot-time issues for the following combination: - building image using Packer on a build VM without local NVME devices - final image used on a VM with local NVME devices In this combination, network configurations persist that do not match the final naming conventions of the network interfaces because of differing PCI bus layout.
tpdownes
force-pushed
the
a3mega_slurm_update
branch
from
7b286e7 to
bc974c8
Compare
|
Ran NCCL tests and verified performance is as per our expectations. |
harshthakkar01
approved these changes
Jul 17, 2024
tpdownes
merged commit f537091
into
GoogleCloudPlatform:release-candidate
10 of 51 checks passed
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the A3 Mega slurm image building solution in 2 ways:
The first change aids alignment of Terraform code with the Python autoscaling code added during image building. We previously (#2746) re-enabled
enable_devel: truewhich should avoid any problems, but alignment is good.The second change is critical to address problems introduced by GoogleCloudPlatform/guest-agent#401, which perfectly describes the A3 Mega solution: an ethernet device naming problem stemming from building an image on a Packer VM without local SSD and then provisioning A3 Mega VMs (with local SSD) using the image. The resulting compute nodes that cannot join the network or the cluster.
Before the PR, we observe <10% of nodes joining initially and rarely managing to succeed after every ~1000s retry cycle. After the PR, we see 100% of nodes joining on the first try.
Future work can be done to update the base debian-12 image (from April 2024) to one that includes the fix to GoogleCloudPlatform/guest-agent#401.
Submission Checklist
Please take the following actions before submitting this pull request.