feat(gpu): add shared nvidia boost clock logic

[ndbaker1]

Issue #, if available:

Description of changes:

adding in default clocks rates for nvidia GPUs queries directly from nvidia-smi.
the logic is used for both al2 and al23.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Testing Done

---

build an al2023 ami using:

make k8s=1.31 ami_name=al23-nvidia-clocks-unit os_distro=al2023 enable_accelerator=nvidia enable_efa=true nvidia_driver_major_version=560

launched an g5.8xlarge and checked the status of the service:

[root@ip-172-31-6-36 bin]# systemctl status set-nvidia-clocks
● set-nvidia-clocks.service - Configure NVIDIA GPU clock rate
     Loaded: loaded (/etc/systemd/system/set-nvidia-clocks.service; enabled; preset: disabled)
     Active: active (exited) since Sat 2024-10-19 21:16:51 UTC; 1min 29s ago
   Main PID: 6301 (code=exited, status=0/SUCCESS)
        CPU: 164ms

Oct 19 21:16:50 ip-172-31-6-36.us-west-2.compute.internal sudo[6311]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/nvidia-smi --persistence-mode=1
Oct 19 21:16:50 ip-172-31-6-36.us-west-2.compute.internal sudo[6311]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 19 21:16:50 ip-172-31-6-36.us-west-2.compute.internal sudo[6311]: pam_unix(sudo:session): session closed for user root
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6378]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/nvidia-smi --auto-boost-default=0
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6378]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6378]: pam_unix(sudo:session): session closed for user root
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6381]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/nvidia-smi --applications-clocks 6251,1710
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6381]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal sudo[6381]: pam_unix(sudo:session): session closed for user root
Oct 19 21:16:51 ip-172-31-6-36.us-west-2.compute.internal systemd[1]: Finished set-nvidia-clocks.service - Configure NVIDIA GPU clock rate.

---

built an al2 ami with:

make k8s=1.31 ami_name=al2-nvidia-clocks-unit os_distro=al2

launched an instance with the al2 ami and checked that the unit was not started:

[root@ip-172-31-6-246 bin]# systemctl status set-nvidia-clocks
● set-nvidia-clocks.service - Configure NVIDIA GPU clock rate
   Loaded: loaded (/etc/systemd/system/set-nvidia-clocks.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Mon 2024-10-21 04:30:15 UTC; 1min 32s ago

Oct 21 04:30:14 localhost systemd[1]: Cannot add dependency job for unit set-nvidia-clocks.service, ignoring: Unit not found.

since this actually failed to start due to missing nvidia-persistence, i removed the After and Requires clauses to simulate the FilePath condition:

[root@ip-172-31-6-246 bin]# systemctl status set-nvidia-clocks
● set-nvidia-clocks.service - Configure NVIDIA GPU clock rate
   Loaded: loaded (/etc/systemd/system/set-nvidia-clocks.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
Condition: start condition failed at Mon 2024-10-21 04:35:42 UTC; 1s ago
           ConditionPathExists=/usr/bin/nvidia-smi was not met

Oct 21 04:30:14 localhost systemd[1]: Cannot add dependency job for unit set-nvidia-clocks.service, ignoring: Unit not found.

See this guide for recommended testing for PRs. Some tests may not apply. Completing tests and providing additional validation steps are not required, but it is recommended and may reduce review time and time to merge.

[cartermckinnon]

loooove that

[cartermckinnon]

/ci

[github-actions]

@cartermckinnon roger that! I've dispatched a workflow. 👍

[github-actions]

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variant	Build	Test
1.24 / al2	success ✅	success ✅
1.24 / al2023	success ✅	success ✅
1.25 / al2	success ✅	success ✅
1.25 / al2023	success ✅	success ✅
1.26 / al2	success ✅	success ✅
1.26 / al2023	success ✅	success ✅
1.27 / al2	success ✅	success ✅
1.27 / al2023	success ✅	success ✅
1.28 / al2	success ✅	success ✅
1.28 / al2023	success ✅	success ✅
1.29 / al2	success ✅	success ✅
1.29 / al2023	success ✅	success ✅
1.30 / al2	success ✅	success ✅
1.30 / al2023	success ✅	success ✅
1.31 / al2	success ✅	success ✅
1.31 / al2023	success ✅	success ✅