Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix benchmarking tests #7461

Merged
merged 10 commits into from
Aug 10, 2024
Merged

Fix benchmarking tests #7461

merged 10 commits into from
Aug 10, 2024

Conversation

pskiran1
Copy link
Member

@pskiran1 pskiran1 commented Jul 22, 2024

What does the PR do?

  • Fix L0_perf_tensorrt_llm
  • Fix L0_perf_vllm

Checklist

  • PR title reflects the change and is of format <commit_type>: <Title>
  • Changes are described in the pull request.
  • Related issues are referenced.
  • Populated github labels field
  • Added test plan and verified test passes.
  • Verified that the PR passes existing CI.
  • Verified copyright is correct on all changed files.
  • Added succinct git squash message before merging ref.
  • All template sections are filled out.
  • Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

Check the conventional commit type
box here and add the label to the github PR.

  • build
  • ci
  • docs
  • feat
  • fix
  • perf
  • refactor
  • revert
  • style
  • test

Related PRs:

Where should the reviewer start?

Test plan:

  • CI Pipeline ID: 16792324

Caveats:

Background

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

  • closes GitHub issue: #xxx

@pskiran1 pskiran1 marked this pull request as ready for review July 22, 2024 07:00
@pskiran1 pskiran1 requested review from tanmayv25 and mc-nv July 22, 2024 07:01
@nv-kmcgill53
Copy link
Contributor

It's out of scope for this PR, but it looks like there are some functions like kill_server which could be pulled up into the qa/common/util.sh script for common use.

else
echo "Installed Open MPI version is not less than 5.0.1. Skipping the upgrade."
echo "The installed Open MPI version ($CURRENT_VERSION) is 5.0.1 or higher. Skipping the upgrade."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DLFW uses specific version of Open MPI, by mocking and upgrading version within test we cause precedent that we start testing something that is won't be shipped to the customers.

@pskiran1 pskiran1 requested a review from mc-nv August 8, 2024 04:40
@tanmayv25 tanmayv25 requested a review from krishung5 August 8, 2024 18:30
cd $TENSORRTLLM_BACKEND_DIR && git lfs install && git submodule update --init --recursive
}

# Update Open MPI to a version compatible with SLURM.
function upgrade_openmpi {
cd /tmp/
local CURRENT_VERSION=$(mpirun --version 2>&1 | awk '/Open MPI/ {gsub(/rc[0-9]+/, "", $NF); print $NF}')

if [ -n "$CURRENT_VERSION" ] && dpkg --compare-versions "$CURRENT_VERSION" lt "5.0.1"; then
# Uninstall the current version of Open MPI
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand that this is due to the SLURM MPI limitation, but agree with Misha's comment that we should be using the exact same version of the one shipped in the container. Can be future enhancement.

Copy link
Contributor

@krishung5 krishung5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, great work!

@pskiran1 pskiran1 merged commit 5320009 into main Aug 10, 2024
3 checks passed
@pskiran1 pskiran1 deleted the spolisetty_dlis_6814 branch August 10, 2024 15:51
pvijayakrish added a commit that referenced this pull request Aug 13, 2024
@pskiran1 pskiran1 restored the spolisetty_dlis_6814 branch August 14, 2024 19:32
mc-nv pushed a commit that referenced this pull request Aug 16, 2024
pvijayakrish added a commit that referenced this pull request Jan 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants