Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parametrize dragon install #703

Merged
merged 16 commits into from
Sep 12, 2024
Merged

Parametrize dragon install #703

merged 16 commits into from
Sep 12, 2024

Conversation

ankona
Copy link
Contributor

@ankona ankona commented Sep 11, 2024

Parameterize the smart build --dragon command to enable specification of a fork/repository and package version

  • add parameter --dragon-repo
  • add parameter --dragon-version

Sample usage:

  • Get latest version from private fork

    GH_TOKEN=xxxxx smart build --dragon-repo ankona/dragonfork

  • Get specific version from private fork

    GH_TOKEN=xxxxx smart build --dragon-repo ankona/dragonfork --dragon-version 0.10

  • Get specific version from public fork

    • GH_TOKEN=xxxxx smart build --dragon-repo dragonhpc/dragon --dragon-version 0.10
    • GH_TOKEN=xxxxx smart build --dragon-version 0.10
    • smart build --dragon-repo dragonhpc/dragon --dragon-version 0.10
    • smart build --dragon-version 0.10

manual test results

  1. OK - smart build --dragon
  [1] % smart build --dragon
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK_FAIL - smart build --dragon-repo dragonhpc/dragon-nightly
  [1] % smart build --dragon-repo dragonhpc/dragon-nightly
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] ERROR An access token must be available to access dragonhpc/dragon-nightly. Set the `GH_TOKEN` env var to pass your access token.
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!
  1. OK_FAIL - smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.9
  • no gh_token supplied to connect to private repo, no version 0.9 in that repo
  [130] % smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.9 
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] ERROR An access token must be available to access dragonhpc/dragon-nightly. Set the `GH_TOKEN` env var to pass your access token.
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!
  1. OK_FAIL - GH_TOKEN=xxx smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.9
  • no version 0.9 in that repo
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] WARNING Please specify a dragon version (e.g. 0.9) of an asset available in the repository:
          dragon-0.10-py3.10.10-ff4c77a60.tar.gz
          dragon-0.10-py3.11.5-ff4c77a60.tar.gz
          dragon-0.10-py3.9.4.1-ff4c77a60.tar.gz
          dragondocs-0.10-ff4c77a60.tar.gz
  [SmartSim] WARNING No dragon runtime asset available to install
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!
  1. OK FAIL - smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.10
  • no gh_token supplied to connect to private repo
  [1] % smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.10
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] ERROR An access token must be available to access dragonhpc/dragon-nightly. Set the `GH_TOKEN` env var to pass your access token.
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!
  1. OK - GH_TOKEN=xxx smart build --dragon-repo dragonhpc/dragon-nightly --dragon-version 0.10
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] WARNING Platform-specific package not found. Using dragon-0.10-py3.11.5-ff4c77a60.tar.gz
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon-nightly/releases/assets/190659388")
  [SmartSim] DEBUG Retrieved asset dragon-0.10-py3.11.5-ff4c77a60.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon-nightly/releases/assets/190659388
  [SmartSim] DEBUG Extracted dragon-0.10-py3.11.5-ff4c77a60.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/190659388
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/190659388/dragon-0.10-py3.11.5-ff4c77a60.tar.gz/dragon-0.10/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/190659388/dragon-0.10-py3.11.5-ff4c77a60.tar.gz/dragon-0.10/dragon-0.10-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/190659388
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK_FAIL - smart build --dragon-version 0.10
  • no v0.10 in that repo
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] WARNING Please specify a dragon version (e.g. 0.9) of an asset available in the repository:
          dragon-0.9-py3.10.10-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.10.10-ec3fc0f8a.tar.gz
          dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.11.5-ec3fc0f8a.tar.gz
          dragon-0.9-py3.9.4.1-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.9.4.1-ec3fc0f8a.tar.gz
  [SmartSim] WARNING No dragon runtime asset available to install
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!
  1. OK - smart build --dragon-version 0.9
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - export GH_TOKEN=xxx smart build --dragon-version 0.9
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - smart build --dragon-repo dragonhpc/dragon
  • handle lower case fine!
  [1] % smart build --dragon-repo dragonhpc/dragon
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - GH_TOKEN=xxx smart build --dragon-repo dragonhpc/dragon
  • token not required for public, but works:
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - smart build --dragon-repo dragonhpc/dragon --dragon-version 0.9
  [1] % smart build --dragon-repo dragonhpc/dragon --dragon-version 0.9 
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - GH_TOKEN=xxx smart build --dragon-repo dragonhpc/dragon --dragon-version 0.9
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK - 1. smart build --dragon --dragon-version 0.9
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] DEBUG Retrieved asset metadata: GitReleaseAsset(url="https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358")
  [SmartSim] DEBUG Retrieved asset dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz metadata from https://api.github.com/repos/DragonHPC/dragon/releases/assets/165524358
  [SmartSim] DEBUG Extracted dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz to /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] DEBUG Installing packages:
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/dragon-0.9-cp311-cp311-linux_x86_64.whl
          /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358/dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz/dragon-0.9/pycapnp-2.0.0-cp311-cp311-linux_x86_64.whl
  [SmartSim] DEBUG Deleted temporary files in: /lus/bnchlu1/mcbridch/code/ss/smartsim/_core/.dragon/165524358
  [SmartSim] INFO Dragon installation complete
  [SmartSim] INFO Redis build complete!
  1. OK_FAIL - 1. smart build --dragon --dragon-version 0.10
  [1] % smart build --dragon --dragon-version 0.10
  [SmartSim] INFO Running SmartSim build process...
  [SmartSim] INFO Checking requested versions...
  [SmartSim] DEBUG Checking for build tools...
  [SmartSim] WARNING Please specify a dragon version (e.g. 0.9) of an asset available in the repository:
          dragon-0.9-py3.10.10-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.10.10-ec3fc0f8a.tar.gz
          dragon-0.9-py3.11.5-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.11.5-ec3fc0f8a.tar.gz
          dragon-0.9-py3.9.4.1-CRAYEX-ce895d2de.tar.gz
          dragon-0.9-py3.9.4.1-ec3fc0f8a.tar.gz
  [SmartSim] WARNING No dragon runtime asset available to install
  [SmartSim] WARNING Dragon installation failed
  [SmartSim] INFO Redis build complete!

  1. OK - smart build -h
  --dragon-repo DRAGON_REPO
                        Specify a git repo containing dragon release assets (e.g. DragonHPC/dragon)
  --dragon-version DRAGON_VERSION
                        Specify the dragon version to install (e.g. 0.9)

@AlyssaCote AlyssaCote self-requested a review September 11, 2024 20:16
Copy link
Contributor

@AlyssaCote AlyssaCote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks really good to me! Just a couple docstring fix requests and a question about to what extent we should document this, but once the tests pass I think this should be good to go.

@al-rigazzi
Copy link
Collaborator

@AlyssaCote do you think we need to add the dragon-config step to install?

@AlyssaCote
Copy link
Contributor

@al-rigazzi I believe that's only necessary if we want to ensure the use of HSTA, otherwise the fallback is TCP. So as of now I don't think we need it for tests? But I could be wrong!

@al-rigazzi
Copy link
Collaborator

@AlyssaCote yeah, not for tests, you're right. I guess, with the current implementation, if a user wanted to use HSTAs, the dragon-config step with appropriate parameters should be run from the installation directories, do you agree?

@AlyssaCote
Copy link
Contributor

@al-rigazzi Ohhh good point. I was only thinking of the green checks for our github actions! Yeah you're right we do need a way for users to enable HSTA.

@ankona ankona requested a review from AlyssaCote September 12, 2024 17:15
Copy link
Contributor

@AlyssaCote AlyssaCote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Can't wait to see all the green check marks!!

@ankona ankona merged commit 3e9bffa into mli-feature Sep 12, 2024
39 of 42 checks passed
@ankona ankona deleted the parametrize-dragon-install branch September 25, 2024 16:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants