Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(plugin-ai): Add 'ai:models:info' command #11

Merged
merged 22 commits into from
Sep 28, 2024

Conversation

eablack
Copy link
Contributor

@eablack eablack commented Sep 20, 2024

Description

This PR implements the ai:models:info command with tests according to the design outlined in the UX design doc.

How to test

Prepare your environment for testing

  • Fetch this branch.
  • Run yarn && yarn build.
  • Set the HEROKU_INFERENCE_HOST to staging via export HEROKU_INFERENCE_HOST=staging.inference.herokai.com

Actual testing

  1. Verify that help looks ok: ./bin/run ai:models:info --help
  2. Create a test app that will be used to check provisioned model resources against via heroku apps:create test-cli-plugin-ai
  3. Create a test model resource to later check more info about via heroku addons:create inference-staging:claude-3-sonnet -a test-cli-plugin-ai
  4. Repeat step 3 so we can check multiple model resources for a single app
  5. Check for more info about a single model resource via ./bin/run ai:models:info inference-staging:claude-3-sonnet -a test-cli-plugin-ai The resulting output should display as follows:
Avg Performance: Feature to be implemented
Ready:           Feature to be implemented
Tokens In:       Feature to be implemented
Tokens Out:      Feature to be implemented
  1. Check for more info about all model resources on a single app via ./bin/run ai:models:info -a test-cli-plugin-ai The resulting output should display as follows:
Avg Performance: Feature to be implemented
Ready:           Feature to be implemented
Tokens In:       Feature to be implemented
Tokens Out:      Feature to be implemented

Avg Performance: Feature to be implemented
Ready:           Feature to be implemented
Tokens In:       Feature to be implemented
Tokens Out:      Feature to be implemented

Test clean up

  • Destroy test app via heroku apps:destroy -a test-cli-plugin-ai (x2 as you will have 2 model resource attachments)
  • Unset HEROKU_INFERENCE_HOST via unset HEROKU_INFERENCE_HOST

SOC2 Compliance

GUS Work Item

@eablack eablack requested a review from a team as a code owner September 20, 2024 18:37
Copy link
Contributor

@sbosio sbosio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple comments. It seems that copies don't align with the latest approved ones, maybe you were following the API design doc, but we should be following this one instead: https://docs.google.com/document/d/1Fo4M5RqbHQILYYmHKty13QGS8hqK1A69swGczJiuDyg/edit#heading=h.2saci73qrjpz.

README.md Outdated Show resolved Hide resolved
src/commands/ai/models/info.ts Outdated Show resolved Hide resolved
src/commands/ai/models/info.ts Outdated Show resolved Hide resolved
@zwhitfield3 zwhitfield3 changed the title Add info command when instance name is supplied Add 'ai:models:info' command Sep 28, 2024
@zwhitfield3 zwhitfield3 changed the title Add 'ai:models:info' command feat(plugin-ai): Add 'ai:models:info' command Sep 28, 2024
Copy link
Contributor

@sbosio sbosio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! I tested and it works as a charm. I suggest adding a header when multiple model resources are returned to differentiate one from another, but that's all. I'm approving.

@zwhitfield3 zwhitfield3 merged commit 7392833 into main Sep 28, 2024
6 checks passed
@zwhitfield3 zwhitfield3 deleted the eb/add-ai-models-info branch September 28, 2024 01:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants