-
Notifications
You must be signed in to change notification settings - Fork 253
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #2454 from Agenta-AI/docs/playground2-docs
docs(app): updated playground page
- Loading branch information
Showing
3 changed files
with
93 additions
and
53 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
143 changes: 91 additions & 52 deletions
143
docs/docs/prompt-management/04-using-the-playground.mdx
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,81 +1,120 @@ | ||
--- | ||
title: "Using the Playground" | ||
title: "Using the Prompt Playground" | ||
description: "Learn how to use Agenta's LLM playground for prompt engineering, model comparison, and deployment. A powerful alternative to OpenAI playground that supports multiple models and frameworks." | ||
--- | ||
|
||
```mdx-code-block | ||
import { Stream } from '@cloudflare/stream-react'; | ||
import Image from "@theme/IdealImage"; | ||
``` | ||
|
||
The Agenta playground is a comprehensive prompt engineering IDE that helps you develop, test, and deploy LLM applications. Unlike traditional LLM playgrounds, Agenta integrates prompt management, evaluation, and deployment into a single platform, supporting multiple models and frameworks. | ||
|
||
<Stream controls src="fcf2b69dacb3e3a624c09af40c2dc154" height="400px" /> | ||
<br /> | ||
|
||
|
||
The agenta playground is a platform that lets you create, modify, and compare different prompts and configurations for your LLM application. | ||
## Key Features | ||
|
||
## Prompt Templates | ||
- **Multi-Model Support**: Test prompts across different providers including OpenAI GPT models, Anthropic Claude, Google Gemini, Mistral, DeepSeek, and OpenRouter models | ||
- **Framework Integration**: Works seamlessly with popular frameworks like LangChain, LlamaIndex, and CrewAI | ||
- **Version Control**: Track and manage prompt versions with built-in prompt management | ||
- **Model Comparison**: Benchmark different LLM models side by side | ||
- **Parameter Control**: Fine-tune model parameters including temperature, top-k, and presence penalty | ||
- **Environment Management**: Deploy configurations to development, staging, and production environments | ||
|
||
An LLM takes a prompt and returns a completion. To make the prompt reusable, we need to be able to dynamically modify it based on a set of inputs. A prompt template is a prompt that can be dynamically modified based on a set of inputs. | ||
## Creating and Managing Prompts | ||
|
||
For instance, a prompt to create a blog post might look like this: | ||
### Prompt Templates | ||
|
||
Prompt templates form the foundation of reusable LLM interactions. They contain variable placeholders that adapt to different inputs, making your prompts dynamic and versatile. | ||
|
||
Example of a prompt template: | ||
``` | ||
Write a blog post about {subject} | ||
Write a blog post about {{subject}} that focuses on {{key_aspect}} | ||
``` | ||
|
||
Here, the input is the subject of the blog post. The prompt can be reused for different subjects. | ||
Agenta uses curly brackets `{{variable}}` for template variables, automatically creating corresponding input fields in the playground interface. | ||
|
||
### Adjusting Model Parameters | ||
|
||
To modify model-specific parameters: | ||
|
||
1. Click on the model name in the configuration panel | ||
2. A settings modal will appear with available parameters | ||
3. Adjust the following settings as needed: | ||
|
||
- **Temperature** (0-2): Controls randomness in the output | ||
- Lower values (0.1-0.4): More focused, deterministic responses | ||
- Higher values (0.7-1.0): More creative, varied outputs | ||
|
||
- **Top-k** (1-100): Limits the pool of tokens the model considers | ||
- Lower values: More focused word choice | ||
- Higher values: Broader vocabulary usage | ||
|
||
- **Presence Penalty** (-2.0 to 2.0): Influences topic diversity | ||
- Positive values: Encourages exploring new topics | ||
- Negative values: Allows topic repetition | ||
|
||
- **Frequency Penalty** (-2.0 to 2.0): Affects word choice | ||
- Positive values: Discourages word repetition | ||
- Negative values: Allows word reuse | ||
|
||
## Comparing Prompts and Models | ||
|
||
Agenta's playground allows you to compare different prompts, model configurations, and LLM providers side by side. This helps you identify the most effective combination for your use case. | ||
|
||
### Creating Multiple Variants | ||
|
||
To compare different configurations: | ||
|
||
The LLM app templates in agenta use the f-string templating language, with prompt variables referenced using a curly bracket syntax. | ||
1. Click the "+Compare" button in the top right corner | ||
2. Select "New Variant" from the dropdown menu | ||
3. Choose a source variant to duplicate (this copies all current parameters) | ||
4. Give your new variant a descriptive name | ||
5. Modify the configuration as needed | ||
|
||
## Adding new inputs | ||
Each variant appears as a separate column in the playground, allowing direct comparison of outputs. | ||
|
||
To add new inputs to the LLM app, navigate to the playground and modify the inputs under "Modify Parameters." | ||
### Side-by-Side Testing | ||
|
||
<Image | ||
className="dark:hidden" | ||
img={require("/images/basic_guides/08_add_new_input_light.png")} | ||
/> | ||
<Image | ||
className="hidden dark:block" | ||
img={require("/images/basic_guides/08_add_new_input_dark.png")} | ||
/> | ||
With multiple variants set up, you can: | ||
- Enter the same input text across all variants | ||
- Compare the generated outputs side by side | ||
- Review response times for each configuration | ||
- Monitor the cost per request for different models | ||
|
||
After adding an input, incorporate it into the prompt templates using the curly bracket syntax. | ||
This parallel testing helps you understand how different prompts and models handle the same input. | ||
|
||
## Creating a variant | ||
### Version Control and Deployment | ||
|
||
You can create a new variant of an application by clicking on "Add Variant" in the "Side-by-side" view or the "+" tab in the "Tab view". | ||
The playground integrates with Agenta's [prompt management system](/prompt-management/overview) to track and deploy changes: | ||
|
||
<Image | ||
className="dark:hidden" | ||
img={require("/images/basic_guides/09_add_variant_light.png")} | ||
/> | ||
<Image | ||
className="hidden dark:block" | ||
img={require("/images/basic_guides/09_add_variant_dark.png")} | ||
/> | ||
1. Click "Commit" to save a new version of your variant | ||
2. Changes remain in development until explicitly deployed | ||
3. Use "Deploy" to push configurations to specific environments (development, staging, production) | ||
|
||
You'll then see a window where you can select the source variant to use as a template and provide a new name for your variant. | ||
## Working with Test Sets | ||
|
||
After creating a variant, you can alter its parameters. Remember to save your changes before deploying the variant. | ||
### Loading Existing Test Sets | ||
|
||
## Testing a variant | ||
You can import test sets that you've created through: | ||
|
||
Fill in the inputs in the cards and click "Run" to test a variant dynamically. | ||
- CSV uploads | ||
- Programmatic creation | ||
- Web UI input | ||
|
||
You can also load a test set to populate the playground with a set of inputs. Then You can click Run all to run all the inputs in the test set. | ||
The test set must contain columns that match your prompt template variables. | ||
|
||
<Image | ||
className="dark:hidden" | ||
img={require("/images/basic_guides/10_testing_a_variant_light.png")} | ||
/> | ||
<Image | ||
className="hidden dark:block" | ||
img={require("/images/basic_guides/10_testing_a_variant_dark.png")} | ||
/> | ||
### Saving Test Cases | ||
|
||
## Comparing variants side by side | ||
To save an interesting test case: | ||
|
||
Click on the "Side-by-side" tab to compare variants. From the dropdowns, select the variants you wish to compare. This allows you to view the results of multiple variants simultaneously. When using a chat application, you can interact with different variants in parallel. | ||
1. Click the three-dot menu on the right side of the test case card | ||
2. Select "Add to Test Set" | ||
3. Choose an existing test set or create a new one | ||
4. Review the input/output mapping (use default settings unless you have specific needs) | ||
5. Preview the changes to your test set | ||
6. Confirm to save | ||
|
||
<Image | ||
className="dark:hidden" | ||
img={require("/images/basic_guides/11_side_by_side_chat_mode_light.png")} | ||
/> | ||
<Image | ||
className="hidden dark:block" | ||
img={require("/images/basic_guides/11_side_by_side_chat_mode_dark.png")} | ||
/> | ||
The Agenta playground serves as a comprehensive prompt engineering environment, offering advantages over traditional playgrounds like OpenAI's by providing integrated version control, deployment capabilities, and support for multiple models and frameworks. Whether you're developing new prompts, comparing model performance, or deploying to production, the playground provides the tools needed for effective LLM application development. |