Skip to content

Commit

Permalink
Merge pull request #2454 from Agenta-AI/docs/playground2-docs
Browse files Browse the repository at this point in the history
docs(app): updated playground page
  • Loading branch information
mmabrouk authored Feb 6, 2025
2 parents 6afed9c + 6289031 commit a67bc3f
Show file tree
Hide file tree
Showing 3 changed files with 93 additions and 53 deletions.
2 changes: 1 addition & 1 deletion docs/docs/getting-started/01-introduction.mdx
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
slug: /
title: What is Agenta?
description: "The open-source end-to-end LLMOps platform."
description: "Agenta is an open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM Observability all in one place."
id: introduction
sidebar_position: 0
---
Expand Down
1 change: 1 addition & 0 deletions docs/docs/prompt-management/01-overview.mdx
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
---
title: "Overview"
description: "Learn how to effectively manage, version, and deploy LLM prompts and configurations. Discover how prompt management helps teams collaborate, track changes, and maintain consistency across development and production environments"
---

```mdx-code-block
Expand Down
143 changes: 91 additions & 52 deletions docs/docs/prompt-management/04-using-the-playground.mdx
Original file line number Diff line number Diff line change
@@ -1,81 +1,120 @@
---
title: "Using the Playground"
title: "Using the Prompt Playground"
description: "Learn how to use Agenta's LLM playground for prompt engineering, model comparison, and deployment. A powerful alternative to OpenAI playground that supports multiple models and frameworks."
---

```mdx-code-block
import { Stream } from '@cloudflare/stream-react';
import Image from "@theme/IdealImage";
```

The Agenta playground is a comprehensive prompt engineering IDE that helps you develop, test, and deploy LLM applications. Unlike traditional LLM playgrounds, Agenta integrates prompt management, evaluation, and deployment into a single platform, supporting multiple models and frameworks.

<Stream controls src="fcf2b69dacb3e3a624c09af40c2dc154" height="400px" />
<br />


The agenta playground is a platform that lets you create, modify, and compare different prompts and configurations for your LLM application.
## Key Features

## Prompt Templates
- **Multi-Model Support**: Test prompts across different providers including OpenAI GPT models, Anthropic Claude, Google Gemini, Mistral, DeepSeek, and OpenRouter models
- **Framework Integration**: Works seamlessly with popular frameworks like LangChain, LlamaIndex, and CrewAI
- **Version Control**: Track and manage prompt versions with built-in prompt management
- **Model Comparison**: Benchmark different LLM models side by side
- **Parameter Control**: Fine-tune model parameters including temperature, top-k, and presence penalty
- **Environment Management**: Deploy configurations to development, staging, and production environments

An LLM takes a prompt and returns a completion. To make the prompt reusable, we need to be able to dynamically modify it based on a set of inputs. A prompt template is a prompt that can be dynamically modified based on a set of inputs.
## Creating and Managing Prompts

For instance, a prompt to create a blog post might look like this:
### Prompt Templates

Prompt templates form the foundation of reusable LLM interactions. They contain variable placeholders that adapt to different inputs, making your prompts dynamic and versatile.

Example of a prompt template:
```
Write a blog post about {subject}
Write a blog post about {{subject}} that focuses on {{key_aspect}}
```

Here, the input is the subject of the blog post. The prompt can be reused for different subjects.
Agenta uses curly brackets `{{variable}}` for template variables, automatically creating corresponding input fields in the playground interface.

### Adjusting Model Parameters

To modify model-specific parameters:

1. Click on the model name in the configuration panel
2. A settings modal will appear with available parameters
3. Adjust the following settings as needed:

- **Temperature** (0-2): Controls randomness in the output
- Lower values (0.1-0.4): More focused, deterministic responses
- Higher values (0.7-1.0): More creative, varied outputs

- **Top-k** (1-100): Limits the pool of tokens the model considers
- Lower values: More focused word choice
- Higher values: Broader vocabulary usage

- **Presence Penalty** (-2.0 to 2.0): Influences topic diversity
- Positive values: Encourages exploring new topics
- Negative values: Allows topic repetition

- **Frequency Penalty** (-2.0 to 2.0): Affects word choice
- Positive values: Discourages word repetition
- Negative values: Allows word reuse

## Comparing Prompts and Models

Agenta's playground allows you to compare different prompts, model configurations, and LLM providers side by side. This helps you identify the most effective combination for your use case.

### Creating Multiple Variants

To compare different configurations:

The LLM app templates in agenta use the f-string templating language, with prompt variables referenced using a curly bracket syntax.
1. Click the "+Compare" button in the top right corner
2. Select "New Variant" from the dropdown menu
3. Choose a source variant to duplicate (this copies all current parameters)
4. Give your new variant a descriptive name
5. Modify the configuration as needed

## Adding new inputs
Each variant appears as a separate column in the playground, allowing direct comparison of outputs.

To add new inputs to the LLM app, navigate to the playground and modify the inputs under "Modify Parameters."
### Side-by-Side Testing

<Image
className="dark:hidden"
img={require("/images/basic_guides/08_add_new_input_light.png")}
/>
<Image
className="hidden dark:block"
img={require("/images/basic_guides/08_add_new_input_dark.png")}
/>
With multiple variants set up, you can:
- Enter the same input text across all variants
- Compare the generated outputs side by side
- Review response times for each configuration
- Monitor the cost per request for different models

After adding an input, incorporate it into the prompt templates using the curly bracket syntax.
This parallel testing helps you understand how different prompts and models handle the same input.

## Creating a variant
### Version Control and Deployment

You can create a new variant of an application by clicking on "Add Variant" in the "Side-by-side" view or the "+" tab in the "Tab view".
The playground integrates with Agenta's [prompt management system](/prompt-management/overview) to track and deploy changes:

<Image
className="dark:hidden"
img={require("/images/basic_guides/09_add_variant_light.png")}
/>
<Image
className="hidden dark:block"
img={require("/images/basic_guides/09_add_variant_dark.png")}
/>
1. Click "Commit" to save a new version of your variant
2. Changes remain in development until explicitly deployed
3. Use "Deploy" to push configurations to specific environments (development, staging, production)

You'll then see a window where you can select the source variant to use as a template and provide a new name for your variant.
## Working with Test Sets

After creating a variant, you can alter its parameters. Remember to save your changes before deploying the variant.
### Loading Existing Test Sets

## Testing a variant
You can import test sets that you've created through:

Fill in the inputs in the cards and click "Run" to test a variant dynamically.
- CSV uploads
- Programmatic creation
- Web UI input

You can also load a test set to populate the playground with a set of inputs. Then You can click Run all to run all the inputs in the test set.
The test set must contain columns that match your prompt template variables.

<Image
className="dark:hidden"
img={require("/images/basic_guides/10_testing_a_variant_light.png")}
/>
<Image
className="hidden dark:block"
img={require("/images/basic_guides/10_testing_a_variant_dark.png")}
/>
### Saving Test Cases

## Comparing variants side by side
To save an interesting test case:

Click on the "Side-by-side" tab to compare variants. From the dropdowns, select the variants you wish to compare. This allows you to view the results of multiple variants simultaneously. When using a chat application, you can interact with different variants in parallel.
1. Click the three-dot menu on the right side of the test case card
2. Select "Add to Test Set"
3. Choose an existing test set or create a new one
4. Review the input/output mapping (use default settings unless you have specific needs)
5. Preview the changes to your test set
6. Confirm to save

<Image
className="dark:hidden"
img={require("/images/basic_guides/11_side_by_side_chat_mode_light.png")}
/>
<Image
className="hidden dark:block"
img={require("/images/basic_guides/11_side_by_side_chat_mode_dark.png")}
/>
The Agenta playground serves as a comprehensive prompt engineering environment, offering advantages over traditional playgrounds like OpenAI's by providing integrated version control, deployment capabilities, and support for multiple models and frameworks. Whether you're developing new prompts, comparing model performance, or deploying to production, the playground provides the tools needed for effective LLM application development.

0 comments on commit a67bc3f

Please sign in to comment.