Skip to content

Commit

Permalink
Add docs for dynamic types tests (#1307)
Browse files Browse the repository at this point in the history
<!-- ELLIPSIS_HIDDEN -->


> [!IMPORTANT]
> This PR adds documentation for testing dynamic types in BAML,
detailing the use of `type_builder` and `dynamic` blocks with examples
for dynamic classes and enums.
> 
>   - **Documentation**:
> - Adds section on testing dynamic types in `testing-functions.mdx`,
`dynamic-types.mdx`, `dynamic.mdx`, `test.mdx`, and `typebuilder.mdx`.
> - Includes examples for dynamic classes and enums using `type_builder`
and `dynamic` blocks.
> - Updates `snippets/dynamic-class-test.mdx` with a code example for
dynamic class testing.
>   - **Dynamic Types**:
> - Explains how to modify dynamic classes and enums in tests using
`type_builder` and `dynamic` blocks.
> - Describes testing scenarios for return types and parameter types
with dynamic modifications.
> 
> <sup>This description was created by </sup>[<img alt="Ellipsis"
src="https://img.shields.io/badge/Ellipsis-blue?color=175173">](https://www.ellipsis.dev?ref=BoundaryML%2Fbaml&utm_source=github&utm_medium=referral)<sup>
for babb416. It will automatically
update as commits are pushed.</sup>


<!-- ELLIPSIS_HIDDEN -->
  • Loading branch information
antoniosarosi authored Feb 4, 2025
1 parent 7f852d0 commit 9c068e0
Show file tree
Hide file tree
Showing 6 changed files with 252 additions and 5 deletions.
13 changes: 12 additions & 1 deletion fern/01-guide/04-baml-basics/testing-functions.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -313,4 +313,15 @@ test MyTest {
the remainder of the checks and asserts in this particular test.

For more information about the syntax used inside `@@check` and `@@assert`
attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts)
attributes, see [Checks and Asserts](/guide/baml-advanced/checks-and-asserts)

## Dynamic Types Tests

Classes and enums marked with the [`@@dynamic`](/ref/baml-client/type-builder)
attribute can be modified in tests using the `type_builder` and `dynamic`
blocks.

<Markdown src="../../snippets/dynamic-class-test.mdx" />

The `type_builder` block can contain new types scoped to the parent `test` block
and also `dynamic` blocks that act as modifiers for dynamic classes or enums.
171 changes: 169 additions & 2 deletions fern/01-guide/05-baml-advanced/dynamic-types.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,173 @@ puts res
</CodeBlocks>

### Testing dynamic types in BAML
This feature is coming soon! Let us know if you're interested in testing it out!

You can still write tests in Python, TypeScript, Ruby, etc in the meantime.
When testing dynamic types there are two different cases:
1. Injecting properties into dynamic types returned by the tested function.
2. Injecting values into dynamic types received as arguments by the tested function.

The first case requires using the `type_builder` and `dynamic` blocks in the
test, whereas the second case only requires specifying the values in the `args`
block.

#### Testing return types

##### Dynamic classes

Suppose we have a dynamic class `Resume` and we want to add a property that
stores the user's work experience when we testing a specific function. We can
do that by specifying the types and properties that we need in the
`type_builder` block.


```baml {4, 14-27}
class Resume {
name string
skills string[]
@@dynamic // Marked as @@dynamic.
}
// Function that returns a dynamic class.
function ExtractResume(from_text: string) -> Resume {
// Prompt
}
test ReturnDynamicClassTest {
functions [ExtractResume]
type_builder {
// Defines a new type available only within this test block.
class Experience {
title string
company string
start_date string
end_date string
}
// Injects new properties into the `@@dynamic` part of the Resume class.
dynamic Resume {
experience Experience[]
}
}
args {
from_text #"
John Doe
Experience
- Software Engineer, Boundary, Sep 2022 - Sep 2023
Skills
- Python
- Java
"#
}
}
```

The rendered prompt for `ExtractResume` will now include the `experience` field
defined in the `dynamic` block and the LLM will correctly extract the experience
in the input text.

##### Dynamic enums

Dynamic enums can be included in the `type_builder` block just like classes. The
only difference is that we inject new variants in the `dynamic` block instead of
properties.

```baml {7, 17-22}
enum Category {
Refund
CancelOrder
TechnicalSupport
AccountIssue
Question
@@dynamic // Marked as @@dynamic.
}
// Function that returns a dynamic enum.
function ClassifyMessage(message: string) -> Category {
// Prompt
}
test ReturnDynamicEnumTest {
functions [ClassifyMessage]
type_builder {
// Injects new variants into the `@@dynamic` part of the Category enum.
dynamic Category {
Feedback
}
}
args {
message "I think the product is great!"
}
}
```

The `Feedback` variant will be rendered in the prompt for `ClassifyMessage`
during the test execution.

#### Testing parameter types

When a dynamic type is used as an input parameter of a function, we can simply
pass any value in the `args` block of the test and the value will be rendered in
the prompt.

##### Dynamic classes

```baml {4, 17-24}
class Resume {
name string
skills string[]
@@dynamic // Marked as @@dynamic.
}
function WriteResume(resume: Resume) -> string {
// Prompt
}
test DynamicClassAsInputTest {
functions [WriteResume]
args {
resume {
name "John Doe"
skills ["C++", "Java"]
experience [
{
title "Software Engineer"
company "Boundary"
start_date "2023-09-01"
end_date "2024-09-01"
}
]
}
}
}
```

##### Dynamic enums

Enums work the same way, any variant defined in the `args` block will be
rendered normally.

```baml {7, 17}
enum Category {
Refund
CancelOrder
TechnicalSupport
AccountIssue
Question
@@dynamic // Marked as @@dynamic.
}
function WriteCustomerMessage(category: Category) -> string {
// Prompt
}
test DynamicEnumAsInputTest {
functions [WriteCustomerMessage]
args {
category Feedback // The enum is dynamic so it accepts a new variant.
}
}
```

For more information about dynamic types, see [Type Builder](/ref/baml-client/type-builder).
8 changes: 8 additions & 0 deletions fern/03-reference/baml/attributes/dynamic.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,3 +78,11 @@ def run
end
```

## Testing Dynamic Types

Dynamic classes and enums can be modified in tests using the `type_builder` and
`dynamic` blocks. All properties added in the `dynamic` block will be available
during the test execution.

<Markdown src="../../../snippets/dynamic-class-test.mdx" />

34 changes: 33 additions & 1 deletion fern/03-reference/baml/test.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ A BAML test consists of:
- Input arguments
- Optional testing configuration
- Optional assertions
- Optional type builders

```baml
test TestName {
Expand All @@ -20,7 +21,7 @@ test TestName {

## Test Declaration

### Syntax
### Basic Syntax

```baml
test name {
Expand All @@ -31,11 +32,36 @@ test name {
}
```

### Optional Features

```baml {3-11, 15, 16}
test name {
functions [function_list]
type_builder {
class NewType {
// Props
}
dynamic ExistingDynamicType {
new_prop NewType
// Inject Props Here
}
}
args {
parameter_assignments
}
@@check( check_length, {{ this.prop|length > 0 }} )
@@assert( {{ this.prop|length < 255 }})
}
```

### Components

- `name`: Test identifier (unique per function)
- `functions`: List of functions to test
- `args`: Input parameters for the test case
- `type_builder`: Block used to inject values into dynamic types
- `@@check`: Conditional check for test validity
- `@@assert`: Assertion for test result

## Input Types

Expand Down Expand Up @@ -212,6 +238,12 @@ test EndToEndFlow {
}
```

## Testing Dynamic Types

Dynamic types can be tested using `type_builder` and `dynamic` blocks:

<Markdown src="../../snippets/dynamic-class-test.mdx" />

## Integration with Development Tools

### VSCode Integration
Expand Down
9 changes: 8 additions & 1 deletion fern/03-reference/baml_client/typebuilder.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,12 @@ tb.User.add_property("middle_name", tb.string.optional)
All types added through TypeBuilder must be connected to the return type of your BAML function. Standalone types that aren't referenced won't affect the output schema.
</Warning>

## Testing Dynamic Types

See the [advanced dynamic types tests guide](/guide/baml-advanced/dynamic-runtime-types#testing-dynamic-types-in-baml)
for examples of testing functions that use dynamic types. See also the
[reference](/ref/baml/test) for syntax.

## Future Features

We're working on additional features for TypeBuilder:
Expand All @@ -335,4 +341,5 @@ We're working on additional features for TypeBuilder:
- OpenAPI schema integration
- Pydantic model support

If you're interested in these features, please join the discussion in our GitHub issues.
If you're interested in these features, please join the discussion in our GitHub
issues.
22 changes: 22 additions & 0 deletions fern/snippets/dynamic-class-test.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
```baml {3, 12-16}
class DynamicClass {
static_prop string
@@dynamic
}
function ReturnDynamicClass(input: string) -> DynamicClass {
// ...
}
test DynamicClassTest {
functions [ReturnDynamicClass]
type_builder {
dynamic DynamicClass {
new_prop_here string
}
}
args {
input "test data"
}
}
```

0 comments on commit 9c068e0

Please sign in to comment.