Merge pull request #202 from BethanyJep/main

Update docs: observability & specification

Author: sethjuarez

web/docs/contributing/code-guidelines/page.mdx

@@ -0,0 +1,19 @@
+---
+title: Code Guidelines
+authors:
+  - bethanyjep
+  - nitya
+  - sethjuarez
+date: 2025-03-11
+tags:
+  - contributing
+  - documentation
+index: 2
+---
+
+## Code contribution guidelines
+
+..to be updated
+
+---
+[Want to Contribute To the Project?](/docs/contributing/)

web/docs/contributing/docs-guidelines/page.mdx

@@ -0,0 +1,36 @@
+---
+title: Docs Guidelines
+authors:
+  - bethanyjep
+  - nitya
+date: 2024-06-10
+tags:
+  - contribution
+  - documentation
+index: 3
+---
+
+## Documentation guidelines
+For contributing to the Prompty documentation:
+
+1. Local Setup
+    - Our documentation uses MDX (Markdown + JSX) and is built with a static site generator.
+    - Install dependencies with `npm install` in the `web` directory.
+    - Use Node.js 18+ for best compatibility.
+
+2. Making Changes
+    - Documentation files are located in the `docs` directory.
+    - Follow the existing folder structure and naming conventions.
+    - Use `.mdx` extension for all documentation files.
+
+3. Previewing Changes
+    - Start the local development server with `npm run dev` from the `web` directory.
+    - View your changes at [http://localhost:3000](http://localhost:3000).
+    - The site will hot-reload as you edit files.
+
+4. Building Documentation
+    - Test a production build with `npm run build` followed by `npm run start`.
+    - Check for any build errors or warnings before submitting your PR.
+
+---
+[Want to Contribute To the Project?](/docs/contributing/) - _Updated Guidance Coming Soon_.

web/docs/contributing/page.mdx

@@ -4,7 +4,7 @@ index: 5
 authors:
   - bethanyjep
   - nitya
-date: 2024-10-09
+date: 2025-03-10
 tags:
   - documentation
   - contributing
@@ -23,7 +23,16 @@ Feedback can come in several forms:
  - Let us know if you found errors or ambiguity in the documentation
  - Report bugs or inconsistent behavior seen with Prompty tools and usage
 
-The easiest way to give us feedback is by [filing an issue](https://github.com/microsoft/prompty/issues/new?template=Blank+issue). **Please check previously logged issues (open and closed) to make sure the topic or bug has not already been raised.** If it does exist, weigh in on that discussion thread to add any additional context of value.
+The easiest way to give us feedback is by [filing an issue](https://github.com/microsoft/prompty/issues/new). **Please check previously logged issues (open and closed) to make sure the topic or bug has not already been raised.** If it does exist, weigh in on that discussion thread to add any additional context of value.
 
 ## Contributor guidelines
-The repository contains both the code and the documentation for the project. Each requires a different set of tools and processes to build and preview outcomes. We hope to document these soon - so check back for **contributor guidelines** that will cover the requirements in more detail.
+The repository contains both the code and the documentation for the project. Each requires a different set of tools and processes to build and preview outcomes. 
+
+## Pull Requests Guidelines
+When submitting a pull request (PR) to the Prompty repository, please use the following guidelines:
+
+- Fork the Repository: Always fork the repository to your own account before making your modifications.
+- Separate pull requests (PR):
+  - Submit each type of change in its own pull request. For example, bug fixes and documentation updates should be submitted in separate PRs.
+  - Typo fixes and minor documentation updates can be combined into a single PR where appropriate.
+  - Handle merge conflicts: If your pull request shows merge conflicts, update your local main branch to mirror the main repository before making your modifications.

web/docs/getting-started/debugging-prompty/page.mdx

@@ -3,7 +3,7 @@ title: Debugging Prompty
 authors:
   - bethanyjep
   - nitya
-date: 2024-10-22
+date: 2025-03-06
 tags:
   - getting-started
   - documentation
@@ -24,7 +24,7 @@ For observability in Prompty, we will use the tracer to visualize and debug the
 - Analyze the trace output to debug and fix the bug
 
 
-## 2. Understandfing Observability in Prompty
+## 2. Understanding Observability in Prompty
 
 Observability refers to the ability to monitor and understand the behavior of a system based on the data it produces, such as logs, metrics, and traces.. It is important as it provides you with insights on your LLM workflows and provides a way to debug your prompt inputs and outputs. 
 

web/docs/guides/prompty-extension/image-1.png

@@ -11,7 +11,49 @@ tags:
 index: 3
 ---
 
-TODO: How does Prompty runtime work for no-code execution in VS Code
+## The Prompty VS Code Extension
+Run Prompty files directly in VS Code. This Visual Studio Code extension offers an intuitive prompt playground within VS Code to streamline the prompt engineering process. You can find the Prompty extension in the Visual Studio Code Marketplace.
+
+Download the [VS Code extension here](https://marketplace.visualstudio.com/items?itemName=ms-toolsai.prompty).
+
+## VSCode Extension Features
+### Quickly Create
+Quickly create a basic prompty by right-clicking in the VS Code explorer and selecting "New Prompty."
+
+![Quick Create](image-2.png)
+
+### Preview
+Preview prompty similar to markdown with dynamic template rendering while typing, allowing you to see the prompt that will be sent to the model.
+
+![Preview](readme_preview.png)
+
+### Define and Switch Model Configurations
+* Define your model configurations directly in VS Code.
+* Quickly switch between different model configurations.
+
+  ![Define Configuration](image-5.png)
+
+  ![Switch Model Configuration](switchModelConfiguration.png)
+* Use VS Code settings to define model configuration at:
+  * User level for use across different prompty files.
+  * Workspace level to share with team members via Git.
+
+    ![ModelConfigurationSettings](modelConfigurationSettings.png)
+
+* We strongly encourage using Azure Active Directory authentication for enhanced security. Leave the `api_key` empty to trigger AAD auth.
+* OpenAI is also supported. You can store the key in VSCode settings or use `${env:xxx}` to read the API key from an environment variable.
+  * You can put environment variables in `.env` file, in the same folder of the prompty file, or in the workspace root folder.
+  * Alternatively, you can also specify it in system variables, follow [OpenAI's Guide](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety) for key safety, setting it through Control Panel/zsh/bash, and then restart VS Code to load new values.
+
+### Quick Run
+Hit **F5** or click the **Run** button at the top. There are two output windows:
+* **Prompty Output** shows a concise view.
+
+  ![Prompty Output](image-3.png)
+
+* **Prompty Output (Verbose)** shows detailed requests sent and received.
+
+  ![Prompty Output (Verbose)](image-8.png)
 
 ---
 [Want to Contribute To the Project?](/docs/contributing/) - _Updated Guidance Coming Soon_.

web/docs/guides/prompty-extension/image-2.png

@@ -4,16 +4,125 @@ authors:
   - bethanyjep
   - nitya
   - sethjuarez
-date: 2024-10-22
+date: 2025-03-06
 tags:
   - tutorials
   - runtime
 index: 4
 ---
 
-Get started with Observability at [debugging Prompty](/docs/getting-started/debugging-prompty), check out additional documentation at [Observability](https://github.com/microsoft/prompty/tree/main/runtime/prompty#using-tracing-in-prompty).
+## Using Tracing in Prompty
+
+Prompty supports tracing to help you understand the execution of your prompts. This functionality is customizable and can be used to trace the execution of your prompts in a way that makes sense to you. Prompty has two default traces built in: `console_tracer` and `PromptyTracer`. The `console_tracer` writes the trace to the console, and the `PromptyTracer` writes the trace to a JSON file. You can also create your own tracer by creating your own hook.
+
+```python
+import prompty
+# import invoker
+import prompty.azure
+from prompty.tracer import trace, Tracer, console_tracer, PromptyTracer
+
+# add console tracer
+Tracer.add("console", console_tracer)
+
+# add PromptyTracer
+json_tracer = PromptyTracer(output_dir="path/to/output")
+Tracer.add("console", json_tracer.tracer)
+
+# execute the prompt
+response = prompty.execute("path/to/prompty/file")
+
+print(response)
+```
+
+You can also bring your own tracer by your own tracing hook. The `console_tracer` is the simplest example of a tracer. It writes the trace to the console.
+This is what it looks like:
+
+```python
+@contextlib.contextmanager
+def console_tracer(name: str) -> Iterator[Callable[[str, Any], None]]:
+    try:
+        print(f"Starting {name}")
+        yield lambda key, value: print(f"{key}:\n{json.dumps(value, indent=4)}")
+    finally:
+        print(f"Ending {name}")
+
+```
+
+It uses a context manager to define the start and end of the trace so you can do whatever setup and teardown you need. The `yield` statement returns a function that you can use to write the trace. The `console_tracer` writes the trace to the console using the `print` function.
+
+The `PromptyTracer` is a more complex example of a tracer. This tracer manages its internal state using a full class. Here's an example of the class based approach that writes each function trace to a JSON file:
+
+```python
+class SimplePromptyTracer:
+    def __init__(self, output_dir: str):
+        self.output_dir = output_dir
+        self.tracer = self._tracer
+
+    @contextlib.contextmanager
+    def tracer(self, name: str) -> Iterator[Callable[[str, Any], None]]:
+        trace = {}
+        try:
+            yield lambda key, value: trace.update({key: value})
+        finally:
+            with open(os.path.join(self.output_dir, f"{name}.json"), "w") as f:
+                json.dump(trace, f, indent=4)
+```
+
+The tracing mechanism is supported for all of the prompty runtime internals and can be used to trace the execution of the prompt along with all of the paramters. There is also a `@trace` decorator that can be used to trace the execution of any function external to the runtime. This is provided as a facility to trace the execution of the prompt and whatever supporting code you have.
+
+```python
+import prompty
+# import invoker
+import prompty.azure
+from prompty.tracer import trace, Tracer, PromptyTracer
+
+json_tracer = PromptyTracer(output_dir="path/to/output")
+Tracer.add("PromptyTracer", json_tracer.tracer)
+
+@trace
+def get_customer(customerId):
+    return {"id": customerId, "firstName": "Sally", "lastName": "Davis"}
+
+@trace
+def get_response(customerId, prompt):
+    customer = get_customer(customerId)
+
+    result = prompty.execute(
+        prompt,
+        inputs={"question": question, "customer": customer},
+    )
+    return {"question": question, "answer": result}
+
+```
+
+In this case, whenever this code is executed, a `.tracy` file will be created in the `path/to/output` directory. This file will contain the trace of the execution of the `get_response` function, the execution of the `get_customer` function, and the prompty internals that generated the response.
+
+## OpenTelemetry Tracing
+You can add OpenTelemetry tracing to your application using the same hook mechanism. In your application, you might create something like `trace_span` to trace the execution of your prompts:
+
+```python
+from opentelemetry import trace as oteltrace
+
+_tracer = "prompty"
+
+@contextlib.contextmanager
+def trace_span(name: str):
+    tracer = oteltrace.get_tracer(_tracer)
+    with tracer.start_as_current_span(name) as span:
+        yield lambda key, value: span.set_attribute(
+            key, json.dumps(value).replace("\n", "")
+        )
+
+# adding this hook to the prompty runtime
+Tracer.add("OpenTelemetry", trace_span)
+
+```
+
+This will produce spans during the execution of the prompt that can be sent to an OpenTelemetry collector for further analysis.
+
+
+To get started with Observability at refer [debugging Prompty](/docs/getting-started/debugging-prompty) in the Getting Started section.
 
 
-TODO: Explain how to trace Prompty execution from input to response
 ---
 [Want to Contribute To the Project?](/docs/contributing/) - _Updated Guidance Coming Soon_.

web/docs/guides/prompty-extension/image-3.png

@@ -11,7 +11,25 @@ tags:
 index: 2
 ---
 
-TODO: Explain how runtimes work and how to build a custom runtime
+## Using this Prompty Runtime
+The Python runtime is a simple way to run your prompts in Python. The runtime is available as a Python package and can be installed using pip. Depending on the type of prompt you are running, you may need to install additional dependencies. The runtime is designed to be extensible and can be customized to fit your needs.
+
+```bash
+pip install "prompty[azure]"
+```
+
+Simple usage example:
+
+```python
+import prompty
+# import invoker
+import prompty.azure
+
+# execute the prompt
+response = prompty.execute("path/to/prompty/file")
+
+print(response)
+```
 
 ---
 [Want to Contribute To the Project?](/docs/contributing/) - _Updated Guidance Coming Soon_.