Add ChatGPT wrapper pipeline

README.MD

@@ -5,16 +5,24 @@ Pyris is an intermediary system that connects the [Artemis](https://github.com/l
 Currently, Pyris powers [Iris](https://artemis.cit.tum.de/about-iris), a virtual AI tutor that assists students with their programming exercises on Artemis in a pedagogically meaningful way.
 
 ## Table of Contents
-- [Features](#features)
-- [Setup](#setup)
-  - [Prerequisites](#prerequisites)
-  - [Local Development Setup](#local-development-setup)
-  - [Docker Setup](#docker-setup)
-    - [Development Environment](#development-environment)
-    - [Production Environment](#production-environment)
-- [Customizing Configuration](#customizing-configuration)
-- [Troubleshooting](#troubleshooting)
-- [Additional Notes](#additional-notes)
+- [Pyris V2](#pyris-v2)
+  - [Table of Contents](#table-of-contents)
+  - [Features](#features)
+  - [Setup](#setup)
+    - [Prerequisites](#prerequisites)
+    - [Local Development Setup](#local-development-setup)
+      - [Steps](#steps)
+    - [Docker Setup](#docker-setup)
+      - [Prerequisites](#prerequisites-1)
+      - [Docker Compose Files](#docker-compose-files)
+      - [Running the Containers](#running-the-containers)
+        - [**Development Environment**](#development-environment)
+        - [**Production Environment**](#production-environment)
+          - [**Option 1: With Nginx**](#option-1-with-nginx)
+          - [**Option 2: Without Nginx**](#option-2-without-nginx)
+      - [Managing the Containers](#managing-the-containers)
+      - [Customizing Configuration](#customizing-configuration)
+  - [Troubleshooting](#troubleshooting)
 
 ## Features
 
@@ -87,10 +95,10 @@ Currently, Pyris powers [Iris](https://artemis.cit.tum.de/about-iris), a virtual
 
    - **Create an LLM Config File**
 
-     Create an `llm-config.local.yml` file in the root directory. You can use the provided `llm-config.example.yml` as a base.
+     Create an `llm_config.local.yml` file in the root directory. You can use the provided `llm_config.example.yml` as a base.
 
      ```bash
-     cp llm-config.example.yml llm-config.local.yml
+     cp llm_config.example.yml llm_config.local.yml
      ```
 
      **Example OpenAI Configuration:**
@@ -176,15 +184,15 @@ Currently, Pyris powers [Iris](https://artemis.cit.tum.de/about-iris), a virtual
      >- GPT-4 equivalent: 4
      >- GPT-3.5 Turbo equivalent: 3.5
 
-     > **Warning:** Most existing pipelines in Pyris require a model with a `gpt_version_equivalent` of **4.5 or higher**. It is advised to define models in the `llm-config.local.yml` file with a `gpt_version_equivalent` of 4.5 or higher.
+     > **Warning:** Most existing pipelines in Pyris require a model with a `gpt_version_equivalent` of **4.5 or higher**. It is advised to define models in the `llm_config.local.yml` file with a `gpt_version_equivalent` of 4.5 or higher.
 
 4. **Run the Server**
 
    Start the Pyris server:
 
    ```bash
    APPLICATION_YML_PATH=./application.local.yml \
-   LLM_CONFIG_PATH=./llm-config.local.yml \
+   LLM_CONFIG_PATH=./llm_config.local.yml \
    uvicorn app.main:app --reload
    ```
 
@@ -203,7 +211,7 @@ Deploying Pyris using Docker ensures a consistent environment and simplifies the
 - **Docker**: Install Docker from the [official website](https://www.docker.com/get-started).
 - **Docker Compose**: Comes bundled with Docker Desktop or install separately on Linux.
 - **Clone the Pyris Repository**: If not already done, clone the repository.
-- **Create Configuration Files**: Create the `application.local.yml` and `llm-config.local.yml` files as described in the [Local Development Setup](#local-development-setup) section.
+- **Create Configuration Files**: Create the `application.local.yml` and `llm_config.local.yml` files as described in the [Local Development Setup](#local-development-setup) section.
 
   ```bash
   git clone https://github.com/ls1intum/Pyris.git Pyris
@@ -312,7 +320,7 @@ Deploying Pyris using Docker ensures a consistent environment and simplifies the
 
   - `PYRIS_DOCKER_TAG`: Specifies the Pyris Docker image tag.
   - `PYRIS_APPLICATION_YML_FILE`: Path to your `application.yml` file.
-  - `PYRIS_LLM_CONFIG_YML_FILE`: Path to your `llm-config.yml` file.
+  - `PYRIS_LLM_CONFIG_YML_FILE`: Path to your `llm_config.yml` file.
   - `PYRIS_PORT`: Host port for Pyris application (default is `8000`).
   - `WEAVIATE_PORT`: Host port for Weaviate REST API (default is `8001`).
   - `WEAVIATE_GRPC_PORT`: Host port for Weaviate gRPC interface (default is `50051`).
@@ -321,7 +329,7 @@ Deploying Pyris using Docker ensures a consistent environment and simplifies the
 
   Modify configuration files as needed:
 
-  - **Pyris Configuration**: Update `application.yml` and `llm-config.yml`.
+  - **Pyris Configuration**: Update `application.yml` and `llm_config.yml`.
   - **Weaviate Configuration**: Adjust settings in `weaviate.yml`.
   - **Nginx Configuration**: Modify Nginx settings in `nginx.yml` and related config files.
 

app/domain/status/text_exercise_chat_status_update_dto.py

@@ -1,5 +1,7 @@
+from typing import Optional
+
 from app.domain.status.status_update_dto import StatusUpdateDTO
 
 
 class TextExerciseChatStatusUpdateDTO(StatusUpdateDTO):
-    result: str
+    result: Optional[str]

app/llm/external/model.py

@@ -1,5 +1,7 @@
 from abc import ABCMeta, abstractmethod
 from typing import Sequence, Union, Dict, Any, Type, Callable
+
+from black import Optional
 from langchain_core.tools import BaseTool
 from openai.types.chat import ChatCompletionMessage
 from pydantic import BaseModel
@@ -42,23 +44,18 @@ def __subclasshook__(cls, subclass) -> bool:
 
     @abstractmethod
     def chat(
-        self, messages: list[PyrisMessage], arguments: CompletionArguments
+        self,
+        messages: list[PyrisMessage],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
     ) -> ChatCompletionMessage:
         """Create a completion from the chat messages"""
         raise NotImplementedError(
             f"The LLM {self.__str__()} does not support chat completion"
         )
 
-    @abstractmethod
-    def bind_tools(
-        self,
-        tools: Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]],
-    ):
-        """Bind tools"""
-        raise NotImplementedError(
-            f"The LLM {self.__str__()} does not support binding tools"
-        )
-
 
 class EmbeddingModel(LanguageModel, metaclass=ABCMeta):
     """Abstract class for the llm embedding wrappers"""

app/llm/external/ollama.py

@@ -2,9 +2,6 @@
 from datetime import datetime
 from typing import Literal, Any, Optional, Sequence, Union, Dict, Type, Callable
 
-from langchain_core.language_models import LanguageModelInput
-from langchain_core.messages import BaseMessage
-from langchain_core.runnables import Runnable
 from langchain_core.tools import BaseTool
 from pydantic import Field, BaseModel
 
@@ -115,7 +112,12 @@ def complete(
         return response["response"]
 
     def chat(
-        self, messages: list[PyrisMessage], arguments: CompletionArguments
+        self,
+        messages: list[PyrisMessage],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
     ) -> PyrisMessage:
         response = self._client.chat(
             model=self.model,
@@ -136,29 +138,5 @@ def embed(self, text: str) -> list[float]:
         )
         return list(response)
 
-    """Bind tools to the language model for function calling capabilities.
-
-    Note: Tool binding is currently not supported in Ollama models. This feature
-    is only available for OpenAI models.
-
-    Args:
-        tools: A sequence of tools to bind to the model.
-
-    Returns:
-        A runnable that can process inputs with the bound tools.
-
-    Raises:
-        NotImplementedError: Always raised as Ollama does not support tool binding.
-    """
-
-    # TODO: Implement tool binding support for Ollama models
-    def bind_tools(
-        self,
-        tools: Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]],
-    ) -> Runnable[LanguageModelInput, BaseMessage]:
-        raise NotImplementedError(
-            f"The LLM {self.__str__()} does not support binding tools"
-        )
-
     def __str__(self):
         return f"Ollama('{self.model}')"

app/llm/external/openai_chat.py

@@ -17,7 +17,7 @@
 from openai.types import CompletionUsage
 from openai.types.chat import ChatCompletionMessage, ChatCompletionMessageParam
 from openai.types.shared_params import ResponseFormatJSONObject
-from pydantic import Field, BaseModel
+from pydantic import BaseModel
 
 from app.domain.data.text_message_content_dto import TextMessageContentDTO
 from ...common.message_converters import map_role_to_str, map_str_to_role
@@ -200,12 +200,14 @@ def convert_to_iris_message(
 class OpenAIChatModel(ChatModel):
     model: str
     api_key: str
-    tools: Optional[
-        Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
-    ] = Field(default_factory=list, alias="tools")
 
     def chat(
-        self, messages: list[PyrisMessage], arguments: CompletionArguments
+        self,
+        messages: list[PyrisMessage],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
     ) -> PyrisMessage:
         # noinspection PyTypeChecker
         retries = 5
@@ -214,6 +216,11 @@ def chat(
         client = self.get_client()
         # Maximum wait time: 1 + 2 + 4 + 8 + 16 = 31 seconds
 
+        for message in messages:
+            if message.sender == "SYSTEM":
+                print("SYSTEM MESSAGE: " + message.contents[0].text_content)
+                break
+
         messages = convert_to_open_ai_messages(messages)
 
         for attempt in range(retries):
@@ -230,8 +237,9 @@ def chat(
                         type="json_object"
                     )
 
-                if self.tools:
-                    params["tools"] = self.tools
+                if tools:
+                    params["tools"] = [convert_to_openai_tool(tool) for tool in tools]
+                    logging.info(f"Using tools: {tools}")
 
                 response = client.chat.completions.create(**params)
                 choice = response.choices[0]
@@ -243,6 +251,20 @@ def chat(
                     # We don't want to retry because the same message will likely be rejected again.
                     # Raise an exception to trigger the global error handler and report a fatal error to the client.
                     raise ContentFilterFinishReasonError()
+
+                if (
+                    choice.message is None
+                    or choice.message.content is None
+                    or len(choice.message.content) == 0
+                ):
+                    logging.error("Model returned an empty message")
+                    logging.error("Finish reason: " + choice.finish_reason)
+                    if (
+                        choice.message is not None
+                        and choice.message.refusal is not None
+                    ):
+                        logging.error("Refusal: " + choice.message.refusal)
+
                 return convert_to_iris_message(choice.message, usage, model)
             except (
                 APIError,
@@ -255,12 +277,6 @@ def chat(
                 time.sleep(wait_time)
         raise Exception(f"Failed to get response from OpenAI after {retries} retries")
 
-    def bind_tools(
-        self,
-        tools: Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]],
-    ):
-        self.tools = [convert_to_openai_tool(tool) for tool in tools]
-
 
 class DirectOpenAIChatModel(OpenAIChatModel):
     type: Literal["openai_chat"]

app/llm/langchain/iris_langchain_chat_model.py

@@ -12,7 +12,7 @@
 from langchain_core.outputs.chat_generation import ChatGeneration
 from langchain_core.runnables import Runnable
 from langchain_core.tools import BaseTool
-from pydantic import BaseModel
+from pydantic import BaseModel, Field
 
 from app.common.PipelineEnum import PipelineEnum
 from app.common.token_usage_dto import TokenUsageDTO
@@ -30,6 +30,9 @@ class IrisLangchainChatModel(BaseChatModel):
     completion_args: CompletionArguments
     tokens: TokenUsageDTO = None
     logger: Logger = logging.getLogger(__name__)
+    tools: Optional[
+        Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+    ] = Field(default_factory=list, alias="tools")
 
     def __init__(
         self,
@@ -65,7 +68,7 @@ def bind_tools(
         if not tools:
             raise ValueError("At least one tool must be provided")
 
-        self.request_handler.bind_tools(tools)
+        self.tools = tools
         return self
 
     def _generate(
@@ -77,7 +80,9 @@ def _generate(
     ) -> ChatResult:
         iris_messages = [convert_langchain_message_to_iris_message(m) for m in messages]
         self.completion_args.stop = stop
-        iris_message = self.request_handler.chat(iris_messages, self.completion_args)
+        iris_message = self.request_handler.chat(
+            iris_messages, self.completion_args, self.tools
+        )
         base_message = convert_iris_message_to_langchain_message(iris_message)
         chat_generation = ChatGeneration(message=base_message)
         self.tokens = TokenUsageDTO(

app/llm/request_handler/basic_request_handler.py

@@ -32,10 +32,15 @@ def complete(
         return llm.complete(prompt, arguments, image)
 
     def chat(
-        self, messages: list[PyrisMessage], arguments: CompletionArguments
+        self,
+        messages: list[PyrisMessage],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
     ) -> PyrisMessage:
         llm = self.llm_manager.get_llm_by_id(self.model_id)
-        return llm.chat(messages, arguments)
+        return llm.chat(messages, arguments, tools)
 
     def embed(self, text: str) -> list[float]:
         llm = self.llm_manager.get_llm_by_id(self.model_id)

app/llm/request_handler/capability_request_handler.py

@@ -1,5 +1,5 @@
 from enum import Enum
-from typing import Sequence, Union, Dict, Any, Type, Callable
+from typing import Sequence, Union, Dict, Any, Type, Callable, Optional
 
 from langchain_core.tools import BaseTool
 from pydantic import ConfigDict
@@ -50,10 +50,15 @@ def complete(self, prompt: str, arguments: CompletionArguments) -> str:
         return llm.complete(prompt, arguments)
 
     def chat(
-        self, messages: list[PyrisMessage], arguments: CompletionArguments
+        self,
+        messages: list[PyrisMessage],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
     ) -> PyrisMessage:
         llm = self._select_model(ChatModel)
-        message = llm.chat(messages, arguments)
+        message = llm.chat(messages, arguments, tools)
         message.token_usage.cost_per_input_token = llm.capabilities.input_cost.value
         message.token_usage.cost_per_output_token = llm.capabilities.output_cost.value
         return message

app/llm/request_handler/request_handler_interface.py

@@ -34,7 +34,14 @@ def complete(
         raise NotImplementedError
 
     @abstractmethod
-    def chat(self, messages: list[any], arguments: CompletionArguments) -> PyrisMessage:
+    def chat(
+        self,
+        messages: list[any],
+        arguments: CompletionArguments,
+        tools: Optional[
+            Sequence[Union[Dict[str, Any], Type[BaseModel], Callable, BaseTool]]
+        ],
+    ) -> PyrisMessage:
         """Create a completion from the chat messages"""
         raise NotImplementedError