Add support for custom LLM implementations

[devin-ai-integration]

Fixes #2276. Adds support for custom LLM implementations that don't rely on litellm's authentication mechanism. This allows users to implement their own LLM classes with custom authentication methods like JWT-based authentication.

Changes

• Added a new abstract base class that defines the interface for all LLM implementations
• Updated the existing class to inherit from
• Updated type annotations in Agent, Crew, and utility classes to use
• Added tests for custom LLM implementations
• Added documentation for creating custom LLM implementations

Documentation

Added a new documentation file that explains how to create and use custom LLM implementations, including examples for JWT-based authentication.

Link to Devin run: https://app.devin.ai/sessions/c45e76c4de5a45d2af4c486bed8044f1

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment: Custom LLM Implementation

Overview

This pull request introduces significant enhancements by implementing support for custom Large Language Models (LLMs) through the new abstract base class BaseLLM. This design enables users to create tailored LLM implementations without relying on external dependencies like litellm's authentication mechanism.

Documentation Review (docs/custom_llm.md)

The documentation is overall well-structured and informative, containing relevant examples. However, improvements can be made in the following areas:

1. Error Handling Best Practices: Adding a dedicated section about error handling will guide users in dealing with common issues during API interactions.

   ## Error Handling Best Practices
   When implementing custom LLMs, consider implementing tries for API calls.

2. Type Hints in Code Examples: Including type hints in all examples can greatly enhance readability and IDE support.

3. Troubleshooting Section: It would be beneficial to integrate a troubleshooting section that addresses common pitfalls users might encounter.

Code Analysis

1. src/crewai/llm.py

Strengths:

• Clearly defined BaseLLM that serves as a robust contract for implementing new models.
• Method signatures are clear, with appropriate type hints.

Issues and Suggestions:

• Docstring Completeness: More detailed docstrings are needed to specify all expected attributes, enhancing clarity.

  class BaseLLM(ABC):
      """Abstract base class for LLM implementations.
      This class defines the required structure and guidelines for implementing LLMs.
      """

• Initialization Validation: Consider adding validation logic to ensure all required methods are implemented.

2. src/crewai/utilities/llm_utils.py

Suggestions:

• Improve type checking within the create_llm function to handle various input types more gracefully and prevent runtime errors.

3. src/crewai/agent.py & src/crewai/crew.py

Suggestions:

• Refine type hints for llm attributes to clarify their expected input and usage structure.

  class Agent(BaseAgent):
      llm: Union[str, BaseLLM, None]

4. Tests (tests/custom_llm_test.py)

Strengths:

• Good coverage of the fundamental functionalities of custom LLMs.

Suggestions for Additional Tests:

• Expand tests to include scenarios for error handling and validate callback functionalities during LLM calls.

Security Considerations

1. JWT Token Input Validation: Ensure that JWT tokens are validated correctly to prevent processing of malformed tokens.

   class JWTAuthLLM(BaseLLM):
       def __init__(self, jwt_token: str):
           if not jwt_token or not isinstance(jwt_token, str):
               raise ValueError("Invalid JWT token")

2. Request Timeout Handling: Implement timeout logic for API requests to prevent indefinite waiting.

   def call(self, messages, **kwargs):
       try:
           response = requests.post(...)
           response.raise_for_status()
           return response.json()
       except requests.Timeout:
           raise TimeoutError("LLM request timed out")

Overall Recommendations

1. Comprehensive Logging: Introduce logging mechanisms throughout the LLM to facilitate debugging and monitoring of operations.
2. Retry Logic for API Calls: Implement retry functionality for external calls to efficiently handle transient failures.
3. Rate Limiting Support: Include support to avoid overwhelming APIs.
4. Metrics Collection: Track usage metrics for better monitoring of the models in a production setting.

In summary, the implementation is structurally sound and well-designed but could benefit from improvements in error handling, robust type checking, and enhanced documentation. These adjustments will help ensure reliability and maintainability for future development.

[lorenzejay]

@bhancockio can you review this PR when you have time

[TaylorPzreal]

Waiting the feature!