Add support for custom memory storage

[devin-ai-integration]

Add support for custom memory storage

Fixes #2278

Description

This PR adds support for custom memory storage implementations for different memory types. Users can now provide their own storage implementation by extending the Storage interface and passing it to the memory instances or through the memory_config parameter.

Features

• Added ability to pass custom storage implementations to memory classes
• Updated memory initialization to support custom storage
• Added documentation with examples of custom storage implementation
• Added tests to verify custom storage functionality
• Maintained backward compatibility with existing memory implementations

Testing

• Added unit tests for custom storage implementation
• Verified all existing tests pass with the new implementation
• Manually tested different memory types with custom storage

Documentation

• Added documentation in docs/concepts/custom_memory_storage.mdx with examples of how to use custom storage
• Included example of Redis-based storage implementation

Link to Devin run

https://app.devin.ai/sessions/a0f6657060134ab59fd10f5dc746b60b

[devin-ai-integration]

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

• Address comments on this PR. Add "(aside)" to your comment to have me ignore it.
• Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

• Disable automatic comment and CI monitoring

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment: Custom Memory Storage Implementation

Overview

The patch effectively introduces configuration support for custom memory storage backends in crewAI, granting users the flexibility to supply their own storage implementations. The changes are comprehensive, spanning multiple core files, documentation enhancements, and thorough test cases.

Key Findings

1. Documentation (custom_memory_storage.mdx)

• Clarity: The documentation is impressively structured and clear, providing users with essential examples and use cases.
• Utility: The Redis example implementation serves as an excellent guide for practical implementation, showing promise for encouraging developer adoption.

2. Core Memory Classes Refactoring

• Typing Improvement:
  • The base Memory class should implement stronger typing for the storage attribute:

    from typing import TypeVar, Generic
    from .storage.interface import Storage
    
    T = TypeVar('T', bound=Storage)
    
    class Memory(BaseModel, Generic[T]):
        storage: T

• Duplicate Code Elimination: In the LongTermMemory class, redundant super() calls were found and should be consolidated for better readability and maintenance:

  def __init__(self, crew=None, embedder_config=None, storage=None, path=None):
      # Consolidate initialization logic here

• Search Method Type Safety: Enhance the search method typing to ensure clarity about expected return types:

  from typing import TypedDict
  
  class SearchResult(TypedDict):
      context: str
      metadata: Dict[str, Any]
      score: float
  
  def search(self, query: str, limit: int = 3) -> List[SearchResult]:

3. Storage Interface Implementation

• A strong typing recommendation for the Storage interface could improve type safety and usability:

  class Storage(ABC, Generic[T]):
      # Ensure methods are correctly typed for their arguments and return values

4. Test Coverage Recommendations

• Edge Case Handling: Tests should include edge cases:

  def test_custom_storage_error_handling():
      # Implement test cases for storage errors

• Concurrency Testing: It would be prudent to include tests that validate concurrency handling in storage operations:

  def test_custom_storage_thread_safety():
      # Implement extensive test cases for multi-threaded interactions

Recommendations for Future Improvements

• Enhance dependency injection for storage components.
• Introduce validation mandates for custom storage implementations.
• Implement performance benchmarking tests to measure scalability and efficiency.
• Incorporate logging mechanisms for storage operations.
• Consider implementing health checks to ensure storage integrity and reliability.

Security Considerations

• Immediate input validation for custom storage implementations is essential to prevent injection attacks.
• Consider implementing sensitive data encryption before storing.

Conclusion

The implementation shows a solid understanding of extensibility in memory management. However, adopting the proposed enhancements regarding type safety, test coverage, and logging will significantly improve code quality and resilience against future changes. The documentation is well-structured and the test coverage is commendable, making the codebase ready for production with the suggested adjustments.

Overall, commendable work!

---

By addressing the above points, we can significantly enhance the maintainability and extensibility of the memory storage solution while ensuring it meets both user expectations and security standards.