Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Robust parallelization with Ray #3

Merged
merged 19 commits into from
Dec 6, 2024
Merged

Robust parallelization with Ray #3

merged 19 commits into from
Dec 6, 2024

Conversation

aalexmmaldonado
Copy link
Member

This PR introduces significant enhancements and optimizations to the package by replacing custom multiprocessing with Ray for parallelization. Key changes include:

  • Ray Parallelization: Replaced custom multiprocessing with Ray, enabling efficient task scheduling and execution.
  • Periodic Saving: Implemented periodic saving during processing to ensure progress is preserved, even in case of interruptions.
  • Performance Optimizations: Improved overall performance by optimizing core functions and reducing bottlenecks.
  • Modular Refactoring: Refactored the codebase to make components more modular, improving maintainability and extensibility.

@aalexmmaldonado aalexmmaldonado changed the title More robust parallelization with Ray Robust parallelization with Ray Dec 6, 2024
@codecov-commenter
Copy link

Welcome to Codecov 🎉

Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests.

Thanks for integrating Codecov - We've got you covered ☂️

@aalexmmaldonado aalexmmaldonado merged commit ffcdf76 into main Dec 6, 2024
2 checks passed
@aalexmmaldonado aalexmmaldonado deleted the ray branch December 14, 2024 14:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants