C/C++/Rust implementation of Scaler's scheduler

[rafa-be]

A C, C++ or Rust implementation of Scaler's scheduler will have a few performance advantages, mainly by relying on a more efficient memory management, and by avoiding unnecessary copies of objects.

Contrary to the client and the worker implementation, re-implementing the scheduler in a language different from Python is less challenging, as it's not actively interacting with Python objects.

The scheduler is also the main scalability blocking piece, which makes its efficiency more critical in making Scaler more scalable.

[bcncpp]

Hello, i am also interested on this. If we go to the rust path we could use maturin, if we go to c++, pybind. Let's discuss here your plans and add a design.

[rafa-be]

Hi @bcncpp,

At the moment, our plans are to fully rewrite the scheduler in C++ or Rust, without any Python code or binding.

The current implementation already communicates with 0mq and Cap’n Proto messages, which are not tied to any specific language.

However, we are still experimenting with new features to the Scaler's scheduler, so this is a mid-to-long term project.

Do you already have performance issues with the current Python implementation?

Raphael.

[bcncpp]

Hi @rafa-be
No, it is something that I can volunteer. it is a good opportunity to me. I provide a draft pr, so slowly i can iterate on your supervision.

[rafa-be]

Hi @rafa-be No, it is something that I can volunteer. it is a good opportunity to me. I provide a draft pr, so slowly i can iterate on your supervision.

Please do 😃 . I'd suggest you keep the implementation as separated as possible from Python's (possibly even creating a new repository), limiting the dependency to the Can'p protocol files.

[bcncpp]

I use my fork, i create a folder native_scheduler that does the game,

[bcncpp]

1. Scheduler uses:

   • SchedulerConfig for configuration.
   • ZMQConfig for monitor address setup.
   • AsyncBinder to bind to the main address.
   • AsyncConnector for monitor connections.
   • VanillaClientManager to manage client interactions.
   • VanillaObjectManager for object-related operations.
   • VanillaGraphTaskManager for managing graph tasks.
   • VanillaTaskManager for task management.
   • VanillaWorkerManager to manage worker interactions.
   • StatusReporter to report statuses.

2. All manager classes (Vanilla*Manager) and StatusReporter are registered with dependencies like:

   • binder
   • binder_monitor
   • Other relevant managers (e.g., client_manager, task_manager).