Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Ballista] Introduce StageManager for managing tasks stage by stage #1936

Closed
yahoNanJing opened this issue Mar 7, 2022 · 2 comments · Fixed by #1983
Closed

[Ballista] Introduce StageManager for managing tasks stage by stage #1936

yahoNanJing opened this issue Mar 7, 2022 · 2 comments · Fixed by #1983
Labels
enhancement New feature or request

Comments

@yahoNanJing
Copy link
Contributor

yahoNanJing commented Mar 7, 2022

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

It's better to manage tasks by stages rather than mess them up for the whole system or the whole job. With Stage, it will be much easier to maintain the task status changes within a stage, which will also be easier for future task error handling. And it will also be easier to manage the stage status changing and error handling.

Describe the solution you'd like

  • Introduce Stage for a bunch of tasks for a stage.
  • Introduce StageManager for managing all of the stages.

With this change, it will be much efficient to fetch a task to be scheduled from some Stage. And it also makes it possible to schedule tasks by the priority of stages.

Additional context

#1704

@yahoNanJing yahoNanJing added the enhancement New feature or request label Mar 7, 2022
@yahoNanJing yahoNanJing changed the title [Ballista] Introduce TaskSet and TaskSetStore for managing tasks of one stage [Ballista] Introduce StageManager for managing tasks stage by stage Mar 9, 2022
@yahoNanJing
Copy link
Contributor Author

yahoNanJing commented Mar 9, 2022

There are three levels of job state: task -> stage -> job.

For the state machine of task:
task_status_state_machine

For the state machine of stage:
stage_state_machine

For the state machine of job:
job_state_machine

@yahoNanJing
Copy link
Contributor Author

yahoNanJing commented Mar 10, 2022

Sequential diagram for Pull-based task scheduling:
Pull-based Task Scheduling

Sequential diagram for Push-based task scheduling:
Push-based Task Scheduling

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant