add cooldown protection plugin #2149

flyhighzy · 2022-04-06T10:53:42Z

resolve issue #2075

Add cooldown time support for preempt action:

provide a new label/annotation named "volcano.sh/preempt-stable-time", whose value means the cool down time for preempt with unit second. This label/annotation can be set for entire vcjob or some dedicated tasks, if set to job, we'll transfer to all tasks' pods.
add a plugin to participate in preempt action, ensure pods whose scheduled time after now - preempt-stable-time will be not in the result victims list

volcano-sh-bot · 2022-04-06T10:53:45Z

Welcome @flyhighzy!

It looks like this is your first PR to volcano-sh/volcano 馃帀.

Thank you, and welcome to Volcano. 😃

flyhighzy · 2022-04-08T08:06:20Z

/assign @shinytang6

flyhighzy · 2022-04-13T02:49:14Z

add volcano.sh/api pr #68, please have a review

Thor-wl · 2022-04-15T03:19:51Z

Please update the vendor locally.

flyhighzy · 2022-04-25T07:36:19Z

thanks for the advice and please have a review :)
since this pr depends on pr of. volcano.sh/api, please also review api's pull request as well
@hudson741 @hzxuzhonghu @wpeng102 @shinytang6

shinytang6

Generally lgtm, can we have a simple doc to claim how to use this plugin?
btw, it seems that this pr contains some api changes that causing ci failed )

shinytang6 · 2022-04-27T15:16:10Z

pls help resolve the conflicts.

flyhighzy · 2022-04-28T02:26:48Z

pls help resolve the conflicts.

updated

Thor-wl · 2022-07-08T09:33:19Z

docs/user-guide/how_to_use_stable_preempt_plugin.md

+#### Edit yaml of vcjob
+
+1. add annotations in volcano job in format below.
+   1. `preemptable` annotation(or label) indicates that job or task is preemptable


IMO, annotation is better for there exists preemption annotaion volcano.sh/preemptable: "true".

Thor-wl

Thanks for your contribution! It's generally LGTM for me now.
/lgtm

william-wang · 2022-07-08T10:20:08Z

docs/user-guide/how_to_use_stable_preempt_plugin.md

+      - name: priority
+      - name: gang
+      - name: conformance
+      - name: stablepreempt


I strongly suggest change another plugin name instead of stablepreempt.
It seems current preempt is not stable without this plugin:)

How about "preempt-protected"?
or any other suggestions for the new plugin name?

Let's name it cdt plugin which means cool down time or cdp which means cool down period. It can be used not only in preemption but also in reclaim as well.

@william-wang @flyhighzy I don't recommend using this name. There may be multiple ways to cool down victim tasks, not only by time interval. The name limits the functionality of the plugin and makes the functionality of this plugin slightly thin. How about vcp which means victim cooldown protection or ecp which means evicted cooldown protection.

@Jason-Liu-Dream I noticed your issue, but maybe you need to put issue in the right repo :). I think it sounds reasonable, but may need to discuss more details.

So what's your opinions? @william-wang @Thor-wl

@Jason-Liu-Dream I noticed your issue, but maybe you need to put issue in the right repo :). I think it sounds reasonable, but may need to discuss more details.

this issue is put here because our inner version implements a demo of this function, we want to combine your design, and retry to design our implements.

@Jason-Liu-Dream Would you clariy some other vailid cool down ways besides the time/period?

@william-wang We think cool down reason maybe need to consider time periods， eviction times， pod height load pressure, block messages, message events and so on and different business scenarios have different requirements.

@Jason-Liu-Dream Thanks for the explaining the different cases for the cool down operation. If possible you can elaborate on pod height load pressure, block messages, message events.

@flyhighzy I think we can refine the name to clp which means cool down protection. It can cover most of use cases so far :)

sure, I think so :)

@Jason-Liu-Dream Thanks for the explaining the different cases for the cool down operation. If possible you can elaborate on pod height load pressure, block messages, message events.
@william-wang
pod height load pressure: Sometimes we need to preempt some pods, but we don't want to preempt it when this pod is at height load pressure, height load pressure means a large scope of influence after stopping service in this pod. And we don't know how long this will last. so cooldown time does not work.
block messages: similar to above, sometimes message block in queue means there is a lot of stress to deal with in this service. Preempting is not a good idea.
message events: In batch tasks, we can record progress. Assuming that a task has N stages, and each stage supports breakpoints to continue working, then the best plan is to complete the goal of the previous stage, send a message event, and then let this task be preempted so that not cause excessive waste of resources. This is very common in pipeline jobs such as video transcoding.

william-wang · 2022-07-08T10:28:23Z

docs/user-guide/how_to_use_stable_preempt_plugin.md

+
+        ```yaml
+            volcano.sh/preemptable: "true"
+            volcano.sh/preempt-stable-time: "600s"


As you described in the background, i think volcano.sh/cooldown-time is more strightforward.

Job label changes already discussed in pr#68, so we still need to change this label name again?

@flyhighzy No worry, just change it and let's do things right the first time :)

william-wang · 2022-07-08T10:33:50Z

pkg/scheduler/plugins/stablepreempt/stable_preempt.go

+	// PluginName is name of plugin
+	PluginName = "stablepreempt"
+)
+


Suggest add a comment here to describe this new plugin.

Thor-wl

I'm generally OK about the modifications. Please rebase all the commits into one and push again.

flyhighzy · 2022-07-14T02:47:31Z

I've finished rebase all the commits into one and rename plugin name to "cdp", pls take a review again~
@Thor-wl @william-wang

Signed-off-by: Zhao Ying <[email protected]> update preempt_stable_time label value to time duration Signed-off-by: Zhao Ying <[email protected]> update local vendor and api dependency Signed-off-by: Zhao Ying <[email protected]> re-generate configs and add user guid Signed-off-by: Zhao Ying <[email protected]> rename stablepreempt plugin to cdt Signed-off-by: Zhao Ying <[email protected]>

william-wang

Overall, looks good to me.
/approve

volcano-sh-bot · 2022-07-15T08:25:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: william-wang

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [william-wang]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

volcano-sh-bot requested review from hudson741, hzxuzhonghu and wpeng102 April 6, 2022 10:53

volcano-sh-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Apr 6, 2022

flyhighzy force-pushed the preempt-stable-time branch from 127c44a to 170234b Compare April 6, 2022 12:18

volcano-sh-bot assigned shinytang6 Apr 8, 2022

flyhighzy mentioned this pull request Apr 11, 2022

add stable preempt label volcano-sh/apis#68

Merged

Thor-wl requested review from william-wang, shinytang6 and k82cn and removed request for hudson741 and hzxuzhonghu April 25, 2022 08:18

flyhighzy force-pushed the preempt-stable-time branch from dad0c8e to 0f78a7b Compare April 25, 2022 09:28

volcano-sh-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 25, 2022

flyhighzy force-pushed the preempt-stable-time branch from 0f78a7b to 44ad4b3 Compare April 25, 2022 09:35

volcano-sh-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Apr 25, 2022

shinytang6 reviewed Apr 26, 2022

View reviewed changes

volcano-sh-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 26, 2022

volcano-sh-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Apr 28, 2022

Thor-wl self-requested a review July 8, 2022 09:12

Thor-wl reviewed Jul 8, 2022

View reviewed changes

volcano-sh-bot assigned Thor-wl Jul 8, 2022

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 8, 2022

william-wang reviewed Jul 8, 2022

View reviewed changes

flyhighzy mentioned this pull request Jul 12, 2022

update PreemptStableTime to CooldownTime volcano-sh/apis#82

Merged

Thor-wl reviewed Jul 12, 2022

View reviewed changes

flyhighzy force-pushed the preempt-stable-time branch 2 times, most recently from 5b713ea to baaf855 Compare July 14, 2022 02:44

flyhighzy force-pushed the preempt-stable-time branch from baaf855 to 0d0c061 Compare July 14, 2022 02:55

flyhighzy mentioned this pull request Jul 15, 2022

Request to be an member of Volcano community volcano-sh/community#29

Closed

william-wang changed the title ~~add stable preempt plugin~~ add cooldown protection plugin Jul 15, 2022

william-wang approved these changes Jul 15, 2022

View reviewed changes

volcano-sh-bot assigned william-wang Jul 15, 2022

volcano-sh-bot added the lgtm Indicates that a PR is ready to be merged. label Jul 15, 2022

volcano-sh-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 15, 2022

volcano-sh-bot merged commit 9def57e into volcano-sh:master Jul 15, 2022

This was referenced Jul 19, 2022

Support protection of the maximum number of cooldown in cdp plugin #2362

Closed

update cooldown protection plugin and supports the protection of the maximum number of cooldown. #2363

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add cooldown protection plugin #2149

add cooldown protection plugin #2149

flyhighzy commented Apr 6, 2022

volcano-sh-bot commented Apr 6, 2022

flyhighzy commented Apr 8, 2022

flyhighzy commented Apr 13, 2022

Thor-wl commented Apr 15, 2022

flyhighzy commented Apr 25, 2022 •

edited

Loading

shinytang6 left a comment

shinytang6 commented Apr 27, 2022

flyhighzy commented Apr 28, 2022

Thor-wl Jul 8, 2022

Thor-wl left a comment

william-wang Jul 8, 2022

flyhighzy Jul 11, 2022

william-wang Jul 12, 2022

Jason-Liu-Dream Jul 12, 2022

flyhighzy Jul 13, 2022

Jason-Liu-Dream Jul 13, 2022

Jason-Liu-Dream Jul 13, 2022 •

edited

Loading

william-wang Jul 14, 2022

flyhighzy Jul 14, 2022

Jason-Liu-Dream Jul 19, 2022

william-wang Jul 8, 2022

flyhighzy Jul 11, 2022

william-wang Jul 12, 2022

william-wang Jul 8, 2022

Thor-wl left a comment

flyhighzy commented Jul 14, 2022

william-wang left a comment

volcano-sh-bot commented Jul 15, 2022

add cooldown protection plugin #2149

add cooldown protection plugin #2149

Conversation

flyhighzy commented Apr 6, 2022

volcano-sh-bot commented Apr 6, 2022

flyhighzy commented Apr 8, 2022

flyhighzy commented Apr 13, 2022

Thor-wl commented Apr 15, 2022

flyhighzy commented Apr 25, 2022 • edited Loading

shinytang6 left a comment

Choose a reason for hiding this comment

shinytang6 commented Apr 27, 2022

flyhighzy commented Apr 28, 2022

Choose a reason for hiding this comment

Thor-wl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Jason-Liu-Dream Jul 13, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thor-wl left a comment

Choose a reason for hiding this comment

flyhighzy commented Jul 14, 2022

william-wang left a comment

Choose a reason for hiding this comment

volcano-sh-bot commented Jul 15, 2022

flyhighzy commented Apr 25, 2022 •

edited

Loading

Jason-Liu-Dream Jul 13, 2022 •

edited

Loading