tidb-scheduler: implement kube-scheduler extender preempt verb #2510
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cofyc
changed the title
cofyc
force-pushed
the
fix1968
branch
2 times, most recently
from
5a9d0c4 to
2cf3f8c
Compare
cofyc
changed the title
cofyc
force-pushed
the
fix1968
branch
3 times, most recently
from
dd41099 to
6e274a7
Compare
|
/run-e2e-tests |
|
cc @mightyguava can you try this patch in your test environment? here are instructions on how to build a custom image: then you can update the tidb-operator image of tidb-scheduler deployment. |
|
I'm still not seeing the low priority pods get preempted. Here are the logs from the current run. tidb-scheduler.logkube-scheduler.log
Could you elaborate here? I'm not familiar with the scheduler phases. We want these low priority overprovisioning pods to be preempted so that PD (and other pods) can be scheduled on them. Why would we want to exclude these nodes from preemption? |
|
@mightyguava I forgot to tell you that scheduler policy add
Can you update the config map and try again? |
|
I made the config change and it's still not working. There's an error in the kube-scheduler logs below as you can see but there is no more info what the error is about. :-( configmap/tidb-scheduler-policytikv-scheduler.logkube-scheduler.log |
|
I guess the pod
The basic scheduler workflow in 1.14.x is like this:
at before, our extender (tidb-scheduler) does not implement preempt API, all nodes are chosen by kube-scheduler. an invalid node (ip-10-136-135-87.us-west-2.compute.internal) is nominated but this node already has a PD running and is rejected by tidb-scheduler Filter API. after this PR, |
|
Thanks! can you show the output of this command? I'd like know the current status: |
|
I'll test with your Kubernetes version (1.14.9-eks) to see why the scheduler emits these errors. Thanks for the feedback! |
|
Here's the kube-scheduler.log(updated with more logs) |
|
The potential nodes are sent to the extender but not parsed from JSON string correctly. There is a bug in Kubernetes 1.16.0 and before that the JSON tag in https://github.com/kubernetes/kubernetes/blob/v1.16.0/pkg/scheduler/api/v1/types.go#L270 |
|
hi, @mightyguava @tirsen PR is updated, can you build the image and try again? I've verified it manually with this example, the scheduler evicts all overprovisioning pods and schedule pods onto the nodes. We will add an integration or e2e test for this scenario. |
|
Yes! This works! This is awesome. The overprovisioning pods were successfully preempted, the tidb pods scheduled, and new ec2 nodes were created to run the overprovisioning pods |
|
Great! |
|
/run-e2e-tests |
1 similar comment
|
/run-e2e-tests |
weekface
approved these changes
May 25, 2020
|
/merge |
|
/run-all-tests |
|
@cofyc merge failed. |
|
/run-e2e-tests |
|
/run-cherry-picker |
sre-bot
pushed a commit
to sre-bot/tidb-operator
that referenced
this pull request
May 26, 2020
Signed-off-by: sre-bot <[email protected]>
|
cherry pick to release-1.1 in PR #2555 |
sre-bot
added a commit
that referenced
this pull request
May 26, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
fixes #1968
What is changed and how does it work?
Check List
Tests
Code changes
Side effects
Related changes
Does this PR introduce a user-facing change?: