Wait for nodes to be ready before proceeding with upgrade. #3190
Conversation
Near the end of node upgrade, we now wait for the node to report Ready before marking it schedulable again. This should help eliminate delays when pods need to relocate as the next node in line is evacuated. Happens near the end of the process, the only remaining task would be to mark it schedulable again so easy for admins to detect and recover from.
|
aos-ci-test |
|
aos-ci-test |
|
3e28106 - State: success - All Test Contexts: aos-ci-jenkins/OS_unit_tests - Logs: https://aos-ci.s3.amazonaws.com/openshift/openshift-ansible/jenkins-openshift-ansible-2-unit-tests-771/3e2810638dda9436697c88d88ddf91a11de287ef.txt |
|
3e28106 - State: success - All Test Contexts: "aos-ci-jenkins/OS_3.4_NOT_containerized, aos-ci-jenkins/OS_3.4_NOT_containerized_e2e_tests" - Logs: https://aos-ci.s3.amazonaws.com/openshift/openshift-ansible/jenkins-openshift-ansible-3-test-matrix-CONTAINERIZED=_NOT_containerized,OSE_VER=3.4,PYTHON=System-CPython-2.7,TOPOLOGY=openshift-cluster,TargetBranch=release-1.4,nodes=openshift-ansible-slave-776/3e2810638dda9436697c88d88ddf91a11de287ef.txt |
|
3e28106 - State: success - All Test Contexts: "aos-ci-jenkins/OS_3.4_containerized, aos-ci-jenkins/OS_3.4_containerized_e2e_tests" - Logs: https://aos-ci.s3.amazonaws.com/openshift/openshift-ansible/jenkins-openshift-ansible-3-test-matrix-CONTAINERIZED=_containerized,OSE_VER=3.4,PYTHON=System-CPython-2.7,TOPOLOGY=openshift-cluster-containerized,TargetBranch=release-1.4,nodes=openshift-ansible-slave-776/3e2810638dda9436697c88d88ddf91a11de287ef.txt |
|
Andrew is fixing travis jobs on release-1.4 and release-1.3 real soon now. |
Backport from master. Without this you can end up with all nodes 'NotReady' for a brief period.
Near the end of node upgrade, we now wait for the node to report Ready
before marking it schedulable again. This should help eliminate delays
when pods need to relocate as the next node in line is evacuated.
Happens near the end of the process, the only remaining task would be to
mark it schedulable again so easy for admins to detect and recover from.
( discussed pulling this in with @sdodson )