Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: [CNI] Add restart node in stage in the load test of cni pipeline #1916

Merged
merged 1 commit into from
Apr 21, 2023

Conversation

vipul-21
Copy link
Contributor

@vipul-21 vipul-21 commented Apr 14, 2023

Reason for Change:

  • Added the test of restarting the nodes in the CNI load testing pipeline. It scales down the deployment and then restart the nodes without any delay.
  • Update the scale_deployment script to include different parameters for CNI load testing

Requirements:

Notes:

@vipul-21 vipul-21 requested a review from rbtr April 14, 2023 19:22
@vipul-21 vipul-21 force-pushed the singhvipul/cniloadtesting branch 2 times, most recently from 329c68f to 705f829 Compare April 18, 2023 17:46
@vipul-21 vipul-21 force-pushed the singhvipul/cniloadtesting branch from 705f829 to 7dd95d3 Compare April 20, 2023 18:08
@vipul-21 vipul-21 removed the request for review from rbtr April 20, 2023 18:15
@vipul-21 vipul-21 added cni Related to CNI. ci Infra or tooling. labels Apr 20, 2023
make -C ./hack/swift set-kubeconf AZCLI=az CLUSTER=${RESOURCE_GROUP}
make -C ./hack/swift azcfg AZCLI=az REGION=$(LOCATION)
echo "Scaling the pods down to 100 per node"
bash ./hack/scripts/scale_deployment.sh -n 0 -u 1000 -s
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it scaling down to 1000 or 100? also number of times is 0?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The script scale up and down as a by default run. So by specifying -n as 0 we are telling it not to do that.
The flag -s is used to define scale up operation. We are scaling down to 1000 pods (i.e 100 pods per node) from 240 pods per node.

echo "Restarting the nodes"
vmss_name=$(az vmss list -g MC_${RESOURCE_GROUP}_${RESOURCE_GROUP}_$(LOCATION) --query "[].name" -o tsv)
make -C ./hack/swift restart-vmss AZCLI=az CLUSTER=${RESOURCE_GROUP} REGION=$(LOCATION) VMSS_NAME=$vmss_name
bash ./hack/scripts/scale_deployment.sh -n 0 -u 1000 -c
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we calling scale script again?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similarly -c will wait until the deployment of 1000 pods is done. The same script is used.
I added the description of what the script can do. And checking the deployment status and deploying pods(scaling pods) can be done via passing flags to the script.
If we don't wait for the deployment to finish after the restart, the validate state script will fail as few of pods( sometimes) are still being created

@vipul-21 vipul-21 enabled auto-merge (squash) April 21, 2023 02:08
@vipul-21 vipul-21 merged commit dcd55b3 into master Apr 21, 2023
@vipul-21 vipul-21 deleted the singhvipul/cniloadtesting branch April 21, 2023 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci Infra or tooling. cni Related to CNI.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants