Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: cilium scale pipeline #2085

Merged
merged 75 commits into from
Aug 14, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
f1fa9d9
ci: adding scale pipeline for cilium
camrynl Jul 25, 2023
6d79671
change timeout
camrynl Jul 26, 2023
f689cdf
change timeout
camrynl Jul 26, 2023
b523941
update net policies
camrynl Jul 26, 2023
41a1b87
use acn build pool to avoid azp delays
camrynl Jul 26, 2023
b68f169
address comments -- update build pool var and remove timeouts from scale
camrynl Jul 27, 2023
b6c38d7
address comments -- set node/pod counts and test input as variables
camrynl Jul 28, 2023
a62049d
remove all test resources
camrynl Jul 28, 2023
44d56cb
check apachebench rollout status
camrynl Jul 31, 2023
6dd4214
collect more cpu/mem results
camrynl Aug 1, 2023
eab0e41
add cns restart and fix artifact upload
camrynl Aug 1, 2023
5583310
add cns restart and fix artifact upload
camrynl Aug 1, 2023
ab33ad2
update name for artifact publishing
camrynl Aug 1, 2023
d320547
update apachebench artifact collection
camrynl Aug 1, 2023
5c918f0
update apachebench artifact collection
camrynl Aug 1, 2023
74c6a1d
test cns version check
camrynl Aug 1, 2023
0d10ded
change cns update
camrynl Aug 1, 2023
57e3356
update artifact directory name
camrynl Aug 1, 2023
08960d7
add netperf testing stage
camrynl Aug 2, 2023
4dc5ce8
give permissions to netperf script
camrynl Aug 2, 2023
b563f81
change netperf steps
camrynl Aug 2, 2023
f2d6660
update path to netperf yaml
camrynl Aug 2, 2023
9b619d3
change netperf deployment to 2
camrynl Aug 2, 2023
93042c6
get correct pods in netperf script
camrynl Aug 2, 2023
50c66be
publish netperf results
camrynl Aug 2, 2023
fe1e503
publish netperf results
camrynl Aug 2, 2023
cfe4dc6
add same vm test for netperf
camrynl Aug 3, 2023
d3dcad8
netperf script to print pod values
camrynl Aug 3, 2023
4417a4b
netperf find nodes logic
camrynl Aug 3, 2023
b783de2
netperf find nodes logic
camrynl Aug 4, 2023
6528aea
ci: adding scale pipeline for cilium
camrynl Jul 25, 2023
093abeb
change timeout
camrynl Jul 26, 2023
d3d6e1d
change timeout
camrynl Jul 26, 2023
e0b1219
update net policies
camrynl Jul 26, 2023
9243ebb
use acn build pool to avoid azp delays
camrynl Jul 26, 2023
aa81448
address comments -- update build pool var and remove timeouts from scale
camrynl Jul 27, 2023
e0cc798
address comments -- set node/pod counts and test input as variables
camrynl Jul 28, 2023
dee90f7
remove all test resources
camrynl Jul 28, 2023
6a8c524
check apachebench rollout status
camrynl Jul 31, 2023
ce9825b
collect more cpu/mem results
camrynl Aug 1, 2023
688d35d
add cns restart and fix artifact upload
camrynl Aug 1, 2023
4505797
add cns restart and fix artifact upload
camrynl Aug 1, 2023
b51c478
update name for artifact publishing
camrynl Aug 1, 2023
c4df0ad
update apachebench artifact collection
camrynl Aug 1, 2023
c6df1c6
update apachebench artifact collection
camrynl Aug 1, 2023
c34720a
test cns version check
camrynl Aug 1, 2023
9589c45
change cns update
camrynl Aug 1, 2023
a91dd8b
update artifact directory name
camrynl Aug 1, 2023
6c91845
add netperf testing stage
camrynl Aug 2, 2023
1e7007b
give permissions to netperf script
camrynl Aug 2, 2023
4dbde07
change netperf steps
camrynl Aug 2, 2023
e4dfd3d
update path to netperf yaml
camrynl Aug 2, 2023
5570052
change netperf deployment to 2
camrynl Aug 2, 2023
fbda33a
get correct pods in netperf script
camrynl Aug 2, 2023
366ddb5
publish netperf results
camrynl Aug 2, 2023
2de4cd8
publish netperf results
camrynl Aug 2, 2023
ebe6f54
add same vm test for netperf
camrynl Aug 3, 2023
4da84e2
netperf script to print pod values
camrynl Aug 3, 2023
52ddeb9
netperf find nodes logic
camrynl Aug 3, 2023
e611c08
netperf find nodes logic
camrynl Aug 4, 2023
047a4ad
Merge branch 'camryn/ciliumscaletest' of https://github.com/Azure/azu…
camrynl Aug 9, 2023
a35ce9a
address comments, add more cpu and mem collection, and rename artifac…
camrynl Aug 9, 2023
cbfeab7
address comments for labeling nodes
camrynl Aug 9, 2023
ddfe7e9
fix artifact directory name
camrynl Aug 10, 2023
bea0975
fix artifact directory name apache
camrynl Aug 10, 2023
4a81387
fix artifact directory name apache
camrynl Aug 10, 2023
45cbda9
fix artifact directory name netperf
camrynl Aug 10, 2023
b2a08b1
fix artifact directory name netperf in script
camrynl Aug 10, 2023
12c547b
Merge branch 'master' of https://github.com/camrynl/azure-container-n…
camrynl Aug 11, 2023
e55497c
Merge branch 'master' into camryn/ciliumscaletest
camrynl Aug 11, 2023
d8aa8e6
Merge branch 'master' of https://github.com/camrynl/azure-container-n…
camrynl Aug 11, 2023
9341075
Merge branch 'camryn/ciliumscaletest' of https://github.com/Azure/azu…
camrynl Aug 11, 2023
d64ed40
Merge branch 'master' into camryn/ciliumscaletest
camrynl Aug 14, 2023
5dd541a
Merge branch 'master' into camryn/ciliumscaletest
camrynl Aug 14, 2023
0225f3c
Merge branch 'master' into camryn/ciliumscaletest
camrynl Aug 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
409 changes: 409 additions & 0 deletions .pipelines/cni/cilium/cilium-scale-test.yaml

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions hack/aks/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -264,3 +264,6 @@ down: ## Delete the cluster

restart-vmss: ## Restarts the nodes in the cluster
$(AZCLI) vmss restart -g MC_${GROUP}_${CLUSTER}_${REGION} --name $(VMSS_NAME)

scale-vmss: ## Scales the nodes in the cluster
$(AZCLI) vmss scale -g MC_${GROUP}_${CLUSTER}_${REGION} --name $(VMSS_NAME) --new-capacity $(NODE_COUNT)
18 changes: 18 additions & 0 deletions hack/manifests/apache.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: apachebench
spec:
selector:
matchLabels:
app: apachebench
template:
metadata:
labels:
app: apachebench
spec:
containers:
- name: ubuntu-tools
image: tamilmani1989/ubuntu18-tools
command: ["/bin/sleep", "3650d"]

28 changes: 28 additions & 0 deletions hack/manifests/netperf-pod.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: container6
namespace: default
spec:
selector:
matchLabels:
app: container6
replicas: 3
template: # create pods using pod definition in this template
metadata:
# unlike pod-nginx.yaml, the name is not included in the meta data as a unique name is
# generated from the deployment name
labels:
app: container6
netperf: "true"
spec:
nodeSelector:
netperf: "true"
containers:
- name: ubuntu
image: tamilmani1989/ubuntu18-tools
imagePullPolicy: Always
command: ["/bin/sh","-c"]
args: ["echo helloworld>hello.txt; php -S 0.0.0.0:9568"]
securityContext:
privileged: true
55 changes: 55 additions & 0 deletions hack/scripts/netperf.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
#!/bin/bash
# find the nodes with netperf pods and assign test vars
node_found=0
for node in $(kubectl get nodes -o name);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can limit this result to three if we use the label selector netperf=true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, in label-nodes.sh I cut off the netperf=true labeling once 3 nodes have it

do
if [ $node_found -lt 2 ]; then
echo "Current : $node"
node_name="${node##*/}"
echo "checking whether the node has any netperf pods deployed to it"
pod_count=$(kubectl get pods -l app=container6 -o wide | grep "$node_name" -c)
netperf_pod=$(kubectl get pods -l app=container6 -o wide | grep "$node_name" | awk '{print $1}')
echo "netperf pod : $netperf_pod"
echo "pod_count: $pod_count"

if [ $pod_count -gt 1 ]; then
target_pod=$(echo $netperf_pod | cut -d" " -f 1)
target_pod_ip=$(kubectl get pod "$target_pod" -o jsonpath='{.status.podIP}')
same_vm_pod=$(echo $netperf_pod | cut -d" " -f 2)
kubectl exec -it $target_pod -- netserver
node_found=$((node_found + 1))
echo "Number of nodes found with netperf pod: $node_found"
else
diff_vm_pod=$netperf_pod
node_found=$((node_found + 1))
echo "Number of nodes found with netperf pod: $node_found"
fi
fi
done

echo "target netperf pod: $target_pod"
echo "target netperf pod IP: $target_pod_ip"
echo "same vm pod: $same_vm_pod"
echo "different vm pod: $diff_vm_pod"

#netperf on same vm pod
iteration=10
while [ $iteration -ge 0 ]
do
echo "============ Iteration $iteration ==============="
kubectl exec -it $same_vm_pod -- netperf -H $target_pod_ip -l 30 -t TCP_STREAM >> "test3_netperf/same_vm_iteration_$iteration.log"
echo "==============================="
sleep 5s
iteration=$((iteration-1))
done

#netperf on different vm pod
iteration=10
while [ $iteration -ge 0 ]
do
echo "============ Iteration $iteration ==============="
kubectl exec -it $diff_vm_pod -- netperf -H $target_pod_ip -l 30 -t TCP_STREAM >> "test3_netperf/diff_vm_iteration_$iteration.log"
echo "==============================="
sleep 5s
iteration=$((iteration-1))
done
9 changes: 8 additions & 1 deletion test/scale/label-nodes.sh
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
#!/bin/sh
cmd=$1
retries=0
node_count=0
while [ $retries -lt 5 ]; do
$cmd
if [ $? -eq 0 ]; then
Expand All @@ -17,15 +18,21 @@ fi

for node in $(kubectl get nodes -o name);
do
node_count=$((node_count + 1))
echo $node_count
echo "Current : $node"
node_name="${node##*/}"
echo "Apply label to the node"
kubectl label node $node_name connectivity-test=true
kubectl label node $node_name scale-test=true
if [ $node_count -lt 3 ]; then
kubectl label node $node_name netperf=true
echo "labeled node for netperf testing"
fi
if [ $? -eq 0 ]; then
echo "Label applied to the node"
else
echo "Error in applying label to the node $node_name"
fi
sleep 2s
sleep 1s
done