CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases #2003

vipul-21 · 2023-06-06T19:21:41Z

Reason for Change:

The bash scripts added as a part of load testing Pipeline are not scalable. This PR:

Remove the bash scripts (scale_deployment.sh) and ('validate_state.sh`) usage from CNI load test pipeline and ACN PR pipeline
Rearranged the utils package to be used for different test scenarios.
Added the load test package with following test scenarios:
- TestLoad - This will be used to test the pod deployments. It will scale up/down the pods certain times and check if all the pods are running and have IPs as well. By default, this test does not validate the state files but can do that via flag -validate-statefile=true
- TestValidate - This will be used to validate the state files based on the OS and cni type.
- TestScaleDeployment - This will be used to scale deployments up to certain replicas and check if they are deployed correctly. We have a flag that can skip that check as well.
It adds a validate package that will be used to validate different state file based on OS and CNI type. Currently only linux + cilium is supported. Will be working on adding more to it.

Testing:
CNI Load test pipeline: https://msazure.visualstudio.com/One/_build/results?buildId=74505898&view=results
ACN : https://msazure.visualstudio.com/One/_build/results?buildId=74493447&view=results

Document for using the test cases:
Issue Fixed:

CNI dependencies on bash scripts to run the load test and validation of state files.

Requirements:

uses conventional commit messages
includes documentation
adds unit tests

Notes:

vipul-21 · 2023-06-07T05:43:13Z

/azp run

azure-pipelines · 2023-06-07T05:43:27Z

Azure Pipelines successfully started running 2 pipeline(s), but failed to run 2 pipeline(s).

vipul-21 · 2023-06-07T18:01:26Z

/azp run

azure-pipelines · 2023-06-07T18:01:51Z

Azure Pipelines successfully started running 2 pipeline(s), but failed to run 2 pipeline(s).

azure-pipelines · 2023-06-08T16:26:29Z

Azure Pipelines successfully started running 1 pipeline(s).

vipul-21 · 2023-06-08T16:33:06Z

/azp run CNI Load test

azure-pipelines · 2023-06-08T16:33:14Z

Azure Pipelines successfully started running 1 pipeline(s).

jpayne3506 · 2023-06-08T00:29:13Z

.pipelines/cni/cilium/cilium-cni-load-test.yaml

            name: "RestartNodes"
            displayName: "Restart Nodes"
          - script: |
-              bash hack/scripts/validate_state.sh -r true
+              cd test/integration/load
+              go test -timeout 30m -tags load -run ^TestValidateState$ -tags=load -restart-case=true


Seeing that this go test will be called multiple times it may use bad data from the caching system within the testing package. You can avoid this with the use of flag -count=1. Ex. go test -count=1 -timeout 30m ...
https://pkg.go.dev/cmd/go#hdr-Test_packages To disable test caching, use any test flag or argument other than the cacheable flags. The idiomatic way to disable test caching explicitly is to use -count=1.

test/validate/linux_validate.go

vipul-21 · 2023-06-08T17:46:15Z

/azp run CNI Load test

azure-pipelines · 2023-06-08T17:46:24Z

Azure Pipelines successfully started running 1 pipeline(s).

vipul-21 · 2023-06-08T18:13:36Z

/azp run CNI Load test

azure-pipelines · 2023-06-08T18:13:45Z

Azure Pipelines successfully started running 1 pipeline(s).

vipul-21 · 2023-06-08T19:13:23Z

/azp run CNI Load test

azure-pipelines · 2023-06-08T19:13:32Z

Azure Pipelines successfully started running 1 pipeline(s).

test/integration/load/load_test.go

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45)

* build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test (#1846) build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test for parity with release image Signed-off-by: Evan Baker <[email protected]> (cherry picked from commit f619259) * ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> (cherry picked from commit 024819d) * CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45) * ci: [CNI] Move Nightly Cilium Pipeline test to ACN (#1963) * CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]> (cherry picked from commit 1514d95) * ci:[CNI] Add windows CNIv1 datapath test (#2016) * ci: Transfer files * test: Working Datapath Test * test: apierror Tests * style: Datapath Package * test: Deployment timing * fix: Error check * fix: Lint (cherry picked from commit 390977d) * fix: [CNI] CNI load test failing due to namespace already created (#2031) fix: CNI load test failing due to namespace already created (cherry picked from commit c10900e) * ci:[CNI] Windows cniv1 load test pipeline (#2024) CI:[CNI] Windows cniv1 load test pipeline (cherry picked from commit e45ad21) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test (#2052) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test * Add validate step to the pipeline * Adding the telemetry config to the cluster (cherry picked from commit 846e508) * ci:[CNI] Replace AKS-Engine Tests with k8s conformance tests (#2062) * Initial Commit * Add attempts to prevent flakyness * Add taint for windows tests * Add k8s e2e tests * Testing vmSizes * Artifact k8se2e binary * Remove NPM E2E * Add testing and increase processes * Addressing comments (cherry picked from commit 451c691) * CI: Removing AKS engine related code (#2089) (cherry picked from commit b45c2c7) * feat: [dropgz] Dropgz for windows (#2075) * feat: [dropgz] Dropgz for windows * Removing the code for killing the process from dropgz for windows (cherry picked from commit 7a41178) * ci: Update dns tests for k8s conformance (#2104) Update dns tests for k8s v1.26 (cherry picked from commit bbf2fd4) * ci: adding cni package as a trigger (#2108) (cherry picked from commit e6a8ea6) * ci: add packages for submodule trigger (#2154) (cherry picked from commit 4aecfd6) * set mellanox reg key (#1768) (cherry picked from commit fa2de6d) --------- Co-authored-by: Evan Baker <[email protected]> Co-authored-by: Camryn Lee <[email protected]> Co-authored-by: Vipul Singh <[email protected]> Co-authored-by: Rajvi <[email protected]>

vipul-21 added cni Related to CNI. ci Infra or tooling. labels Jun 6, 2023

vipul-21 force-pushed the singhvipul/cnitesting branch 15 times, most recently from d3aac31 to 5954e0a Compare June 7, 2023 04:45

vipul-21 marked this pull request as ready for review June 7, 2023 18:02

vipul-21 requested a review from a team as a code owner June 7, 2023 18:02

vipul-21 requested a review from jaer-tsun June 7, 2023 18:02

vipul-21 enabled auto-merge (squash) June 7, 2023 18:03

vipul-21 force-pushed the singhvipul/cnitesting branch 5 times, most recently from b3127bc to 02410a8 Compare June 7, 2023 22:03

jpayne3506 reviewed Jun 8, 2023

View reviewed changes

vipul-21 force-pushed the singhvipul/cnitesting branch 2 times, most recently from 0375e42 to bda7277 Compare June 8, 2023 17:40

vipul-21 force-pushed the singhvipul/cnitesting branch 2 times, most recently from 8061d50 to 6390020 Compare June 8, 2023 19:11

vipul-21 requested a review from jpayne3506 June 8, 2023 19:42

jpayne3506 requested changes Jun 8, 2023

View reviewed changes

test/integration/load/load_test.go Show resolved Hide resolved

vipul-21 force-pushed the singhvipul/cnitesting branch from 6390020 to 813525c Compare June 8, 2023 21:13

CI:[CNI] Replace the bash scripts with the golang test cases

2ded00a

vipul-21 force-pushed the singhvipul/cnitesting branch from 813525c to 2ded00a Compare June 8, 2023 21:14

vipul-21 disabled auto-merge June 8, 2023 21:27

jpayne3506 approved these changes Jun 9, 2023

View reviewed changes

camrynl approved these changes Jun 9, 2023

View reviewed changes

vipul-21 merged commit 008ae45 into master Jun 9, 2023

vipul-21 deleted the singhvipul/cnitesting branch June 9, 2023 00:17

paulyufan2 pushed a commit that referenced this pull request Jun 9, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang t…

1bbf782

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases

jpayne3506 pushed a commit that referenced this pull request Sep 11, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang t…

b7bf170

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45)

jpayne3506 pushed a commit that referenced this pull request Sep 12, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang t…

bb2aeb3

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45)

jpayne3506 pushed a commit that referenced this pull request Sep 12, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang t…

695506d

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45)

jpayne3506 pushed a commit that referenced this pull request Sep 17, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang t…

f97aa82

…est cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases #2003

CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases #2003

vipul-21 commented Jun 6, 2023 •

edited

Loading

vipul-21 commented Jun 7, 2023

azure-pipelines bot commented Jun 7, 2023

vipul-21 commented Jun 7, 2023

azure-pipelines bot commented Jun 7, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

jpayne3506 Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases #2003

CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases #2003

Conversation

vipul-21 commented Jun 6, 2023 • edited Loading

vipul-21 commented Jun 7, 2023

azure-pipelines bot commented Jun 7, 2023

vipul-21 commented Jun 7, 2023

azure-pipelines bot commented Jun 7, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

jpayne3506 Jun 8, 2023

Choose a reason for hiding this comment

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 8, 2023

azure-pipelines bot commented Jun 8, 2023

vipul-21 commented Jun 6, 2023 •

edited

Loading