Skip to content

Commit c8c74d8

Browse files
jpayne3506jpayne3506
committed
ci: [CNI] Move Nightly Cilium Pipeline test to ACN (#1963)
* CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]> (cherry picked from commit 1514d95)
1 parent 52686bc commit c8c74d8

16 files changed

+505
-10
lines changed

‎.pipelines/cni/cilium/cilium-cni-load-test.yaml

+3
Original file line numberDiff line numberDiff line change
@@ -48,6 +48,9 @@ stages:
4848
echo "install Cilium onto Overlay Cluster"
4949
kubectl apply -f test/integration/manifests/cilium/cilium-agent
5050
kubectl apply -f test/integration/manifests/cilium/cilium-operator
51+
# Passes Cilium image to daemonset and deployment
52+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
53+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
5154
kubectl get po -owide -A
5255
echo "deploy ip-masq-agent for overlay"
5356
kubectl create -f test/integration/manifests/ip-masq-agent/ip-masq-agent.yaml --validate=false

‎.pipelines/cni/cilium/nightly-release-test.yml

+54
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
pr: none
2+
trigger: none
3+
4+
stages:
5+
- stage: init
6+
displayName: "Build and Push Cilium Image"
7+
jobs:
8+
- job: build_and_push_cilium_image
9+
steps:
10+
- bash: |
11+
set -ex
12+
cd .pipelines/
13+
git clone https://github.com/cilium/cilium.git
14+
cd cilium
15+
make docker-cilium-image
16+
make docker-operator-generic-image
17+
name: BuildCiliumImage
18+
displayName: "Build Cilium Image"
19+
- task: Docker@2
20+
displayName: Login
21+
inputs:
22+
containerRegistry: $(CONTAINER_REGISTRY)
23+
command: "login"
24+
- task: AzureCLI@1
25+
inputs:
26+
azureSubscription: $(TEST_SUB_SERVICE_CONNECTION)
27+
scriptLocation: "inlineScript"
28+
inlineScript: |
29+
set -ex
30+
docker tag quay.io/cilium/cilium:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
31+
docker tag quay.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG) $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
32+
docker push $(ACR).azurecr.io/cilium/cilium:$(DOCKER_IMAGE_TAG)
33+
docker push $(ACR).azurecr.io/cilium/operator-generic:$(DOCKER_IMAGE_TAG)
34+
name: "PushCiliumImage"
35+
displayName: "Push Cilium Image"
36+
- task: Docker@2
37+
displayName: Logout
38+
inputs:
39+
containerRegistry: $(CONTAINER_REGISTRY)
40+
command: "logout"
41+
- stage: cilium_nightly
42+
displayName: E2E - Cilium Nightly
43+
variables:
44+
GOPATH: "$(Agent.TempDirectory)/go" # Go workspace path
45+
GOBIN: "$(GOPATH)/bin" # Go binaries path
46+
modulePath: "$(GOPATH)/src/github.com/Azure/azure-container-networking"
47+
jobs:
48+
- job: cilium_nightly
49+
steps:
50+
- template: ../../singletenancy/cilium-overlay/cilium-overlay-e2e-step-template.yaml
51+
parameters:
52+
name: "cilium_nightly"
53+
testDropgz: ""
54+
clusterName: "ciliumnightly"

‎.pipelines/singletenancy/cilium-overlay/cilium-overlay-e2e-step-template.yaml

+1
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,7 @@ steps:
4646
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
4747
kubectl cluster-info
4848
kubectl get po -owide -A
49+
if [ "$CILIUM_VERSION_TAG" = "cilium-nightly-pipeline" ]; then FILE_PATH=-nightly && echo "Running nightly"; fi
4950
echo "deploy Cilium ConfigMap"
5051
kubectl apply -f cilium/configmap.yaml
5152
kubectl apply -f test/integration/manifests/cilium/cilium${FILE_PATH}-config.yaml

‎.pipelines/singletenancy/cilium/cilium-e2e-step-template.yaml

+12-1
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,9 @@ steps:
4949
echo "deploy Cilium ConfigMap"
5050
kubectl apply -f cilium/configmap.yaml
5151
kubectl apply -f test/integration/manifests/cilium/cilium-config.yaml
52-
echo "install Cilium"
52+
echo "install Cilium ${CILIUM_VERSION_TAG}"
53+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/daemonset.yaml | kubectl apply -f -
54+
envsubst '${CILIUM_VERSION_TAG},${CILIUM_IMAGE_REGISTRY}' < test/integration/manifests/cilium/deployment.yaml | kubectl apply -f -
5355
kubectl apply -f test/integration/manifests/cilium/cilium-agent
5456
kubectl apply -f test/integration/manifests/cilium/cilium-operator
5557
kubectl get po -owide -A
@@ -111,6 +113,15 @@ steps:
111113
pathtoPublish: "$(Build.ArtifactStagingDirectory)/test-output"
112114
condition: always()
113115

116+
- script: |
117+
kubectl get pods -A
118+
echo "Waiting < 2 minutes for cilium to be ready"
119+
# Ensure Cilium is ready Xm\Xs
120+
cilium status --wait --wait-duration 2m
121+
retryCountOnTaskFailure: 3
122+
name: "CiliumStatus"
123+
displayName: "Cilium Status"
124+
114125
- script: |
115126
kubectl get pods -A
116127
echo "Waiting < 2 minutes for cilium to be ready"

‎test/integration/manifests/cilium/cilium-agent/clusterrole.yaml

+7
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,13 @@ rules:
7171
- create
7272
- apiGroups:
7373
- cilium.io
74+
# To synchronize garbage collection of such resources
75+
resources:
76+
- ciliumidentities
77+
verbs:
78+
- update
79+
- apiGroups:
80+
- cilium.io
7481
resources:
7582
- ciliumendpoints
7683
verbs:

‎test/integration/manifests/cilium/cilium-nightly-agent/clusterrole.yaml

+104
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRole
3+
metadata:
4+
name: cilium
5+
rules:
6+
- apiGroups:
7+
- networking.k8s.io
8+
resources:
9+
- networkpolicies
10+
verbs:
11+
- get
12+
- list
13+
- watch
14+
- apiGroups:
15+
- discovery.k8s.io
16+
resources:
17+
- endpointslices
18+
verbs:
19+
- get
20+
- list
21+
- watch
22+
- apiGroups:
23+
- ""
24+
resources:
25+
- namespaces
26+
- services
27+
- pods
28+
- endpoints
29+
- nodes
30+
verbs:
31+
- get
32+
- list
33+
- watch
34+
- apiGroups:
35+
- apiextensions.k8s.io
36+
resources:
37+
- customresourcedefinitions
38+
verbs:
39+
- list
40+
- watch
41+
# This is used when validating policies in preflight. This will need to stay
42+
# until we figure out how to avoid "get" inside the preflight, and then
43+
# should be removed ideally.
44+
- get
45+
- apiGroups:
46+
- cilium.io
47+
resources:
48+
#Naming changed from ciliumbgploadbalancerippools
49+
- ciliumloadbalancerippools
50+
- ciliumbgppeeringpolicies
51+
- ciliumclusterwideenvoyconfigs
52+
- ciliumclusterwidenetworkpolicies
53+
- ciliumegressgatewaypolicies
54+
- ciliumendpoints
55+
- ciliumendpointslices
56+
- ciliumenvoyconfigs
57+
- ciliumidentities
58+
- ciliumlocalredirectpolicies
59+
- ciliumnetworkpolicies
60+
- ciliumnodes
61+
- ciliumnodeconfigs
62+
#Added in 1.14.0 snapshot 2
63+
- ciliumcidrgroups
64+
verbs:
65+
- list
66+
- watch
67+
- apiGroups:
68+
- cilium.io
69+
resources:
70+
- ciliumidentities
71+
- ciliumendpoints
72+
- ciliumnodes
73+
verbs:
74+
- create
75+
- apiGroups:
76+
- cilium.io
77+
resources:
78+
- ciliumidentities
79+
verbs:
80+
- update
81+
- apiGroups:
82+
- cilium.io
83+
resources:
84+
- ciliumendpoints
85+
verbs:
86+
- delete
87+
- get
88+
- apiGroups:
89+
- cilium.io
90+
resources:
91+
- ciliumnodes
92+
- ciliumnodes/status
93+
verbs:
94+
- get
95+
- update
96+
- apiGroups:
97+
- cilium.io
98+
resources:
99+
- ciliumnetworkpolicies/status
100+
- ciliumclusterwidenetworkpolicies/status
101+
- ciliumendpoints/status
102+
- ciliumendpoints
103+
verbs:
104+
- patch

‎test/integration/manifests/cilium/cilium-nightly-agent/clusterrolebinding.yaml

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
apiVersion: rbac.authorization.k8s.io/v1
2+
kind: ClusterRoleBinding
3+
metadata:
4+
name: cilium
5+
roleRef:
6+
apiGroup: rbac.authorization.k8s.io
7+
kind: ClusterRole
8+
name: cilium
9+
subjects:
10+
- kind: ServiceAccount
11+
name: "cilium"
12+
namespace: kube-system

‎test/integration/manifests/cilium/cilium-nightly-agent/serviceaccount.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: "cilium"
5+
namespace: kube-system
6+

‎test/integration/manifests/cilium/cilium-nightly-config.yaml

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
apiVersion: v1
2+
data:
3+
agent-not-ready-taint-key: node.cilium.io/agent-not-ready
4+
arping-refresh-period: 30s
5+
auto-direct-node-routes: "false"
6+
bpf-lb-external-clusterip: "false"
7+
bpf-lb-map-max: "65536"
8+
bpf-lb-mode: snat
9+
bpf-map-dynamic-size-ratio: "0.0025"
10+
bpf-policy-map-max: "16384"
11+
bpf-root: /sys/fs/bpf
12+
cgroup-root: /run/cilium/cgroupv2
13+
cilium-endpoint-gc-interval: 5m0s
14+
cluster-id: "0"
15+
cluster-name: default
16+
debug: "false"
17+
disable-cnp-status-updates: "true"
18+
disable-endpoint-crd: "false"
19+
enable-auto-protect-node-port-range: "true"
20+
enable-bgp-control-plane: "false"
21+
enable-bpf-clock-probe: "true"
22+
enable-endpoint-health-checking: "false"
23+
enable-endpoint-routes: "true"
24+
enable-health-check-nodeport: "true"
25+
enable-health-checking: "true"
26+
enable-host-legacy-routing: "true"
27+
enable-hubble: "false"
28+
enable-ipv4: "true"
29+
enable-ipv4-masquerade: "false"
30+
enable-ipv6: "false"
31+
enable-ipv6-masquerade: "false"
32+
enable-k8s-terminating-endpoint: "true"
33+
enable-l2-neigh-discovery: "true"
34+
enable-l7-proxy: "false"
35+
enable-local-node-route: "false"
36+
enable-local-redirect-policy: "false"
37+
enable-metrics: "true"
38+
enable-policy: default
39+
enable-remote-node-identity: "true"
40+
enable-session-affinity: "true"
41+
enable-svc-source-range-check: "true"
42+
enable-vtep: "false"
43+
enable-well-known-identities: "false"
44+
enable-xt-socket-fallback: "true"
45+
identity-allocation-mode: crd
46+
install-iptables-rules: "true"
47+
install-no-conntrack-iptables-rules: "false"
48+
ipam: delegated-plugin
49+
kube-proxy-replacement: strict
50+
kube-proxy-replacement-healthz-bind-address: ""
51+
local-router-ipv4: 169.254.23.0
52+
metrics: +cilium_bpf_map_pressure
53+
monitor-aggregation: medium
54+
monitor-aggregation-flags: all
55+
monitor-aggregation-interval: 5s
56+
node-port-bind-protection: "true"
57+
nodes-gc-interval: 5m0s
58+
operator-api-serve-addr: 127.0.0.1:9234
59+
operator-prometheus-serve-addr: :9963
60+
preallocate-bpf-maps: "false"
61+
procfs: /host/proc
62+
prometheus-serve-addr: :9962
63+
remove-cilium-node-taints: "true"
64+
set-cilium-is-up-condition: "true"
65+
sidecar-istio-proxy-image: cilium/istio_proxy
66+
synchronize-k8s-nodes: "true"
67+
tofqdns-dns-reject-response-code: refused
68+
tofqdns-enable-dns-compression: "true"
69+
tofqdns-endpoint-max-ip-per-hostname: "50"
70+
tofqdns-idle-connection-grace-period: 0s
71+
tofqdns-max-deferred-connection-deletes: "10000"
72+
tofqdns-min-ttl: "3600"
73+
tofqdns-proxy-response-max-delay: 100ms
74+
#Replaces tunnel: disabled in v1.15
75+
routing-mode: "native"
76+
unmanaged-pod-watcher-interval: "15"
77+
vtep-cidr: ""
78+
vtep-endpoint: ""
79+
vtep-mac: ""
80+
vtep-mask: ""
81+
kind: ConfigMap
82+
metadata:
83+
annotations:
84+
meta.helm.sh/release-name: cilium
85+
meta.helm.sh/release-namespace: kube-system
86+
labels:
87+
app.kubernetes.io/managed-by: Helm
88+
name: cilium-config
89+
namespace: kube-system

0 commit comments

Comments
 (0)