feat: [WIN-NPM] metrics for latencies and failures #1959

huntergregory · 2023-05-11T18:52:06Z

Reason for Change:
Add Windows metrics for SysCall latencies and failures.

Also, add a metric capturing the maximum members in any single IPSet.

Issue Fixed:

Requirements:

uses conventional commit messages
includes documentation
adds unit tests

Notes:

Metrics

Latencies (in seconds) and number of failures for these operations:

List Endpoints.
Get Endpoint.
List Network.
ACLs (operation=create|update)
SetPolicies (operation=create|update|delete and is_nested=true|false)

Also, the max IPSet member metric.

Examples

Max IPSet Members

# HELP npm_windows_ipset_members_max Maximum number of members in a single IPSet
# TYPE npm_windows_ipset_members_max gauge
npm_windows_ipset_members_max 13

Get Endpoint Latency & Failures

# HELP npm_windows_get_endpoint_failure_total Number of failures while getting a single HNS endpoint
# TYPE npm_windows_get_endpoint_failure_total counter
npm_windows_get_endpoint_failure_total 0
# HELP npm_windows_get_endpoint_latency_seconds Latency in seconds to get a single HNS endpoint
# TYPE npm_windows_get_endpoint_latency_seconds histogram
npm_windows_get_endpoint_latency_seconds_bucket{le="0.016"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="0.032"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="0.064"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="0.128"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="0.256"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="0.512"} 1
npm_windows_get_endpoint_latency_seconds_bucket{le="1.024"} 2
npm_windows_get_endpoint_latency_seconds_bucket{le="2.048"} 7
npm_windows_get_endpoint_latency_seconds_bucket{le="4.096"} 7
npm_windows_get_endpoint_latency_seconds_bucket{le="8.192"} 8
npm_windows_get_endpoint_latency_seconds_bucket{le="16.384"} 8
npm_windows_get_endpoint_latency_seconds_bucket{le="32.768"} 8
npm_windows_get_endpoint_latency_seconds_bucket{le="65.536"} 8
npm_windows_get_endpoint_latency_seconds_bucket{le="131.072"} 8
npm_windows_get_endpoint_latency_seconds_bucket{le="+Inf"} 8
npm_windows_get_endpoint_latency_seconds_sum 12.217500000000003
npm_windows_get_endpoint_latency_seconds_count 8

SetPolicy Latency

# HELP npm_windows_setpolicy_latency_seconds Latency in seconds to add/update/delete SetPolicies by operation & is_nested label
# TYPE npm_windows_setpolicy_latency_seconds histogram
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.016"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.032"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.064"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.128"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.256"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="0.512"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="1.024"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="2.048"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="4.096"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="8.192"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="16.384"} 2
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="32.768"} 2
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="65.536"} 2
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="131.072"} 2
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="create",le="+Inf"} 2
npm_windows_setpolicy_latency_seconds_sum{is_nested="false",operation="create"} 27.3388
npm_windows_setpolicy_latency_seconds_count{is_nested="false",operation="create"} 2
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.001"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.002"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.004"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.008"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.016"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.032"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.064"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.128"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.256"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="0.512"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="1.024"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="2.048"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="4.096"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="8.192"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="16.384"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="32.768"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="65.536"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="131.072"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="false",operation="update",le="+Inf"} 1
npm_windows_setpolicy_latency_seconds_sum{is_nested="false",operation="update"} 577.6455
npm_windows_setpolicy_latency_seconds_count{is_nested="false",operation="update"} 1
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.001"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.002"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.004"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.008"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.016"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.032"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.064"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.128"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.256"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="0.512"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="1.024"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="2.048"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="4.096"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="8.192"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="16.384"} 0
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="32.768"} 1
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="65.536"} 1
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="131.072"} 1
npm_windows_setpolicy_latency_seconds_bucket{is_nested="true",operation="create",le="+Inf"} 1
npm_windows_setpolicy_latency_seconds_sum{is_nested="true",operation="create"} 17.3044
npm_windows_setpolicy_latency_seconds_count{is_nested="true",operation="create"} 1

matmerr · 2023-05-11T23:01:07Z

npm/metrics/endpoints.go

+import "github.com/Azure/azure-container-networking/npm/util"
+
+func RecordListEndpointsLatency(timer *Timer) {
+	if util.IsWindowsDP() {


could probably move all these checks to a *_windows.go file but this works too :)

moved all new metrics to *_windows.go files

npm/metrics/setpolicies_windows.go

* implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT

add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981)

add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981)

add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981)

* CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]>

* CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]> (cherry picked from commit 1514d95)

* build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test (#1846) build azure-vnet-telemetry and azure-vnet-ipam in dropgz-test for parity with release image Signed-off-by: Evan Baker <[email protected]> (cherry picked from commit f619259) * ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> (cherry picked from commit 024819d) * CI: [CNI] Replace the bash scripts for CNI load testing with golang test cases (#2003) CI:[CNI] Replace the bash scripts with the golang test cases (cherry picked from commit 008ae45) * ci: [CNI] Move Nightly Cilium Pipeline test to ACN (#1963) * CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: Eliminating duplicate lines * ci: Add update permission for ciliumidentity * fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) fix: Parameterize Image Registry add retry to nnc update during scaledown (#1970) * add retry to nnc update during scaledown Signed-off-by: Evan Baker <[email protected]> * test for panic in pool monitor Signed-off-by: Evan Baker <[email protected]> --------- Signed-off-by: Evan Baker <[email protected]> fix: reserve 0th IP as gateway for overlay on Windows (#1968) * fix: reserve 0th IP as gateway for overlay on Windows * fix: allow gateway to be updated ci: windows profile container image (#1988) Always use 0 for NC version in Overlay (#1979) always use 0 for NC version in overlay Signed-off-by: Evan Baker <[email protected]> [Vnet Scale - CNS]: Flattening CIDR ranges for Node NNC to a list (#1921) * Read secondary CIDRs from VnetScale NNC * fix comment * update comment * For VnetScale mode, Use 1st IP for def gateway instead of 0th for windows * fix/add import * address pr comments * add comments * address pr comments * wrap error * fix typo * fix UT fix: [NPM] check if policy exists in case of nil pointer (#1974) fix: check for nil first ci: disable kube-proxy for test clusters (#1965) * disable kube-proxy for byocni cluster creation * test config mapping * shell pwd * use CURDIR * check current directory * test with repo root dir * test azp format * test azp format * test azp format * change e2e steps to remove kube proxy * fix load test update args * fix ns and rg in update * update ciliume2e * fix kubectl cmd in load test * adding new targets for no kube proxy * remove cluster update * update overlay e2e * test behavior of load test * test grep for azure-cns * look for container deployment * testing * restart node variable check * update if condition * add skip node case --------- Co-authored-by: tamilmani1989 <[email protected]> perf: [WIN-NPM] fast bootup (#1900) * wip * wip2 * use other apply DP func * address comment about if statement * finish bootup for both DPs * fix lint * fix lint 2 * fix lint 3 * longer UT timeout and add missing UTs for apply in background tool: [NPM] script to clean up iptable chains (#1978) tool: script to clean up NPM iptable chains feat: [WIN-NPM] metrics for latencies and failures (#1959) * implement metrics * add npm prefix * rename windows files * metrics pkg UTs * allow reinitializing prometheus metrics * fix: hns wrapper should not throw error for empty SetPolicy values * test: metric UTs in dataplane * fix: record list endpoint latency always * remove flaky UT * feat: metric for max ipset members * fix lint * fix lint 2 * fix build * fix lint 3 * simplify conditionals and protect against maxMembers becoming negative * remove bottom 4 histogram buckets. start at 16 ms * reset metrics for ipset UTs * style: don't check for windows dp in *_windows.go files * build: remove unused import * test: reset windows metrics in UT Remove SSH port 22 rule from aks-engine clusters (#1983) ci: change overlaye2e stage to cilium-overlay (#1997) * renaming overlaye2e for cilium * update display names for stages Initial getHomeAZ 404 changes (#1994) * initial getHomeAZ 404 changes * treat 404 as success * address comments CNS to be able to generate dualstack overaly CNI conflist (#1981) * fix: File Directory * style: Comments * Addressing Comments --------- Co-authored-by: Paul Johnston <[email protected]> (cherry picked from commit 1514d95) * ci:[CNI] Add windows CNIv1 datapath test (#2016) * ci: Transfer files * test: Working Datapath Test * test: apierror Tests * style: Datapath Package * test: Deployment timing * fix: Error check * fix: Lint (cherry picked from commit 390977d) * fix: [CNI] CNI load test failing due to namespace already created (#2031) fix: CNI load test failing due to namespace already created (cherry picked from commit c10900e) * ci:[CNI] Windows cniv1 load test pipeline (#2024) CI:[CNI] Windows cniv1 load test pipeline (cherry picked from commit e45ad21) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test (#2052) * ci: [CNI] Adding aks cluster creation steps for k8s e2e test * Add validate step to the pipeline * Adding the telemetry config to the cluster (cherry picked from commit 846e508) * ci:[CNI] Replace AKS-Engine Tests with k8s conformance tests (#2062) * Initial Commit * Add attempts to prevent flakyness * Add taint for windows tests * Add k8s e2e tests * Testing vmSizes * Artifact k8se2e binary * Remove NPM E2E * Add testing and increase processes * Addressing comments (cherry picked from commit 451c691) * CI: Removing AKS engine related code (#2089) (cherry picked from commit b45c2c7) * feat: [dropgz] Dropgz for windows (#2075) * feat: [dropgz] Dropgz for windows * Removing the code for killing the process from dropgz for windows (cherry picked from commit 7a41178) * ci: Update dns tests for k8s conformance (#2104) Update dns tests for k8s v1.26 (cherry picked from commit bbf2fd4) * ci: adding cni package as a trigger (#2108) (cherry picked from commit e6a8ea6) * ci: add packages for submodule trigger (#2154) (cherry picked from commit 4aecfd6) * set mellanox reg key (#1768) (cherry picked from commit fa2de6d) --------- Co-authored-by: Evan Baker <[email protected]> Co-authored-by: Camryn Lee <[email protected]> Co-authored-by: Vipul Singh <[email protected]> Co-authored-by: Rajvi <[email protected]>

implement metrics

b17d136

huntergregory added npm Related to NPM. windows labels May 11, 2023

huntergregory requested a review from vakalapa May 11, 2023 18:52

add npm prefix

3813829

huntergregory marked this pull request as ready for review May 11, 2023 22:28

huntergregory requested a review from a team as a code owner May 11, 2023 22:28

matmerr previously approved these changes May 11, 2023

View reviewed changes

rename windows files

9e5042a

huntergregory dismissed matmerr’s stale review via 9e5042a May 12, 2023 17:01

huntergregory added 5 commits May 12, 2023 13:35

metrics pkg UTs

3e9c732

allow reinitializing prometheus metrics

7994b0b

fix: hns wrapper should not throw error for empty SetPolicy values

90a5907

test: metric UTs in dataplane

dfb6a4a

fix: record list endpoint latency always

01b6101

huntergregory requested a review from a team as a code owner May 12, 2023 22:27

huntergregory added 2 commits May 15, 2023 10:51

remove flaky UT

48ed7d6

feat: metric for max ipset members

68274c3

huntergregory force-pushed the hgregory/05-09-metrics branch from f90e0a8 to 68274c3 Compare May 15, 2023 17:54

huntergregory and others added 8 commits May 15, 2023 11:07

fix lint

962619b

fix lint 2

a8c0756

fix build

86d4133

fix lint 3

c65fac1

simplify conditionals and protect against maxMembers becoming negative

01ddf46

remove bottom 4 histogram buckets. start at 16 ms

4c9e014

reset metrics for ipset UTs

963c913

Merge branch 'master' into hgregory/05-09-metrics

952df48

vakalapa previously approved these changes Jun 2, 2023

View reviewed changes

vakalapa reviewed Jun 2, 2023

View reviewed changes

npm/metrics/setpolicies_windows.go Outdated Show resolved Hide resolved

Merge branch 'master' into hgregory/05-09-metrics

239c177

huntergregory dismissed vakalapa’s stale review via 25a629d June 2, 2023 22:41

huntergregory added 3 commits June 2, 2023 15:41

style: don't check for windows dp in *_windows.go files

25a629d

build: remove unused import

c9d4d6e

test: reset windows metrics in UT

f2e218c

vakalapa approved these changes Jun 5, 2023

View reviewed changes

vakalapa merged commit 04f9285 into master Jun 5, 2023

vakalapa deleted the hgregory/05-09-metrics branch June 5, 2023 19:43

huntergregory mentioned this pull request Jun 16, 2023

fix: [WIN-NPM] fix units of new latency metrics #2018

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: [WIN-NPM] metrics for latencies and failures #1959

feat: [WIN-NPM] metrics for latencies and failures #1959

huntergregory commented May 11, 2023 •

edited

Loading

matmerr May 11, 2023

huntergregory May 12, 2023

feat: [WIN-NPM] metrics for latencies and failures #1959

feat: [WIN-NPM] metrics for latencies and failures #1959

Conversation

huntergregory commented May 11, 2023 • edited Loading

Metrics

Examples

Max IPSet Members

Get Endpoint Latency & Failures

SetPolicy Latency

matmerr May 11, 2023

Choose a reason for hiding this comment

huntergregory May 12, 2023

Choose a reason for hiding this comment

huntergregory commented May 11, 2023 •

edited

Loading