Add histogram metrics for infoblox calls

[jkremser]

Fix issue #713

Introducing a new metric called k8gb_infoblox_request_duration_bucket of type histogram with labels:

• request (name of the method that was called on infoblox client ~ those CRUD methods for txt record and zone delegation call)
• success (true/false) did the call ended up ok?

It's a histogram, so it splits the space into buckets (in our case it's exponential scale - [0 -.2], [.2 -.8] , [.8 - 3.2], [3.2 - 12.8], [12.8 - 51.2], [51.2- infinity]) and the value of the metric denotes the number of hits that fell into that buckets, i.e. how many calls took that many seconds.

• there are two special records w/ _sum and _count prefix to be able to calculate mean and what not

Note: I did not use the common labels we have on gauge metrics (gslb name and namespace), because I don't think they are useful in this context and the "combinatoric explosion" w/ all the possible vector(/label) values would be too huge.

ibclient was renamed to ibcl because the line were too long and linter was not happy about it

I did some manual testing and it looks like this:

bash-5.1# curl 10.42.0.7:8080/metrics | grep infoblo
# HELP k8gb_infoblox_request_duration How long it took for Infoblox requests to complete, partitioned by request type. Round-trip time of http communication is included.
# TYPE k8gb_infoblox_request_duration histogram
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="0.2"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="0.8"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="3.2"} 1
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="12.8"} 1
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="51.2"} 2
k8gb_infoblox_request_duration_bucket{request="TXTRecordCreate",success="false",le="+Inf"} 3
k8gb_infoblox_request_duration_sum{request="TXTRecordCreate",success="false"} 221.0000727
k8gb_infoblox_request_duration_count{request="TXTRecordCreate",success="false"} 3
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="0.2"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="0.8"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="3.2"} 1
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="12.8"} 1
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="51.2"} 2
k8gb_infoblox_request_duration_bucket{request="TXTRecordDelete",success="false",le="+Inf"} 3
k8gb_infoblox_request_duration_sum{request="TXTRecordDelete",success="false"} 221.00009020000002
k8gb_infoblox_request_duration_count{request="TXTRecordDelete",success="false"} 3
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="0.2"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="0.8"} 1
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="3.2"} 6
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="12.8"} 6
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="51.2"} 10
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="false",le="+Inf"} 13
k8gb_infoblox_request_duration_sum{request="TXTRecordUpdate",success="false"} 718.3002365
k8gb_infoblox_request_duration_count{request="TXTRecordUpdate",success="false"} 13
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="0.2"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="0.8"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="3.2"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="12.8"} 0
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="51.2"} 2
k8gb_infoblox_request_duration_bucket{request="TXTRecordUpdate",success="true",le="+Inf"} 3
k8gb_infoblox_request_duration_sum{request="TXTRecordUpdate",success="true"} 300.0000806
k8gb_infoblox_request_duration_count{request="TXTRecordUpdate",success="true"} 3

Signed-off-by: Jirka Kremser [email protected]

[ytsarev]

Is this change related to ibclient version?

[jkremser]

nope, there is a new private function w/ almost the same signature that makes the actual call (objMgr.GetZoneDelegated) but this new function surrounds the call w/ time measurement so that we measure the request times in the histogram.

[k0da]

LGTM

[ytsarev]

LGTM assuming it was e2e tested( and according to description it was )