NETOBSERV-333: GRPC ingest metrics

[mariomac]

Implements the following metrics that are already implemented by the IPFIX ingestor:

flow_decoder_count
flow_process_nf_delay_summary_seconds
flow_process_nf_delay_summary_seconds_sum
flow_process_nf_delay_summary_seconds_count
flow_traffic_summary_size_bytes
flow_traffic_summary_size_bytes_sum
flow_traffic_summary_size_bytes_count
ingest_collector_flow_logs_processed
flow_process_nf_errors_count

It does not implement the following metrics:

• flow_traffic_bytes, because it's equivalent to existing flow_traffic_summary_size_bytes_sum.
• flow_process_nf_count, because it's equivalent to existing ingest_collector_flow_logs_processed.
• flow_process_nf_flowset_sum and flow_process_nf_flowset_records_sum because the GRPC+Protobuf service does not have different message types as IPFIX (DataFlowSet, OptionsDataFlowSet, TemplateFlowSet, etc...). All the values here would be equivalent to ingest_collector_flow_logs_processed.
• flow_process_nf_templates_count because GRPC+Protobuf does not use templates.
• flow_traffic_packets_counter because GRPC does not work at packet level and there is no affordable way to inspect GRPC communication at IP/TCP level.
• flow_summary_decoding_time because GRPC+Protobuf does not provide an affordable way to inspect the raw-->object decoding time.
• ingest_collector_queue_length value will be always 0, since GRPC ingestor does not implement any internal queue to batch/forward the flow records.

Also, the flow_decoder_count might have very different values as for the IPFIX scenario. This metric specifies how many messages have been decoded. In IPFIX, its value would be the same as ingest_collector_flow_logs_processed, as Goflow2 decodes each flow record individually. In GRPC+Protobuf (eBPF agent), this value would be much lower because flow records are sent in big batches.

[memodi]

@mariomac - is it possible to include flow_process_nf_errors_count metric as well?

[mariomac]

@memodi adding it

[codecov-commenter]

Codecov Report

Merging #223 (aaeae92) into main (debee4d) will increase coverage by 0.31%.
The diff coverage is 82.00%.

@@            Coverage Diff             @@
##             main     #223      +/-   ##
==========================================
+ Coverage   60.96%   61.28%   +0.31%     
==========================================
  Files          66       66              
  Lines        3784     3833      +49     
==========================================
+ Hits         2307     2349      +42     
- Misses       1336     1339       +3     
- Partials      141      145       +4     

Flag	Coverage Δ
unittests	61.28% <82.00%> (+0.31%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/pipeline/decode/decode_protobuf.go	74.54% <ø> (ø)
pkg/pipeline/ingest/ingest_grpc.go	77.10% <82.00%> (+6.52%)	⬆️
pkg/pipeline/ingest/ingest_collector.go	53.96% <0.00%> (+1.58%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update debee4d...aaeae92. Read the comment docs.

[ronensc]

Isn't there an issue with cardinality when adding IPs to the labels?

[mariomac]

This does not refer to the IPs in the flow but the IP of the eBPF agents. The number of remote IPs should be small (one agent per node).

I added this field to replicate the behavior of the equivalent metric in the IPFIX ingestor.

[eranra]

Maybe move protobuf1.0 to const ???

[eranra]

maybe move protobuf to const ... or change to GRPC instead

[eranra]

Minor comments. LGTM