NETOBSERV-1458: reduce loki streams, index FlowLayer

[jotak]

No description provided.

[openshift-ci-robot]

@jotak: This pull request references NETOBSERV-1458 which is a valid jira issue.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[jotak]

cc @msherif1234 @nathan-weinberg : I would like to compare perf with this set of loki index, against the previous runs and also against this other PR #552

[github-actions]

New images:

• quay.io/netobserv/network-observability-operator:405439d
• quay.io/netobserv/network-observability-operator-bundle:v0.0.0-405439d
• quay.io/netobserv/network-observability-operator-catalog:v0.0.0-405439d

They will expire after two weeks.

To deploy this build:

# Direct deployment, from operator repo
IMAGE=quay.io/netobserv/network-observability-operator:405439d make deploy

# Or using operator-sdk
operator-sdk run bundle quay.io/netobserv/network-observability-operator-bundle:v0.0.0-405439d

Or as a Catalog Source:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: netobserv-dev
  namespace: openshift-marketplace
spec:
  sourceType: grpc
  image: quay.io/netobserv/network-observability-operator-catalog:v0.0.0-405439d
  displayName: NetObserv development catalog
  publisher: Me
  updateStrategy:
    registryPoll:
      interval: 1m

[jotak]

Some frontend perf test --
I've been following the same steps that I ran previously and described here: #409 (comment)

---------------------
1st RUN: main
---------------------
streams: ~2K
---------------------
min: 171ms
max: 393ms
avg: ~210ms
=====================
---------------------
2nd RUN: 88ec5d6
Remove some labels
---------------------
streams: ~1.2K
---------------------
min: 1321ms
max: 2784ms
avg: ~1800ms
=====================
---------------------
3rd RUN: 405439d
New indexes
---------------------
streams: ~1.3K
---------------------
min: 202ms
max: 399ms
avg: ~210ms
=====================
---------------------
4th RUN: local build
Only removing Duplicate
---------------------
streams: ~1.3K
---------------------
min: 253ms
max: 493ms
avg: ~300ms
=====================

To summarize: it's clear that the labels introduced in 1.5 caused an increased number of streams (RUN 1). We can remove some labels to reduce that number:

• Removing both Duplicate and Src/DstType (RUN 2), like it was in 1.4, degrades too much the query performance (x10 slower)
• Removing only Duplicate (RUN 4) makes us retrieve almost the same number of streams than previously, while keeping pretty good query performance
• Removing Duplicate and Adding FlowLayer (RUN 3 / this PR) is comparable to RUN 4 for streams but with a slightly better query performance

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (74bb07b) 57.77% compared to head (b2ff2fe) 57.80%.
Report is 1 commits behind head on release-1.5.

Additional details and impacted files

@@               Coverage Diff               @@
##           release-1.5     #554      +/-   ##
===============================================
+ Coverage        57.77%   57.80%   +0.03%     
===============================================
  Files               72       72              
  Lines             9541     9541              
===============================================
+ Hits              5512     5515       +3     
+ Misses            3690     3688       -2     
+ Partials           339      338       -1     

Flag	Coverage Δ
unittests	57.80% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nathan-weinberg]

@jotak ack will test this! see table in NETOBSERV-1470 for comparisons

[jpinsonneau]

Sounds good to me; let's wait @nathan-weinberg feedback on the perfs before merging

[nathan-weinberg]

/label qe-approved

[openshift-ci-robot]

@jotak: This pull request references NETOBSERV-1458 which is a valid jira issue.

In response to this:

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[jotak]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jotak

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [jotak]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment