NETOBSERV-309 skip TLS checks & add TenantID #120

jpinsonneau · 2022-06-14T14:46:05Z

Following conversation with @memodi, this PR force skip TLS checks on both FLP & console plugin until implementation and allow configuration of Loki tenant ID

memodi · 2022-06-14T14:47:45Z

/ok-to-test

github-actions · 2022-06-14T14:50:35Z

New image: ["quay.io/netobserv/network-observability-operator:6536b7c"]. It will expire after two weeks.

jotak · 2022-06-14T15:27:15Z

controllers/consoleplugin/consoleplugin_objects.go

@@ -103,6 +103,9 @@ func buildArgs(desired *flowsv1alpha1.FlowCollectorConsolePlugin, desiredLoki *f
 		"-key", "/var/serving-cert/tls.key",
 		"-loki", querierURL(desiredLoki),
 		"-loki-labels", strings.Join(constants.LokiIndexFields, ","),
+		"-loki-tenant-id", desiredLoki.TenantID,
+		//TODO: add loki tls config https://issues.redhat.com/browse/NETOBSERV-309
+		"-loki-skip-tls", "true",


this could also be promoted right now as a CR config, like you did for TenantID.

I can do that if you prefer but the goal is to remove this after TLS implementation

I don't think it has to be removed, it happens commonly that people want to just quick try software without securing everything, so this is still a good option to have IMO, especially as people can "bring their own Loki". What matters more is that it defaults to false.

memodi · 2022-06-14T15:51:20Z

New image: ["quay.io/netobserv/network-observability-operator:6536b7c"]. It will expire after two weeks.

@jpinsonneau - after deploying NOO off this PR and above image, I am no longer seeing cert error on console side, however for FLP post request I am seeing below error:
time=2022-06-14T20:12:01Z level=info component=client error=Post "http://lokistack-sample-distributor-http.network-observability:3100/loki/api/v1/push": proxyconnect tcp: dial tcp :0: connect: connection refused fields.level=warn fields.msg=error sending batch, will retry host=lokistack-sample-distributor-http.network-observability:3100 module=export/loki status=-1

when I do push manually it succeeds:
$ oc exec -it flowlogs-pipeline-55flk -- curl -w "%{http_code}\\n" -k -XPOST https://lokistack-sample-distributor-http.network-observability:3100/loki/api/v1/push --data-raw \ '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1655238630742828000", "fizzbuzz" ] ] }]}' -H "X-Scope-OrgID: netobserv" -H "Content-Type: application/json" 204

controllers/flowlogspipeline/flp_objects.go

memodi · 2022-06-14T21:01:45Z

Following up on my comment above: #120 (comment):

I am getting same error when I try to work with zero-click loki server:
time=2022-06-14T20:51:16Z level=info component=client error=Post "http://loki:3100/loki/api/v1/push": proxyconnect tcp: dial tcp :0: connect: connection refused fields.level=warn fields.msg=error sending batch, will retry host=loki:3100 module=export/loki status=-1

not sure if there needs to be some updates to be made to FLP to be able to work with NOO created off this PR

jpinsonneau · 2022-06-15T11:16:46Z

@jpinsonneau - after deploying NOO off this PR and above image, I am no longer seeing cert error on console side, however for FLP post request I am seeing below error: time=2022-06-14T20:12:01Z level=info component=client error=Post "http://lokistack-sample-distributor-http.network-observability:3100/loki/api/v1/push": proxyconnect tcp: dial tcp :0: connect: connection refused fields.level=warn fields.msg=error sending batch, will retry host=lokistack-sample-distributor-http.network-observability:3100 module=export/loki status=-1

Hm sorry about that ! It seems HTTPClientConfig ProxyURL is used even if it's not set on our side. Since the proxy is not resolved, the query fails.
I have tried to set nil or "" but it doesn't work as is.

We didn't had this in the PoC because we used a map[string]interface{}

I'm still looking for a solution

jotak · 2022-06-15T11:57:57Z

@jpinsonneau Could it be related to @OlivierCazade 's netobserv/flowlogs-pipeline#229 ? (missing omitempty in FLP)

jpinsonneau · 2022-06-15T12:01:35Z

@jpinsonneau Could it be related to @OlivierCazade 's netobserv/flowlogs-pipeline#229 ? (missing omitempty in FLP)

No it actually comes from prometheus http_config file

jpinsonneau · 2022-06-15T12:47:58Z

/retest

memodi · 2022-06-16T15:07:35Z

/ok-to-test

github-actions · 2022-06-16T15:09:59Z

New image: ["quay.io/netobserv/network-observability-operator:ba70519"]. It will expire after two weeks.

memodi · 2022-06-16T21:57:20Z

@jpinsonneau - I don't know if this PR needs rebase after this PR was merged, since I am running into below error on FLP side:

time=2022-06-16T21:41:14Z level=debug msg=findStageType: stage = decode
time=2022-06-16T21:41:14Z level=fatal msg=failed to initialize pipeline invalid stage type: unknown

jpinsonneau · 2022-06-17T07:56:09Z

@jpinsonneau - I don't know if this PR needs rebase after this PR was merged, since I am running into below error on FLP side

Yeah seems to ! Let's rebase

jpinsonneau · 2022-06-17T14:05:53Z

/ok-to-test

jpinsonneau · 2022-06-17T14:06:53Z

@memodi I have tested today with latest flowlogs-pipeline version and it works fine 👍

github-actions · 2022-06-17T14:08:31Z

New image: ["quay.io/netobserv/network-observability-operator:474f67e"]. It will expire after two weeks.

memodi · 2022-06-17T18:10:16Z

@memodi I have tested today with latest flowlogs-pipeline version and it works fine 👍

Yes, it works for me too. Thank you. Although I couldn't see any flows on Netflow Table and always ends up in 503 error after trying to load :D

I had extra small LokiStack, not sure if Loki from Loki-operator has poor performance than what we install with zero-click Loki config,

jpinsonneau · 2022-06-20T07:47:03Z

Performances are better on my side with LokiStack @memodi
I use gp2 AWS s3 bucket witth extra small config. Do you have a running cluster so I can check what happens ?

memodi · 2022-07-08T15:28:11Z

/ok-to-test

memodi · 2022-07-08T15:38:58Z

/retest

memodi · 2022-07-08T15:52:46Z

/ok-to-test

jotak · 2022-07-13T09:05:43Z

/lgtm
cc @OlivierCazade once this is merged we'll need to rebase and fix conflicts on the TLS branch work

vendor/github.com/prometheus/common/config/http_config.go

jpinsonneau · 2022-07-21T11:24:53Z

/retest

jotak · 2022-07-22T06:49:55Z

/lgtm

jpinsonneau · 2022-07-22T08:04:48Z

/approve

openshift-ci · 2022-07-22T08:04:54Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpinsonneau

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jpinsonneau]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Integrate components from Goflow-Kube

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 14, 2022

jotak reviewed Jun 14, 2022

View reviewed changes

memodi reviewed Jun 14, 2022

View reviewed changes

controllers/flowlogspipeline/flp_objects.go Show resolved Hide resolved

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 15, 2022

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 16, 2022

jpinsonneau added 2 commits June 17, 2022 09:56

skip tls checks & add tenant id

03c8718

temporary fix

01f35e8

jpinsonneau force-pushed the skip_tls branch from d035423 to 01f35e8 Compare June 17, 2022 07:56

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 17, 2022

openshift-ci bot added the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jun 17, 2022

openshift-ci bot assigned jotak Jul 13, 2022

openshift-ci bot added the lgtm label Jul 13, 2022

OlivierCazade reviewed Jul 13, 2022

View reviewed changes

vendor/github.com/prometheus/common/config/http_config.go Outdated Show resolved Hide resolved

jotak removed the lgtm label Jul 13, 2022

This was referenced Jul 20, 2022

NETOBSERV-101 R&D: Kube enricher write path for downstream operator #135

Merged

NETOBSERV-101 R&D: Kube enricher write path for downstream operator netobserv/documents#22

Merged

added HTTPClientConfig test check

476519e

github-actions bot removed the ok-to-test To set manually when a PR is safe to test. Triggers image build on PR. label Jul 20, 2022

forked prometheus/common

46ad22c

jpinsonneau force-pushed the skip_tls branch from 6118cdc to 46ad22c Compare July 21, 2022 11:33

jpinsonneau requested review from jotak and OlivierCazade July 21, 2022 14:52

openshift-ci bot added the lgtm label Jul 22, 2022

OlivierCazade approved these changes Jul 22, 2022

View reviewed changes

openshift-ci bot assigned OlivierCazade Jul 22, 2022

openshift-ci bot added the approved label Jul 22, 2022

jpinsonneau merged commit e33031a into netobserv:main Jul 22, 2022

KalmanMeth pushed a commit to KalmanMeth/network-observability-operator that referenced this pull request Feb 13, 2023

Merge pull request netobserv#120 from mariomac/netobserv-flows

e7919b2

Integrate components from Goflow-Kube

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

NETOBSERV-309 skip TLS checks & add TenantID #120

NETOBSERV-309 skip TLS checks & add TenantID #120

jpinsonneau commented Jun 14, 2022 •

edited

Loading

memodi commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

jotak Jun 14, 2022

jpinsonneau Jun 14, 2022

jotak Jun 14, 2022

memodi commented Jun 14, 2022 •

edited

Loading

memodi commented Jun 14, 2022

jpinsonneau commented Jun 15, 2022

jotak commented Jun 15, 2022

jpinsonneau commented Jun 15, 2022

jpinsonneau commented Jun 15, 2022

memodi commented Jun 16, 2022

github-actions bot commented Jun 16, 2022

memodi commented Jun 16, 2022

jpinsonneau commented Jun 17, 2022

jpinsonneau commented Jun 17, 2022

jpinsonneau commented Jun 17, 2022

github-actions bot commented Jun 17, 2022

memodi commented Jun 17, 2022

jpinsonneau commented Jun 20, 2022

memodi commented Jul 8, 2022

memodi commented Jul 8, 2022

memodi commented Jul 8, 2022

jotak commented Jul 13, 2022

jpinsonneau commented Jul 21, 2022

jotak commented Jul 22, 2022

jpinsonneau commented Jul 22, 2022

openshift-ci bot commented Jul 22, 2022

NETOBSERV-309 skip TLS checks & add TenantID #120

NETOBSERV-309 skip TLS checks & add TenantID #120

Conversation

jpinsonneau commented Jun 14, 2022 • edited Loading

memodi commented Jun 14, 2022

github-actions bot commented Jun 14, 2022

jotak Jun 14, 2022

Choose a reason for hiding this comment

jpinsonneau Jun 14, 2022

Choose a reason for hiding this comment

jotak Jun 14, 2022

Choose a reason for hiding this comment

memodi commented Jun 14, 2022 • edited Loading

memodi commented Jun 14, 2022

jpinsonneau commented Jun 15, 2022

jotak commented Jun 15, 2022

jpinsonneau commented Jun 15, 2022

jpinsonneau commented Jun 15, 2022

memodi commented Jun 16, 2022

github-actions bot commented Jun 16, 2022

memodi commented Jun 16, 2022

jpinsonneau commented Jun 17, 2022

jpinsonneau commented Jun 17, 2022

jpinsonneau commented Jun 17, 2022

github-actions bot commented Jun 17, 2022

memodi commented Jun 17, 2022

jpinsonneau commented Jun 20, 2022

memodi commented Jul 8, 2022

memodi commented Jul 8, 2022

memodi commented Jul 8, 2022

jotak commented Jul 13, 2022

jpinsonneau commented Jul 21, 2022

jotak commented Jul 22, 2022

jpinsonneau commented Jul 22, 2022

openshift-ci bot commented Jul 22, 2022

jpinsonneau commented Jun 14, 2022 •

edited

Loading

memodi commented Jun 14, 2022 •

edited

Loading