[Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs #3622

crespocarlos · 2022-06-29T12:46:51Z

What does this PR do?

This PR is the initial work for creating the new Platform Observability package. We will start with ingesting Kibana logs but its capabilities will be extended over time. This package is heavily based on the solution provided here #2972

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

Define the package name
Align current Kibana package ingest pipelines with this one (This is a duplication of the existing Kibana package)
Decide whether we will go with Filebeat processors to parse the json object or not

How to test this PR locally

In order to tests this we first need to edit the configuration used by elastic-packages

Enable kibana logging, by adding the configuration below to ~/.elastic-package/profiles/default/stack/kibana.config.8x.yml

xpack.security.audit.enabled: true
xpack.security.audit.appender:
  type: rolling-file
  fileName: ./.kbn-logs/audit.log
  policy:
    type: time-interval
    interval: 24h
  strategy:
    type: numeric
    max: 10
  layout:
    type: json

logging:
  appenders:
    file:
      type: file
      fileName: ./.kbn-logs/kibana.log
      layout:
        type: json
  root:
    appenders: [default, file]
    level: all

  loggers:
    - name: server
      level: fatal
    - name: telemetry
      appenders: [file]
      level: all
    - name: metrics.ops
      appenders: [file]
      level: debug

Edit ~/.elastic-package/profiles/default/stack/snapshot.yml, adding "../../../tmp/service_logs:/usr/share/kibana/.kbn-logs" to the list of volumes for kibana container, to enable the agent to see the logs
From the repo root folder go to packages/platform_observability folder and run elastic-package build
Run elastic-package stack up -v -d
On Kibana, go to Integrations
Install Platform Observability with default configuration, adding it to Elastic-Agent host

Related issues

Closes #2226

Screenshots

Integration page

Kibana logs

Kibana audit logs

...form-observability/data_stream/audit/elasticsearch/ingest_pipeline/kibana-audit-logs-ecs.yml

packages/platform-observability/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

packages/platform-observability/data_stream/log/fields/ecs.yml

elasticmachine · 2022-06-29T12:52:02Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-07-22T07:06:36.514+0000
Duration: 15 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	8
Skipped	0
Total	8

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

matschaffer · 2022-06-30T03:29:56Z

The first time I made the changes I hit this which reverted them:

❯ elastic-package version
2022/06/30 12:26:04  WARN Default profile has been changed by user or updated by elastic-package. The current profile will be moved.
elastic-package has been installed.
elastic-package version-hash 3144604 (build time: 2022-06-30T12:23:43+09:00)

Maybe due to me having run elastic-package on an older version. Thought I'd mention it incase other testers bumped into the same thing.

matschaffer · 2022-06-30T05:16:16Z

heh, so far my attempts at starting the stack are failing. Will have to dig:

Creating elastic-package-stack_kibana_1                    ... done

ERROR: for fleet-server  Container "a6027ac4a639" is unhealthy.

ERROR: for kibana_is_ready  Container "a6027ac4a639" is unhealthy.
ERROR: Encountered errors while bringing up the project.
Error: booting up the stack failed: running docker-compose failed: running command failed: running Docker Compose up command failed: exit status 1

matschaffer · 2022-06-30T05:36:10Z

Interestingly enough, the kibana healthcheck checks for the ES security index. which is red. Go figure 😆

https://github.com/elastic/elastic-package/blob/3b2d1e91503707f6be7685c9478a2107bd2a2ed7/internal/install/_static/kibana_healthcheck.sh#L6

matschaffer · 2022-06-30T05:57:39Z

ah!

❯ docker logs elastic-package-stack_elasticsearch_1 -n 1
{"@timestamp":"2022-06-30T05:55:51.586Z", "log.level": "WARN", "message":"high disk watermark [90%] exceeded on [Vo4wyn23ReWufzMT2R9n3g][5b4a83809124][/usr/share/elasticsearch/data] free: 5.6gb[9.6%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[5b4a83809124][masterService#updateTask][T#1]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor","elasticsearch.cluster.uuid":"wzlK4-GBTymeP6n_Izf6ZA","elasticsearch.node.id":"Vo4wyn23ReWufzMT2R9n3g","elasticsearch.node.name":"5b4a83809124","elasticsearch.cluster.name":"elasticsearch"}

matschaffer

Success! I also had to do this stuff to get your package to show up https://github.com/elastic/integrations/blob/main/docs/developer_workflow_design_build_test_integration.md#build

The elastic-package stack was surprisingly hard to setup. A full docker disk failed in a way that looked like kibana was wrong, I had to import & trust the self-signed SSL cert. Either I'm holding it wrong or we have some work to do there. :)

Oh, and not sure where the kibana logo SVG is from. It's a little different than https://brand.elastic.co/302f66895/p/41ff52-solution-logos/b/14d954/i/11013297 - maybe good to re-import?

packages/platform-observability/data_stream/audit/fields/ecs.yml

matschaffer · 2022-06-30T06:39:28Z

packages/platform-observability/data_stream/log/manifest.yml

+        required: true
+        show_user: true
+        default:
+          - /tmp/service_logs/kibana.log


I see a /usr/share/kibana/logs in the container. Wondering if we should default to that path and update the elastic-package stack bindmounts to go there.

Yeah. Should we also enable Kibana logging by default?

Definitely :)

matschaffer · 2022-06-30T06:43:03Z

packages/platform-observability/data_stream/audit/agent/stream/log.yml.hbs

+  - {{this}}
+{{/each}}
+{{#if processors}}
+{{processors}}  


I'm guessing we can add "filebeat" processors (like https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html) to the manifest and have them show up here. Or maybe just put them in this file directly.

Not sure what's in-fashion for packages today regarding reader-vs-ES side processing but personally I like to push as many simple transformations as I can to the collection point. Maybe @mtojek can advise.

I did a quick test and it seems to work. I'll explore it a bit more. It would simplify the pipeline config

It works.

this is all I have for the pipeline:

[ { "remove": { "field": "data_stream.dataset", "ignore_missing": true } }, { "remove": { "field": "event.dataset", "ignore_missing": true } }, { "set": { "description": "Uses event.dataset as a default for data_stream.dataset if the latter is not set.", "field": "data_stream.dataset", "copy_from": "event.dataset", "if": "ctx.event?.dataset instanceof String && ctx.event.dataset.length() > 1", "override": false } }, { "script": { "source": "ctx.data_stream.dataset = /[\\/*?\"<>|, #:-]/.matcher(ctx.data_stream.dataset).replaceAll('_')\n", "if": "ctx.data_stream?.dataset != null" } }, { "script": { "source": "ctx.data_stream.namespace = /[\\/*?\"<>|, #:]/.matcher(ctx.data_stream.namespace).replaceAll('_')\n", "if": "ctx.data_stream?.namespace != null" } }, { "set": { "field": "data_stream.type", "value": "logs" } }, { "set": { "field": "data_stream.dataset", "value": "kibana-logs", "override": false } }, { "set": { "field": "data_stream.namespace", "value": "platform-observability", "override": false } }, { "set": { "field": "event.dataset", "copy_from": "data_stream.dataset" } }, { "rename": { "field": "message", "target_field": "event.original", "ignore_missing": true } } ]

A downside of this approach is that the tests are not able to use filebeat processor to parse the logs

Another surprise here :) Hoping we can have some integration experts weigh-in. Are we maybe not supposed to use processors anymore?

I used to follow the rule of thumb. Are you collecting large observability data and is the network loaded? If so, limit those with processors.

We had a slack chat with @crespocarlos, and decided to go with the ingest pipeline due to code simplicity and convenient use of pipeline tests.

Are you collecting large observability data and is the network loaded?

Interesting. Does the end user get to set processors? I'd expect network load to be a factor of deployment size.

crespocarlos · 2022-07-04T13:17:28Z

I have changed and polished the PoC as I started to understand a bit better how to create integration packages.

Solutions

For kibana/audit logs, the pipeline is configured to parse the json message using filebeat processors. With this approach, I noticed that the tests are not able to properly parse the json using the filebeat processor, but kibana is.

For kibana/logs, I kept the configuration from the previous iteration, using a similar approach as here #2972

The question is: which approach is the best?

Field files

Field files can impact in some ways. If we follow the approach for kibana/logs all fields present in the test scenario files must be documented in order for the tests to pass. New fields produced by the integration must be documented.

During my tests, I noticed that ECS fields parsed from a json object don't necessarily need to be documented in the package and only what's documented in base-fields.yml must be present at all times in order for the pipeline to work

...-observability/data_stream/audit/_dev/test/pipeline/test-kibana-audit-logs.log-expected.json

elasticmachine · 2022-07-06T10:43:37Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`2/2`)	💚
Files	100.0% (`4/4`)	💚 2.941
Classes	100.0% (`4/4`)	💚 2.941
Methods	80.0% (`16/20`)	👎 -9.251
Lines	88.889% (`96/108`)	👎 -1.694
Conditionals	100.0% (`0/0`)	💚

klacabane · 2022-07-06T11:05:52Z

I've started a doc when playing with Integrations and updated it with some findings from this document https://github.com/elastic/observability-dev/pull/2253. Feel free to add anything you feel relevant :)

klacabane · 2022-07-06T15:55:28Z

Is this implementation a drop-in replacement for the one we have in the filebeat module ? If we go the single-package route for SM and PO integration, it'll be interesting to align both implementations. Also for a single-package we'll have to carry over all the ingest pipelines for the legacy pipelines as well, unless we decide to only support starting from v8

crespocarlos · 2022-07-07T08:18:39Z

Good point @klacabane. I focused more on ingesting the ECS log as-is, but there are fields in the existing Kibana package that haven't been ported to this new package and fields from Kibana logs that I didn't remove in this new pipeline. So I'll probably have to align at least the missing fields and/or make sure the new pipeline generates the same expected output.

Since Kibana generates ECS formatted logs starting from V8 (correct me if I'm wrong) I think we'll probably support this new PO package starting from V8

@matschaffer, what's your opinion here?

matschaffer · 2022-07-07T08:44:05Z

Since Kibana generates ECS formatted logs starting from V8 (correct me if I'm wrong) I think we'll probably support this new PO package starting from V8

I wasn't thinking we'd try to support anything prior to 8 with this package for now. That will get handled by existing modules.

.github/CODEOWNERS

packages/platform_observability/manifest.yml

packages/platform_observability/_dev/build/build.yml

packages/platform_observability/changelog.yml

packages/platform_observability/data_stream/kibana_log/manifest.yml

packages/platform_observability/data_stream/kibana_audit/manifest.yml

…n version

mtojek

👍

I didn't review data stream ingestion as I don't have domain-specific knowledge. I will leave it for your team

…ields

matschaffer

I had a couple of final-polish questions. But overall this is looking pretty good.

I guess once we add ES we'll need to rethink the package logo (maybe just use the elastic "cluster" logo), but at least it's an up-to-date kibana logo for now. Thanks for that :)

packages/platform_observability/_dev/build/docs/README.md

...es/platform_observability/data_stream/kibana_audit/_dev/test/pipeline/test-common-config.yml

...es/platform_observability/data_stream/kibana_audit/elasticsearch/ingest_pipeline/default.yml

...ages/platform_observability/data_stream/kibana_log/elasticsearch/ingest_pipeline/default.yml

packages/platform_observability/data_stream/kibana_log/fields/ecs.yml

matschaffer · 2022-07-21T06:16:30Z

...ability/data_stream/kibana_audit/_dev/test/pipeline/test-kibana-audit-logs.log-expected.json

+                }
+            },
+            "kibana": {
+                "session_id": "ccZ0sbxrmmJwo+/y2Mn1tmGIrKOuZYaF8voUh0SkA/k=",


@jkakavas this feels like it could be risky to log (from a audit kibana log sample). Do you know if we should chase it down, or with whom? Or maybe if you know it's safe already?

why would it be risky in your opinion? By looking at the docs, it seems that it's an id associated with the current login.

ah, thanks that helps.

At least this is nothing new so if it's an issue we don't have to fix it in this PR.

I just get a little nervous whenever I see a base64 string in a log stream. Too many occasions when they turned out to be access-providing tokens (like JWT). Hopefully this is just an encoded UUID. 😅

...orm_observability/data_stream/kibana_log/elasticsearch/ingest_pipeline/ecs-logs-pipeline.yml

matschaffer

I think this is probably good to merge, especially since it's 0.0.1 so shouldn't end up in a released UI (I think).

This is the deepest I've gone into integrations/packages so could be good to have @klacabane or @neptunian give a look too. Either in this PR or the next one.

matschaffer · 2022-07-22T05:17:30Z

packages/platform_observability/data_stream/kibana_log/manifest.yml

+        required: true
+        show_user: true
+        default:
+          - /var/log/kibana/kibana.stdout


Oh, sorry about the late feedback, but it took an hour for the stack to start today. Seems like this should be kibana.log no? That's what appears in https://www.elastic.co/guide/en/kibana/master/logging-configuration.html anyway

good catch, I've also aligned the audit log file name with what's in Kibana docs

matschaffer · 2022-07-27T07:01:03Z

🎉

crespocarlos added 2 commits June 29, 2022 14:28

create package for ingesting kibana ECS formatted logs

ba764cb

changelog fix

f91647a

crespocarlos commented Jun 29, 2022

View reviewed changes

...form-observability/data_stream/audit/elasticsearch/ingest_pipeline/kibana-audit-logs-ecs.yml Outdated Show resolved Hide resolved

crespocarlos commented Jun 29, 2022

View reviewed changes

packages/platform-observability/data_stream/audit/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

crespocarlos commented Jun 29, 2022

View reviewed changes

packages/platform-observability/data_stream/log/fields/ecs.yml Outdated Show resolved Hide resolved

matschaffer reviewed Jun 30, 2022

View reviewed changes

crespocarlos added 2 commits June 30, 2022 10:35

update test expected result

2da545b

adds a filebeat processor alternative

a8e4920

crespocarlos changed the title ~~create package for ingesting kibana ECS formatted logs~~ [Platform Observability] Create package for ingesting kibana ECS formatted logs Jul 6, 2022

crespocarlos changed the title ~~[Platform Observability] Create package for ingesting kibana ECS formatted logs~~ [Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs Jul 6, 2022

clean up poc code

b171f93

crespocarlos commented Jul 6, 2022

View reviewed changes

...-observability/data_stream/audit/_dev/test/pipeline/test-kibana-audit-logs.log-expected.json Outdated Show resolved Hide resolved

fix json identation

d6cacfd

crespocarlos marked this pull request as ready for review July 6, 2022 11:08

crespocarlos marked this pull request as draft July 7, 2022 07:59

crespocarlos requested review from mtojek, andresrc and ruflin July 7, 2022 08:29

mtojek reviewed Jul 8, 2022

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

crespocarlos marked this pull request as ready for review July 8, 2022 12:04

crespocarlos added 3 commits July 12, 2022 13:18

Align ingest pipeline approach and improve README.md

a3bd0c0

Improve README.md; Fix package folder name; Document ECS fields

6e781a3

Fix supported kibana version on readme

0b03ec6

mtojek suggested changes Jul 18, 2022

View reviewed changes

crespocarlos added 4 commits July 18, 2022 16:52

Remove experimental attribute from manifest; Fix changelog PR; Fix mi…

33f32eb

…n version

Support current and old model for license

8e47160

Fix manifest

3e74c73

Add event.ingested field; Improve Kibana audit fields documentation

14dc44f

crespocarlos requested a review from mtojek July 18, 2022 15:14

Fix pipeline test

21524e8

mtojek approved these changes Jul 19, 2022

View reviewed changes

Extract ECS processor into a new file; Fix kibana logo; Add event.* f…

8cb34da

…ields

matschaffer reviewed Jul 20, 2022

View reviewed changes

crespocarlos added 2 commits July 20, 2022 10:39

Simplify ingest pipeline and clean-ups

55c80e8

Fix json format

5dcc1fe

crespocarlos added the enhancement New feature or request label Jul 20, 2022

matschaffer reviewed Jul 21, 2022

View reviewed changes

...orm_observability/data_stream/kibana_log/elasticsearch/ingest_pipeline/ecs-logs-pipeline.yml Show resolved Hide resolved

matschaffer approved these changes Jul 22, 2022

View reviewed changes

matschaffer reviewed Jul 22, 2022

View reviewed changes

Align default log file name with kibana docs

27a23aa

klacabane approved these changes Jul 25, 2022

View reviewed changes

crespocarlos merged commit 09ef8fd into elastic:main Jul 26, 2022

crespocarlos deleted the 2226-platform-observility-package-poc branch July 26, 2022 14:31

klacabane mentioned this pull request Aug 3, 2022

[Stack Monitoring] Create pre-configured elastic-package environment elastic/kibana#138052

Closed

andrewkroh added Integration:platform_observability Platform Observability New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs #3622

[Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs #3622

crespocarlos commented Jun 29, 2022 •

edited by matschaffer

Loading

elasticmachine commented Jun 29, 2022 •

edited

Loading

Build stats

Test stats 🧪

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer left a comment

matschaffer Jun 30, 2022

crespocarlos Jun 30, 2022

matschaffer Jun 30, 2022

matschaffer Jun 30, 2022 •

edited

Loading

crespocarlos Jul 4, 2022

crespocarlos Jul 4, 2022

crespocarlos Jul 4, 2022

matschaffer Jul 4, 2022

mtojek Jul 11, 2022

mtojek Jul 11, 2022

matschaffer Jul 21, 2022

crespocarlos commented Jul 4, 2022 •

edited

Loading

elasticmachine commented Jul 6, 2022 •

edited

Loading

klacabane commented Jul 6, 2022

klacabane commented Jul 6, 2022

crespocarlos commented Jul 7, 2022 •

edited

Loading

matschaffer commented Jul 7, 2022

mtojek left a comment

matschaffer left a comment •

edited

Loading

matschaffer Jul 21, 2022

crespocarlos Jul 21, 2022 •

edited

Loading

matschaffer Jul 22, 2022

matschaffer left a comment

matschaffer Jul 22, 2022

crespocarlos Jul 22, 2022

matschaffer commented Jul 27, 2022

[Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs #3622

[Platform Observability] Create initial PO package for ingesting kibana ECS formatted logs #3622

Conversation

crespocarlos commented Jun 29, 2022 • edited by matschaffer Loading

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

elasticmachine commented Jun 29, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer commented Jun 30, 2022

matschaffer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matschaffer Jun 30, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crespocarlos commented Jul 4, 2022 • edited Loading

Solutions

Field files

elasticmachine commented Jul 6, 2022 • edited Loading

🌐 Coverage report

klacabane commented Jul 6, 2022

klacabane commented Jul 6, 2022

crespocarlos commented Jul 7, 2022 • edited Loading

matschaffer commented Jul 7, 2022

mtojek left a comment

Choose a reason for hiding this comment

matschaffer left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crespocarlos Jul 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matschaffer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

matschaffer commented Jul 27, 2022

crespocarlos commented Jun 29, 2022 •

edited by matschaffer

Loading

elasticmachine commented Jun 29, 2022 •

edited

Loading

matschaffer Jun 30, 2022 •

edited

Loading

crespocarlos commented Jul 4, 2022 •

edited

Loading

elasticmachine commented Jul 6, 2022 •

edited

Loading

crespocarlos commented Jul 7, 2022 •

edited

Loading

matschaffer left a comment •

edited

Loading

crespocarlos Jul 21, 2022 •

edited

Loading