[Fleet][Kafka Output] Tweaks for Kafka UI/API #162875

szwarckonrad · 2023-08-01T09:19:43Z

Removed Kerberos option from auth type selection
Used EuiFieldPassword component for password input
State:
Added None authentication option. This will save auth_type as none and won't pass usernamer, password nor ssl to the SO.
Renamed broker_ack_reliability to required_acks in UI and API. Changed value type to 1 || 0 || -1
Renamed broker_buffer_size to channel_buffer_size in UI and API.
Temporarily disabled password field encryption.
partition key stored in yaml follows proper requirements
topics key stored in yaml follows proper requirements

…te --fix'

kilfoyle · 2023-08-02T19:21:01Z

Hi @szwarckonrad! I'm working on a docs PR for these UI changes.

I have a couple of suggestions if you don't mind:

It might be good to have the Kafka help text match what we have for the Logstash output settings UI.
Kafka output settings UI:

Logstash output settings UI:

Could we change the Kafka UI text to Specify the URLs that your agents will use to connect to Kafka. Learn more?
Once merged, my docs PR will create individual pages for the Logstash output settings and the Kafka output settings. For those "Learn more" links, would it be possible to have them point to <base-path>/ls-output-settings.html and <base-path>/kafka-output-settings.html, respectively, instead of the main UI settings page (fleet-settings.html)?

Thanks!

kevinlog · 2023-08-02T22:52:58Z

@szwarckonrad Thank you for putting together these changes, it's a massive help!

I was chatting with @faec and there are a couple more changes in the conversion to elastic-agent.yml that we should tweak:

specifically partition and topics

Partition

We need to follow the rules here: https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#_partition

Right now partition is the following:

random

    partition: random
    random:
      group_events: 1

So this should look like

    partition:
      random:
        group_events: 1

partition should be an object, i.e. partition.random.group_events: 1

round_robin

Similarly, for round_robin

Right now it looks like

    partition: round_robin
    round_robin:
      group_events: 1

it should be

    partition:
       round_robin:
         group_events: 1

hash

for hash:

    partition:
      hash:
        hash: 'asdasd,asdasd'

yaml generation bug?

I also think there might be a bug here:

When I change the partition a few times, it looks like hash, round_robin, and random get appended to the yaml. We should just be replacing the entire partition field each time depending on the user selection.

I'm seeing this in the yaml, we should follow the format above and ensure we just have partition.{strategy}.{group_events | hash}

    partition: hash
    random:
      group_events: 1
    round_robin:
      group_events: 1
    hash:
      hash: 'asdasd,asdasd'

Topics

We need to follow the rules here: https://www.elastic.co/guide/en/beats/filebeat/current/kafka-output.html#topic-option-kafka

For topics, I'm seeing the following

  topics:
    - topic: asd
      when:
        type: contains
        condition: sdf
    - topic: asd

We should actually see this as:

topics:
  - topic: asd
    when:
      contains:
        message: sdf
  - topic: asd

type: contains should a field under when: that is just called contains.
condition: sdf should be message: sdf and be nested under when.

szwarckonrad · 2023-08-03T12:46:16Z

@kilfoyle I changed the copy, however, hrefs come from kbn-doc-links package and we use them, following logstash example, like that href={docLinks.links.fleet.settings}. My guess would be that change there is required so that we can consume these links for both logstash and kafka. CC @kevinlog

kilfoyle · 2023-08-03T13:43:41Z

I changed the copy, however, hrefs come from kbn-doc-links package and we use them, following logstash example, like that href={docLinks.links.fleet.settings}. My guess would be that change there is required so that we can consume these links for both logstash and kafka.

Nice. Thanks a lot for fixing the copy, Konrad! Now that I understand how the hrefs are handled, I'll wait for my docs PR to merge (so that the URLs exist) and then I'll open a separate issue or PR to have the hrefs updated.

This reverts commit 4fce3b3.

This reverts commit 7248f32.

cmacknz · 2023-08-03T16:55:58Z

Can we add #163093 to the changes in this PR as well?

We should remove the Channel Buffer Size configuration, it's an advanced option so can go in the Advanced YAML parameters box when needed.

kevinlog · 2023-08-03T18:00:49Z

@elasticmachine merge upstream

kevinlog · 2023-08-03T21:24:33Z

@szwarckonrad a few more changes after some additional testing.

Remove channel buffer size

We should remove the channel buffer size field entirely.

#163093

One more tweak to topics

Right now, we're getting the following:

    topics:
      - topic: asd
        contains:
          message: asd
      - topic: asd
        range:
          message: asd
      - topic: sad

We should be getting:

    topics:
      - topic: asd
        when:
          contains:
            message: asd
      - topic: asd
        when:
          range:
            message: asd
      - topic: sad

Note that we should have when above the conditions (i.e. contains, range).

Hosts field should accept `host:port`

The hosts field should actually only support host:port - we should not include the protocol

for instance:

http://localhost:1234 ❌
localhost:1234 ✅

Right now the validation is requiring that we enter a URL. We should just be enforcing host:port.

The client ID field should not have any spaces

The Client ID field should NOT have any spaces. We should change the default to Elastic (one word). Also, we should add validation with this regex ^[A-Za-z0-9._-]+$

nchaulet · 2023-08-08T14:07:01Z

x-pack/plugins/fleet/server/saved_objects/index.ts

@@ -229,6 +232,29 @@ const getSavedObjectTypes = (): { [key: string]: SavedObjectsType } => ({
        broker_timeout: { type: 'integer' },
        broker_ack_reliability: { type: 'text' },
        broker_buffer_size: { type: 'integer' },
+        required_acks: { type: 'integer' },


We should probably add index: false to all of those kafka fields as it seems to me that we will not search on that

elasticmachine · 2023-08-08T14:07:12Z

Pinging @elastic/fleet (Team:Fleet)

juliaElastic · 2023-08-09T09:44:39Z

...lic/applications/fleet/sections/settings/components/edit_output_flyout/output_form_kafka.tsx

@@ -84,13 +84,13 @@ export const OutputFormKafkaSection: React.FunctionComponent<Props> = (props) =>
        helpText={
          <FormattedMessage
            id="xpack.fleet.settings.editOutputFlyout.kafkaHostsInputDescription"
-            defaultMessage="Specify the URLs that your agents will use to connect to Kafka. For more information, see the {guideLink}."
+            defaultMessage="Specify the URLs that your agents will use to connect to Kafka. {guideLink}."
            values={{
              guideLink: (
                <EuiLink href={docLinks.links.fleet.settings} target="_blank" external>


Are we going to update this to a Kafka specific doc link?

#162875 (comment)
:)

juliaElastic · 2023-08-09T09:47:57Z

x-pack/plugins/fleet/server/saved_objects/migrations/to_v8_10_0.ts

could we add a unit test on the migration?

juliaElastic

LGTM, please check comments.

gsoldevila

LGTM changes in the ingest-outputs mappings are compatible (non-breaking).

nchaulet · 2023-08-09T14:24:24Z

I think there is one part we may been missing for Kafka output (this should probably be a follow up PR) it's the whole preconfiguration thing to be able to define a kafka output in kibana config file, probably need to update that function and check the schema support it

kibana/x-pack/plugins/fleet/server/services/preconfiguration/outputs.ts

Line 167 in 40db94a

function isPreconfiguredOutputDifferentFromCurrent(

kibana-ci · 2023-08-09T15:35:36Z

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

[job] [logs] Defend Workflows Cypress Tests #1 / Endpoint Policy Response from Fleet Agent Details page should display policy response with errors should display policy response with errors
[job] [logs] Defend Workflows Endpoint Cypress Tests #5 / Isolate command From cases should isolate and release host should isolate and release host
[job] [logs] FTR Configs #52 / ObservabilityApp Observability Rules page Rules table shows the rules table

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	1012.8KB	1016.1KB	+3.2KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	136.5KB	136.7KB	+163.0B

History

💚 Build #148744 succeeded 4128117
💔 Build #148709 failed 1e5cb09
💚 Build #148651 succeeded 88ec5e7
💛 Build #148282 was flaky 0bf9fcb
💚 Build #147998 succeeded ef21a4a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

1. Removed Kerberos option from auth type selection 2. Used EuiFieldPassword component for password input State: ![Screenshot 2023-08-01 at 11 16 30](https://github.com/elastic/kibana/assets/29123534/3ef0b3f8-c98b-42d0-9645-fdb6437859f5) 3. Added `None` authentication option. This will save `auth_type` as `none` and won't pass `usernamer`, `password` nor `ssl` to the SO. ![Screenshot 2023-08-02 at 15 59 08](https://github.com/elastic/kibana/assets/29123534/9bbe9ce7-2749-484f-833d-72d1c9e5ee46) 4. Renamed `broker_ack_reliability` to `required_acks` in UI and API. Changed value type to `1 || 0 || -1` 5. Renamed `broker_buffer_size` to `channel_buffer_size` in UI and API. 6. Temporarily disabled `password` field encryption. 7. `partition` key stored in yaml follows proper requirements 8. `topics` key stored in yaml follows proper requirements --------- Co-authored-by: kibanamachine <[email protected]>

This PR addresses comment #162875 (comment) I believe schema is already covered here - https://github.com/elastic/kibana/pull/159110/files#diff-2a6e1a6445463ee01c508ec3b6a7441635045a203dc94e58784fcba721747914R85-R89 --------- Co-authored-by: Julia Bardi <[email protected]>

szwarckonrad added 5 commits August 1, 2023 11:17

Remove Kerberos option, mask password

375a009

align cypress

a161d19

Merge branch 'main' into kafka-tweaks

cede695

align cypress

2b56d8d

jest tests aligned

50cafe4

kevinlog added the ci:cloud-deploy Create or update a Cloud deployment label Aug 2, 2023

szwarckonrad added release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels Aug 2, 2023

szwarckonrad and others added 2 commits August 2, 2023 19:17

mappings

0d0ffc5

[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…

c0625bd

…te --fix'

szwarckonrad added 2 commits August 3, 2023 09:12

Merge branch 'main' into kafka-tweaks

47a9416

partition

7248f32

szwarckonrad added 2 commits August 3, 2023 14:55

Merge branch 'main' into kafka-tweaks

20df38a

topics

4fce3b3

szwarckonrad added 3 commits August 3, 2023 16:23

Revert "topics"

d866966

This reverts commit 4fce3b3.

Revert "partition"

a47ca08

This reverts commit 7248f32.

topics, partition yaml transfers

c8e1915

kevinlog added ci:cloud-redeploy Always create a new Cloud deployment ci:cloud-deploy Create or update a Cloud deployment and removed ci:cloud-deploy Create or update a Cloud deployment labels Aug 3, 2023

Merge branch 'main' into kafka-tweaks

86c51cd

kilfoyle mentioned this pull request Aug 3, 2023

Add docs for Kafka output UI for Fleet managed agents elastic/ingest-docs#336

Merged

nested when property on topic

13c0a61

szwarckonrad requested a review from a team as a code owner August 8, 2023 13:43

nchaulet reviewed Aug 8, 2023

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Aug 8, 2023

kevinlog added the v8.10.0 label Aug 8, 2023

szwarckonrad added 2 commits August 9, 2023 09:58

Merge branch 'main' into kafka-tweaks

adc8a69

encrypted password SO field

88ec5e7

juliaElastic reviewed Aug 9, 2023

View reviewed changes

juliaElastic approved these changes Aug 9, 2023

View reviewed changes

gsoldevila approved these changes Aug 9, 2023

View reviewed changes

szwarckonrad added 4 commits August 9, 2023 14:08

Merge branch 'main' into kafka-tweaks

47806a4

so mappings and unit test

1e5cb09

so mappings reverted

b79ac72

Merge branch 'main' into kafka-tweaks

4128117

nchaulet self-requested a review August 9, 2023 13:37

nchaulet approved these changes Aug 9, 2023

View reviewed changes

Merge branch 'main' into kafka-tweaks

022ac86

szwarckonrad enabled auto-merge (squash) August 9, 2023 14:59

szwarckonrad merged commit e2b10cb into elastic:main Aug 9, 2023

kilfoyle mentioned this pull request Aug 9, 2023

Fix links from Kibana/Fleet UI for Kafka and Logstash output docs elastic/ingest-docs#405

Closed

szwarckonrad mentioned this pull request Aug 10, 2023

[Fleet] Kafka preconfiguration #163601

Merged

jlind23 mentioned this pull request Aug 14, 2023

[Fleet] Implement Kafka output form UI #143324

Closed

62 tasks

kilfoyle mentioned this pull request Aug 21, 2023

[Elastic Agent] Document Kafka Output UI for fleet managed agents elastic/ingest-docs#68

Closed

nchaulet mentioned this pull request Feb 13, 2024

[Fleet] Do not require configuring the Kafka output Channel Buffer Size parameter #163093

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet][Kafka Output] Tweaks for Kafka UI/API #162875

[Fleet][Kafka Output] Tweaks for Kafka UI/API #162875

szwarckonrad commented Aug 1, 2023 •

edited

Loading

kilfoyle commented Aug 2, 2023 •

edited

Loading

kevinlog commented Aug 2, 2023

szwarckonrad commented Aug 3, 2023

kilfoyle commented Aug 3, 2023

cmacknz commented Aug 3, 2023

kevinlog commented Aug 3, 2023

kevinlog commented Aug 3, 2023

nchaulet Aug 8, 2023

elasticmachine commented Aug 8, 2023

juliaElastic Aug 9, 2023

szwarckonrad Aug 9, 2023

juliaElastic Aug 9, 2023

juliaElastic left a comment

gsoldevila left a comment

nchaulet commented Aug 9, 2023

kibana-ci commented Aug 9, 2023 •

edited

Loading

[Fleet][Kafka Output] Tweaks for Kafka UI/API #162875

[Fleet][Kafka Output] Tweaks for Kafka UI/API #162875

Conversation

szwarckonrad commented Aug 1, 2023 • edited Loading

kilfoyle commented Aug 2, 2023 • edited Loading

kevinlog commented Aug 2, 2023

Partition

random

round_robin

hash

yaml generation bug?

Topics

szwarckonrad commented Aug 3, 2023

kilfoyle commented Aug 3, 2023

cmacknz commented Aug 3, 2023

kevinlog commented Aug 3, 2023

kevinlog commented Aug 3, 2023

Remove channel buffer size

One more tweak to topics

Hosts field should accept host:port

The client ID field should not have any spaces

nchaulet Aug 8, 2023

Choose a reason for hiding this comment

elasticmachine commented Aug 8, 2023

juliaElastic Aug 9, 2023

Choose a reason for hiding this comment

szwarckonrad Aug 9, 2023

Choose a reason for hiding this comment

juliaElastic Aug 9, 2023

Choose a reason for hiding this comment

juliaElastic left a comment

Choose a reason for hiding this comment

gsoldevila left a comment

Choose a reason for hiding this comment

nchaulet commented Aug 9, 2023

kibana-ci commented Aug 9, 2023 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

Page load bundle

History

szwarckonrad commented Aug 1, 2023 •

edited

Loading

kilfoyle commented Aug 2, 2023 •

edited

Loading

Hosts field should accept `host:port`

kibana-ci commented Aug 9, 2023 •

edited

Loading