Enforce and default Kafka configuration

[nasark]

• Enforce Kafka by making the orchestrator wait for Kafka service to be ready (requires Kafka network policy)
• default deployMessagingService to true from CR as Kafka will be the default configuration -> this will retain the option to disable Kafka in case things go wrong

Depends on:

• Allow Kafka communications #935
• Upsize default Kafka volume capacity #936

Needed for:

• Replace Bitnami with Strimzi Kafka #1005

Ref:

• [RFE] Require Kafka manageiq#22225

@miq-bot add_label enhancement
@miq-bot assign @Fryguy
@miq-bot add_reviewer @agrare

[Fryguy]

@bdunne Please review.

[agrare]

@nasark so this removes the option to deploy kafka, does this actually do the deployment?

[nasark]

@nasark so this removes the option to deploy kafka, does this actually do the deployment?

@agrare Actually this removes the flag that would disable Kafka by default. Removing the flag altogether makes it so that Kafka is deployed alongside the other services by default.

[agrare]

Awesome thanks @nasark can you tell what it deploys is it ManageIQ/container-kafka ?

[nasark]

If removing the flag altogether is a concern for backwards compatibility, then we could just set the default value for deployMessagingService to true as well. Some clarification/elaboration regarding #896 (comment) would be appreciated @Fryguy

[Fryguy]

If removing the flag altogether is a concern for backwards compatibility, then we could just set the default value for deployMessagingService to true as well. Some clarification/elaboration regarding #896 (comment) would be appreciated @Fryguy

We may also need a CR migration, but it might not matter if we just don't honor it? cc @bdunne

[Fryguy]

Awesome thanks @nasark can you tell what it deploys is it ManageIQ/container-kafka ?

It would be whatever the value is in the CR - https://github.com/ManageIQ/manageiq-pods/blob/master/manageiq-operator/config/crd/bases/manageiq.org_manageiqs.yaml#L139-L148 seems to show bitnami/kafka and https://github.com/ManageIQ/manageiq-pods/blob/master/manageiq-operator/config/crd/bases/manageiq.org_manageiqs.yaml#L306-L316 shows bitnami/zookeeper.

[Fryguy]

@agrare That would explain why that image hasn't been updated in years 😆

[nasark]

Returning to this, perhaps it is easier to just default deployMessagingService to true instead of removing it from the CR altogether. Let me know what you think and how we can proceed with this @Fryguy @bdunne

[nasark]

I've split some of this up as it was getting big, and moved into other PRs based on issues I have found in my testing:

• Allow Kafka communications #935
• Upsize default Kafka volume capacity #936

[miq-bot]

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s)

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[nasark]

@miq-bot remove_label stale

[miq-bot]

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s)

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[nasark]

@miq-bot remove_label stale

[nasark]

@miq-bot add_label quinteros/yes?

[miq-bot]

Checked commits nasark/manageiq-pods@2a5faca~...4d4fdbf with ruby 2.7.8, rubocop 1.56.3, haml-lint 0.51.0, and yamllint
1 file checked, 1 offense detected

**

• 💣 💥 🔥 🚒 - Linter/Yaml - missing config files

[Fryguy]

Backported to quinteros in commit fe79478.

commit fe79478dc8a65610d2743801b4973825fffe5985
Author: Brandon Dunne <[email protected]>
Date:   Wed Jan 3 12:06:32 2024 -0500

    Merge pull request #896 from nasark/require_kafka
    
    Enforce and default Kafka configuration
    
    (cherry picked from commit 09da4c7f3083fad1b56f345d7e6e8c0063a37f58)