Init container updates for persisting kibana plugins. #8389

naemono · 2025-01-02T22:45:33Z

resolves: #8388

With the recent changes in #7787 we broke users being able to install custom Kibana plugins. This updates the existing init container for Kibana configuration, and adds the ability to copy plugins to an emptyDir volume, and then mount this to /usr/share/kibana/plugins in the primary Kibana container. (effectively mirroring what we do with Elasticsearch)

Review Notes

Changes

I've moved the initconatiner logic into it's own package, similar to what's done for elasticsearch. (pkg/controller/elasticsearch/initcontainer)
I've moved many/most of the variables for volume naming/mountPoints to a new settings package to avoid import cycles.
I've combined both the plugins copying logic, and the kibana config creation logic into the same init container that already existed, and renamed it appropriately.

Testing

I've manually tested in an existing cluster that the kibana plugins copying works as expected, and noted this in the comments (will update with additional testing from last commit shortly). See notes/comments here, and here.
I've also tested a keystore along-side this new init container.
I'll be triggering a full set of e2e tests today.
I've updated/added many unit tests.

Signed-off-by: Michael Montgomery <[email protected]>

updating unit tests. Signed-off-by: Michael Montgomery <[email protected]>

pebrc · 2025-01-03T14:47:04Z

Should we reuse the existing init container for this? What is the overhead of adding another container?

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-01-03T16:14:12Z

Should we reuse the existing init container for this? What is the overhead of adding another container?

The problem with using the existing init container is that the init-config init container is always included in the pod, and the init-filesystem is only included when the version >= 7.10 and the flag for setting security context is enabled. I thought this would make the logic more straight-forward. I guess we could include some templating logic in the original init container to only include that block when enabled, and only include the volumes as well. I'll verify that the approach I'm taking is working, then I'll see about adjusting to use the original init.

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-01-06T15:54:36Z

Ok, I've verified that this is working as intended by:

Creating a fake kibana plugin and hosting it on GH.
Creating a Dockerfile to use the fake plugin:

❯ cat Dockerfile.kibana-plugins
FROM docker.elastic.co/kibana/kibana:8.17.0
RUN /usr/share/kibana/bin/kibana-plugin install https://github.com/naemono/fake-kibana-plugin/releases/download/v0.0.1/fakePlugin-v8.17.0.zip
RUN /usr/share/kibana/bin/kibana --optimize

Updating Kibana to use the image:

❯ kc get pod -n elastic eck-stack-eck-kibana-kb-5b9c947db-2nwq9 -o yaml | yq '.spec.containers[].image'
mmontg1/kibana-with-plugin:8.17.0

Verifying in the logs that the copy operation is doing what we would expect:

❯ kc logs -n elastic eck-stack-eck-kibana-kb-5b9c947db-2nwq9 -c elastic-internal-init-filesystem
Copying /usr/share/kibana/plugins/* to /mnt/elastic-internal/kibana-plugins-local/
'/usr/share/kibana/plugins/fakePlugin' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin'
'/usr/share/kibana/plugins/fakePlugin/.i18nrc.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/.i18nrc.json'
'/usr/share/kibana/plugins/fakePlugin/common' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/common'
'/usr/share/kibana/plugins/fakePlugin/common/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/common/index.js'
'/usr/share/kibana/plugins/fakePlugin/kibana.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/kibana.json'
'/usr/share/kibana/plugins/fakePlugin/node_modules' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/node_modules'
'/usr/share/kibana/plugins/fakePlugin/node_modules/.yarn-integrity' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/node_modules/.yarn-integrity'
'/usr/share/kibana/plugins/fakePlugin/package.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/package.json'
'/usr/share/kibana/plugins/fakePlugin/server' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server'
'/usr/share/kibana/plugins/fakePlugin/server/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/index.js'
'/usr/share/kibana/plugins/fakePlugin/server/plugin.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/plugin.js'
'/usr/share/kibana/plugins/fakePlugin/server/routes' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/routes'
'/usr/share/kibana/plugins/fakePlugin/server/routes/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/routes/index.js'
'/usr/share/kibana/plugins/fakePlugin/server/types.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/types.js'
'/usr/share/kibana/plugins/fakePlugin/target' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target'
'/usr/share/kibana/plugins/fakePlugin/target/public' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.chunk.1.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.chunk.1.js'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.chunk.1.js.br' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.chunk.1.js.br'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.plugin.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.plugin.js'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.plugin.js.br' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.plugin.js.br'
'/usr/share/kibana/plugins/fakePlugin/translations' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/translations'
'/usr/share/kibana/plugins/fakePlugin/translations/ja-JP.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/translations/ja-JP.json'
'/usr/share/kibana/plugins/fakePlugin/tsconfig.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/tsconfig.json'
Files copy duration: 0 sec.

Verifying in the KB UI that the plugin appears:

I'm going to now see about adding this functionality to the existing init container, and not adding a 2nd....

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-01-08T19:49:40Z

Logs from recent commit showing init container that does both plugins initialization/copy, and config init.

❯ kc logs -n elastic eck-stack-eck-kibana-kb-6cb66db678-lpdwr -c elastic-internal-init
++ date +%s
+ mv_start=1736365111
++ ls -A /usr/share/kibana/plugins
+ [[ -z fakePlugin ]]
+ echo 'Copying /usr/share/kibana/plugins/* to /mnt/elastic-internal/kibana-plugins-local/'
Copying /usr/share/kibana/plugins/* to /mnt/elastic-internal/kibana-plugins-local/
+ yes
+ cp -avf /usr/share/kibana/plugins/fakePlugin /mnt/elastic-internal/kibana-plugins-local/
'/usr/share/kibana/plugins/fakePlugin' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin'
'/usr/share/kibana/plugins/fakePlugin/.i18nrc.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/.i18nrc.json'
'/usr/share/kibana/plugins/fakePlugin/common' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/common'
'/usr/share/kibana/plugins/fakePlugin/common/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/common/index.js'
'/usr/share/kibana/plugins/fakePlugin/kibana.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/kibana.json'
'/usr/share/kibana/plugins/fakePlugin/node_modules' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/node_modules'
'/usr/share/kibana/plugins/fakePlugin/node_modules/.yarn-integrity' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/node_modules/.yarn-integrity'
'/usr/share/kibana/plugins/fakePlugin/package.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/package.json'
'/usr/share/kibana/plugins/fakePlugin/server' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server'
'/usr/share/kibana/plugins/fakePlugin/server/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/index.js'
'/usr/share/kibana/plugins/fakePlugin/server/plugin.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/plugin.js'
'/usr/share/kibana/plugins/fakePlugin/server/routes' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/routes'
'/usr/share/kibana/plugins/fakePlugin/server/routes/index.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/routes/index.js'
'/usr/share/kibana/plugins/fakePlugin/server/types.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/server/types.js'
'/usr/share/kibana/plugins/fakePlugin/target' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target'
'/usr/share/kibana/plugins/fakePlugin/target/public' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.chunk.1.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.chunk.1.js'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.chunk.1.js.br' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.chunk.1.js.br'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.plugin.js' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.plugin.js'
'/usr/share/kibana/plugins/fakePlugin/target/public/fakePlugin.plugin.js.br' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/target/public/fakePlugin.plugin.js.br'
'/usr/share/kibana/plugins/fakePlugin/translations' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/translations'
'/usr/share/kibana/plugins/fakePlugin/translations/ja-JP.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/translations/ja-JP.json'
'/usr/share/kibana/plugins/fakePlugin/tsconfig.json' -> '/mnt/elastic-internal/kibana-plugins-local/fakePlugin/tsconfig.json'
++ duration 1736365111
++ local start=1736365111
+++ date +%s
++ end=1736365111
++ echo 0
+ echo 'Files copy duration: 0 sec.'
+ init_config_initialized_flag=/mnt/elastic-internal/kibana-config-local/elastic-internal-init-config.ok
Files copy duration: 0 sec.
+ [[ -f /mnt/elastic-internal/kibana-config-local/elastic-internal-init-config.ok ]]
+ echo 'Setup Kibana configuration'
+ ln -sf /mnt/elastic-internal/kibana-config/kibana.yml /mnt/elastic-internal/kibana-config/telemetry.yml /mnt/elastic-internal/kibana-config-local/
Setup Kibana configuration
+ touch /mnt/elastic-internal/kibana-config-local/elastic-internal-init-config.ok
+ echo 'Kibana configuration successfully prepared.'
Kibana configuration successfully prepared.

Also keystore init container logs:

❯ kc logs -n elastic eck-stack-eck-kibana-kb-6cb66db678-lpdwr -c elastic-internal-init-keystore
Initializing keystore.
+ keystore_initialized_flag=/usr/share/kibana/config/elastic-internal-init-keystore.ok
+ [[ -f /usr/share/kibana/config/elastic-internal-init-keystore.ok ]]
+ echo 'Initializing keystore.'
+ /usr/share/kibana/bin/kibana-keystore create
Created Kibana keystore in /usr/share/kibana/config/kibana.keystore
+ for filename in /mnt/elastic-internal/secure-settings/*
+ [[ -e /mnt/elastic-internal/secure-settings/server.ssl.enabled ]]
++ basename /mnt/elastic-internal/secure-settings/server.ssl.enabled
+ key=server.ssl.enabled
+ echo 'Adding server.ssl.enabled to the keystore.'
+ /usr/share/kibana/bin/kibana-keystore add server.ssl.enabled --stdin
Adding server.ssl.enabled to the keystore.
+ touch /usr/share/kibana/config/elastic-internal-init-keystore.ok
Keystore initialization successful.
+ echo 'Keystore initialization successful.'

Signed-off-by: Michael Montgomery <[email protected]>

Always include plugins copying. Signed-off-by: Michael Montgomery <[email protected]>

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-01-13T15:06:34Z

buildkite test this -f E2E_TAGS=kb -m p=gke,p=ocp

pebrc

LGTM (a few small nits)

pkg/apis/kibana/v1/name.go

pkg/apis/kibana/v1/name_test.go

pkg/controller/kibana/settings/settings.go

Signed-off-by: Michael Montgomery <[email protected]>

barkbay

LGTM! 👍

Let's maybe not forget to include something in the release notes to mention that this change is going to restart Kibana Pods for which hardened security is supported.

Signed-off-by: Michael Montgomery <[email protected]>

naemono · 2025-01-14T16:01:57Z

LGTM! 👍

Let's maybe not forget to include something in the release notes to mention that this change is going to restart Kibana Pods for which hardened security is supported.

I've updated the highlights in 0a79a1a. Lmk if this wording works @barkbay .

barkbay · 2025-01-14T16:10:48Z

docs/release-notes/highlights-2.16.0.asciidoc

@@ -32,6 +32,8 @@ securityContext:
  readOnlyRootFilesystem: true
 ----

+Also note that this change will cause the Kibana pod(s) to be restarted as this change is applied during the upgrade.


I think I would expect this in docs/release-notes/highlights-2.16.1.asciidoc?

🤦‍♂️ ~~I'll get this updated. My apologies.~~ ☕

Actually, this change is perfectly valid as this change in 2.16.0 (with security context added, and additional volumes added) will most certainly cause a restart of the Kibana pods. I'm going to not add any 2.16.1 additions to this PR, and create them separately in an additional PR to include highlights/release notes/etc.

naemono · 2025-01-15T20:21:13Z

💚 All backports created successfully

Status	Branch	Result
✅	2.16.1-wip

Questions ?

Please refer to the Backport tool documentation

* Init container for persisting kibana plugins. --------- Signed-off-by: Michael Montgomery <[email protected]> (cherry picked from commit 0efeb94)

naemono added 2 commits January 2, 2025 14:00

Init container for persisting kibana plugins.

2e7be97

Signed-off-by: Michael Montgomery <[email protected]>

rendering configmap script.

af17827

updating unit tests. Signed-off-by: Michael Montgomery <[email protected]>

botelastic bot added the triage label Jan 2, 2025

naemono added 3 commits January 3, 2025 09:09

Adding missing file.

0ca1bca

Signed-off-by: Michael Montgomery <[email protected]>

Fixing newpodtemplatespec unit test

e7488db

Signed-off-by: Michael Montgomery <[email protected]>

Fix unit tests.

30aca6c

Signed-off-by: Michael Montgomery <[email protected]>

naemono added 2 commits January 3, 2025 10:21

Remove unneeded vars/funcs

39d0f3e

Signed-off-by: Michael Montgomery <[email protected]>

Add missing duration function

2b6b9ef

Signed-off-by: Michael Montgomery <[email protected]>

naemono added 13 commits January 6, 2025 13:19

wip

4280ae5

Signed-off-by: Michael Montgomery <[email protected]>

cleanup

77346ac

Signed-off-by: Michael Montgomery <[email protected]>

cleanup

d43755a

Signed-off-by: Michael Montgomery <[email protected]>

cleanup

4989861

Signed-off-by: Michael Montgomery <[email protected]>

re-order the initcontainer config script

a485485

Signed-off-by: Michael Montgomery <[email protected]>

Cleanup

437ef96

Signed-off-by: Michael Montgomery <[email protected]>

Fix unit tests

ee769ba

Signed-off-by: Michael Montgomery <[email protected]>

Add test for rendering template.

3712ed6

Signed-off-by: Michael Montgomery <[email protected]>

fix e2e kb test telemetry.

87c6bf4

Signed-off-by: Michael Montgomery <[email protected]>

optimization

e56d346

Signed-off-by: Michael Montgomery <[email protected]>

Add additional unit tests

24b1a88

Signed-off-by: Michael Montgomery <[email protected]>

Add additional unit tests

e1bd658

Signed-off-by: Michael Montgomery <[email protected]>

Add back commented tests

c7c9e86

Signed-off-by: Michael Montgomery <[email protected]>

thbkrkr added the >enhancement Enhancement of existing functionality label Jan 8, 2025

botelastic bot removed the triage label Jan 8, 2025

naemono added 3 commits January 8, 2025 11:42

Add additional comments/godoc.

0ff1cad

Signed-off-by: Michael Montgomery <[email protected]>

Fix comments.

47ec183

Signed-off-by: Michael Montgomery <[email protected]>

Add comments, and update comments.

00b24ed

Signed-off-by: Michael Montgomery <[email protected]>

naemono added 10 commits January 10, 2025 11:23

use consistent if err syntax.

bf06b56

Signed-off-by: Michael Montgomery <[email protected]>

don't export.

99547b8

Signed-off-by: Michael Montgomery <[email protected]>

Check for subset of maps, not equality.

566a81d

Signed-off-by: Michael Montgomery <[email protected]>

Add labels and annotations to updateReconciled as well.

43a8087

Signed-off-by: Michael Montgomery <[email protected]>

Inline some functions.

af7edef

Signed-off-by: Michael Montgomery <[email protected]>

remove var

9f95cbf

Signed-off-by: Michael Montgomery <[email protected]>

Fix unit tests

70799c3

Always include plugins copying. Signed-off-by: Michael Montgomery <[email protected]>

Also watch configmaps owned by kb.

e477764

Signed-off-by: Michael Montgomery <[email protected]>

Fix linting issues

a867725

Signed-off-by: Michael Montgomery <[email protected]>

more linting issues

6e9d36a

Signed-off-by: Michael Montgomery <[email protected]>

naemono requested review from barkbay and pebrc January 13, 2025 15:06

pebrc approved these changes Jan 13, 2025

View reviewed changes

pkg/apis/kibana/v1/name.go Outdated Show resolved Hide resolved

pkg/apis/kibana/v1/name_test.go Outdated Show resolved Hide resolved

pkg/apis/kibana/v1/name_test.go Outdated Show resolved Hide resolved

pkg/controller/kibana/settings/settings.go Outdated Show resolved Hide resolved

naemono added 3 commits January 13, 2025 16:20

review changes.

1feceff

Signed-off-by: Michael Montgomery <[email protected]>

Fix namer

88f995d

Signed-off-by: Michael Montgomery <[email protected]>

Up timeout of linter.

a3382b5

Signed-off-by: Michael Montgomery <[email protected]>

barkbay approved these changes Jan 14, 2025

View reviewed changes

naemono added 2 commits January 14, 2025 09:55

remove commented code

198b885

Signed-off-by: Michael Montgomery <[email protected]>

Update 2.16.0 highlights to note the kibana pod restart.

0a79a1a

Signed-off-by: Michael Montgomery <[email protected]>

barkbay reviewed Jan 14, 2025

View reviewed changes

pebrc approved these changes Jan 14, 2025

View reviewed changes

naemono merged commit 0efeb94 into elastic:main Jan 14, 2025
5 checks passed

naemono deleted the kibana-plugins-init-container branch January 14, 2025 17:37

naemono mentioned this pull request Jan 15, 2025

[2.16.1-wip] Init container updates for persisting kibana plugins. (#8389) #8411

Merged

FlorisFeddema mentioned this pull request Jan 20, 2025

Kibana elastic-internal-init-config initcontainer does not complete after ECK Operator 2.16.1 #8426

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Init container updates for persisting kibana plugins. #8389

Init container updates for persisting kibana plugins. #8389

naemono commented Jan 2, 2025 •

edited

Loading

pebrc commented Jan 3, 2025

naemono commented Jan 3, 2025

naemono commented Jan 6, 2025

naemono commented Jan 8, 2025

naemono commented Jan 13, 2025

pebrc left a comment

barkbay left a comment

naemono commented Jan 14, 2025

barkbay Jan 14, 2025

naemono Jan 14, 2025 •

edited

Loading

naemono commented Jan 15, 2025

Init container updates for persisting kibana plugins. #8389

Init container updates for persisting kibana plugins. #8389

Conversation

naemono commented Jan 2, 2025 • edited Loading

Review Notes

Changes

Testing

pebrc commented Jan 3, 2025

naemono commented Jan 3, 2025

naemono commented Jan 6, 2025

naemono commented Jan 8, 2025

naemono commented Jan 13, 2025

pebrc left a comment

Choose a reason for hiding this comment

barkbay left a comment

Choose a reason for hiding this comment

naemono commented Jan 14, 2025

barkbay Jan 14, 2025

Choose a reason for hiding this comment

naemono Jan 14, 2025 • edited Loading

Choose a reason for hiding this comment

naemono commented Jan 15, 2025

💚 All backports created successfully

Questions ?

naemono commented Jan 2, 2025 •

edited

Loading

naemono Jan 14, 2025 •

edited

Loading