[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations #211847

markjhoy · 2025-02-19T20:38:55Z

Summary

The previous PR for deprecating Enterprise Search indices before 8.x did not handle all of the necessary datastreams to be rolled over. This adds those. Also, the configuration check for enterpriseSearch.customHeaders has now been rolled into the Enterprise Search node removal and configuration check.

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.

jloleysens

Did not test locally, just did a code review. Left a q about testing on cloud and I think we should also add hidden to expand wildcards for extra safety in pickup up indices.

jloleysens · 2025-02-20T08:59:06Z

x-pack/solutions/search/plugins/enterprise_search/server/deprecations/index.ts

@@ -120,6 +124,7 @@ export function getEnterpriseSearchNodeDeprecation(
              'Enterprise Search is not supported in versions >= 9.x.\n\n' +
              'Please note the following:\n' +
              '- You must remove any Enterprise Search nodes from your deployment to proceed with the upgrade.\n' +
+              '- You must also remove any Enterprise Search configuration elements in your Kibana config.\n' +


Should we list the enterprise search configuration elements? Or perhaps the namespace xpack.enterpriseSearch.*?

Have we verified that this config can be removed by cloud users?

Have we verified that this config can be removed by cloud users?

We tried... and this config cannot be removed by cloud users - these instructions only appear in self-managed.

jloleysens · 2025-02-20T09:04:21Z

...solutions/search/plugins/enterprise_search/server/deprecations/pre_eight_index_deprecator.ts

-  }
+  const entSearchDatastreams = await esClient.indices.getDataStream({
+    name: ENT_SEARCH_DATASTREAM_PATTERN.join(','),
+    expand_wildcards: ['all'],


Suggested change

expand_wildcards: ['all'],

expand_wildcards: ['all' ,'hidden'],

Doesn't all include hidden? Or is that only for getting indices?

jloleysens · 2025-02-20T09:09:39Z

...solutions/search/plugins/enterprise_search/server/deprecations/pre_eight_index_deprecator.ts

@@ -24,9 +30,10 @@ export async function getPreEightEnterpriseSearchIndices(
    expand_wildcards: ['all'],


Suggested change

expand_wildcards: ['all'],

expand_wildcards: ['all', 'hidden'],

jloleysens · 2025-02-20T09:17:16Z

...solutions/search/plugins/enterprise_search/server/deprecations/pre_eight_index_deprecator.ts

+      existingIndex.hasDatastream = true;
+      existingIndex.datastreams.push(datastream);
+    } else {
+      returnIndices.push({
+        name: lastIndexName,
+        hasDatastream: true,
+        datastreams: [datastream],


I think this makes sense: if the index is already picked up we make sure it's marked as having a data stream. Otherwise it's not been picked up and we add to the list of indices to make read-only.

jloleysens

Wait there was one other thing that stood out to me, from a previous PR actually:

      indexData.settings?.index?.version?.created?.startsWith('7') &&
      indexData.settings?.index?.blocks?.write !== 'true'

I think we should remove the indexData.settings?.index?.blocks?.write !== 'true'. It's technically not the thing we should be checking for. There's a special verified_read_only setting.

But besides that, I think it's safe to do this action multiple times against the same indices.

jloleysens · 2025-02-20T09:38:23Z

...solutions/search/plugins/enterprise_search/server/deprecations/pre_eight_index_deprecator.ts

+  'logs-enterprise_search.*',
+  'logs-app_search.*',
+  'logs-workplace_search.*',
+  'logs-crawler-*',


Sorry for the late thoughts... I am a bit concerned about this pattern... Is it conceivable that users may have data streams NOT created/owned by enterprise search may match logs-crawler-*?

All of these are managed by Enterprise Search... The logs-crawler-* items are for the App Search crawler, that is from 7.x onwards... the managed crawler has the logs-elastic_crawler-* prefix.. .;)

jloleysens

Did not test locally, thanks for addressing my feedback!

markjhoy · 2025-02-20T15:59:40Z

Just an FYI - I've removed the logs-crawler-* pattern, because the underlying index will be .ent-search-crawler-ecs-ilm-* which should be picked up by the GET on the indices, and have the associated datastream there that will be rolled over

markjhoy · 2025-02-21T17:08:10Z

With the latest commit, I've moved the pre 8.x index deprecation pieces into the Kibana Upgrade Assistant plugin - as it's possible that if the Enterprise Search plugin is disabled, these may not show up

jloleysens

Great work @markjhoy , I think we're finally there haha. Left a few non-blocker comments.

jloleysens · 2025-02-24T08:15:56Z

...rivate/upgrade_assistant/server/lib/enterprise_search/enterprise_search_deprecations.test.ts

Would you mind adding a CODEOWNERS entry like:

x-pack/platform/plugins/private/upgrade_assistant/server/lib/enterprise_search @elastic/search-kibana

jloleysens · 2025-02-24T08:17:18Z

...plugins/private/upgrade_assistant/server/lib/enterprise_search/pre_eight_index_deprecator.ts

@@ -8,11 +8,16 @@
 import { ElasticsearchClient } from '@kbn/core/server';

 const ENT_SEARCH_INDEX_PREFIX = '.ent-search-';
+const ENT_SEARCH_DATASTREAM_PATTERN = [


I'll update my PR to use these values directly 👌🏻

jloleysens · 2025-02-24T08:19:00Z

...k/platform/plugins/private/upgrade_assistant/server/routes/enterprise_search_deprecations.ts

WDYT about moving these into your ../lib/enterprise_search folder? Since we're already doing something a little weird, having routes dir in ../lib/enterprise_search/routes seems OK?

…ocs'

elasticmachine · 2025-02-24T16:32:58Z

💚 Build Succeeded

Buildkite Build
Commit: b62b2ab

Metrics [docs]

✅ unchanged

History

💛 Build #277904 was flaky c2b0ae9
💚 Build #277860 succeeded 707eaf3

markjhoy · 2025-02-24T17:19:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.18

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ic#211847) ## Summary The [previous PR](elastic#210688) for deprecating Enterprise Search indices before 8.x did not handle all of the necessary datastreams to be rolled over. This adds those. Also, the configuration check for `enterpriseSearch.customHeaders` has now been rolled into the Enterprise Search node removal and configuration check. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/src/platform/packages/shared/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit bf8cfca)

…#211847) (#212285) # Backport This will backport the following commits from `8.x` to `8.18`: - [[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations (#211847)](#211847)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport)

…ions (#211881) ## Summary Close #211712 Related #211847

…ions (elastic#211881) ## Summary Close elastic#211712 Related elastic#211847 (cherry picked from commit 4c42363) # Conflicts: # x-pack/platform/plugins/private/upgrade_assistant/server/lib/es_deprecations_status/index.test.ts

ensure datastream deprecations; modify config dep.

707eaf3

markjhoy added release_note:skip Skip the PR/issue when compiling release notes Feature:Upgrade Assistant Team:EnterpriseSearch v8.18.0 v8.19.0 backport:8.18 labels Feb 19, 2025

markjhoy requested a review from kibanamachine as a code owner February 19, 2025 20:38

markjhoy requested a review from jloleysens February 19, 2025 20:39

Merge branch '8.x' into markjhoy/additional_ent_search_deprecations

c2b0ae9

jloleysens approved these changes Feb 20, 2025

View reviewed changes

jloleysens reviewed Feb 20, 2025

View reviewed changes

jloleysens mentioned this pull request Feb 20, 2025

[UA] Exclude critical, reindex enterprise search data stream deprecations #211881

Merged

markjhoy added 2 commits February 20, 2025 09:03

Merge branch '8.x' into markjhoy/additional_ent_search_deprecations

4476c2d

use verified_read_only; expand hidden wildcards

7a1c2db

jloleysens approved these changes Feb 20, 2025

View reviewed changes

remove crawler datastream pattern

4eac146

move EntSearch pre 8.x index deps to UA

c305b5d

markjhoy requested a review from jloleysens February 21, 2025 17:07

[CI] Auto-commit changed files from 'node scripts/yarn_deduplicate'

28575a2

jloleysens approved these changes Feb 24, 2025

View reviewed changes

markjhoy and others added 3 commits February 24, 2025 08:48

Merge branch '8.x' into markjhoy/additional_ent_search_deprecations

85b8e85

move routing; add codeowners

6713b5d

[CI] Auto-commit changed files from 'node scripts/build_plugin_list_d…

b62b2ab

…ocs'

markjhoy enabled auto-merge (squash) February 24, 2025 14:56

markjhoy merged commit bf8cfca into elastic:8.x Feb 24, 2025
8 checks passed

markjhoy mentioned this pull request Feb 24, 2025

[8.18] [8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations (#211847) #212285

Merged

jloleysens added a commit that referenced this pull request Feb 25, 2025

[UA] Exclude critical, reindex enterprise search data stream deprecat…

4c42363

…ions (#211881) ## Summary Close #211712 Related #211847

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations #211847

[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations #211847

markjhoy commented Feb 19, 2025

jloleysens left a comment

jloleysens Feb 20, 2025

markjhoy Feb 20, 2025

jloleysens Feb 20, 2025

markjhoy Feb 20, 2025

jloleysens Feb 20, 2025

jloleysens Feb 20, 2025

jloleysens left a comment

jloleysens Feb 20, 2025 •

edited

Loading

markjhoy Feb 20, 2025 •

edited

Loading

jloleysens left a comment

markjhoy commented Feb 20, 2025

markjhoy commented Feb 21, 2025

jloleysens left a comment

jloleysens Feb 24, 2025

jloleysens Feb 24, 2025

jloleysens Feb 24, 2025

elasticmachine commented Feb 24, 2025

markjhoy commented Feb 24, 2025

	expand_wildcards: ['all'],
	expand_wildcards: ['all' ,'hidden'],

		@@ -24,9 +30,10 @@ export async function getPreEightEnterpriseSearchIndices(
		expand_wildcards: ['all'],

	expand_wildcards: ['all'],
	expand_wildcards: ['all', 'hidden'],

[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations #211847

[8.x] Ensure Enterprise Search pre 8.x Datastream Deprecations #211847

Conversation

markjhoy commented Feb 19, 2025

Summary

Checklist

jloleysens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

jloleysens Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

markjhoy Feb 20, 2025 • edited Loading

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

markjhoy commented Feb 20, 2025

markjhoy commented Feb 21, 2025

jloleysens left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Feb 24, 2025

💚 Build Succeeded

Metrics [docs]

History

markjhoy commented Feb 24, 2025

💚 All backports created successfully

Questions ?

jloleysens Feb 20, 2025 •

edited

Loading

markjhoy Feb 20, 2025 •

edited

Loading