[EPM] /packages/{package} endpoint to support upgrades #63629

neptunian · 2020-04-15T18:02:00Z

Implements updating to a compatible package working by changing the order in which items are installed and now, updated. this can be found here #59910

installs/updates the index templates after pipeline creation
differentiates between install and reinstall to avoid deleting the pipelines which need to be deleted after successful updating of index templates and current write indices. since the version is the same if its a "reinstall", don't delete the pipeline (it's already been overwritten during creation time).
checks if the package is already installed (an update) and deletes all kibana assets before installing again. this in case the new package has deleted some assets so overwriting existing ones is not sufficient
removes functionality that merges the current saved objects "installed" fields on each epm package with the new installation and now totally overwrites the installed attribute (some assets may not exist anymore so it needs to get wiped out)
adds functionality to update the current write indices. query for every index associated with this package
will "reinstall" or install the same version of a package if it matches the already installed one

What this PR does not do

if a package fails to install for whatever reason, there needs to be a "rollback" as the install will be partial and they could be in a state where they have different assets of both versions. @ruflin said that for now, in a blog post, they’ll have to uninstall the old package and then try to install again (the new version doesn't get written to the saved object unless it was successful)
logging of errors so we can see what went wrong as compatible packages should be able to update
block API from attempting to upgrade or downgrade major versions (not allowed right now)
handle failure of mappings update on the current write index. if the mapping fails for the current write index that will most likely be because they are incompatible, such as trying to change a data type on an existing field (not allowed). typically, this is the point where we'd trigger a rollover to a new index which could inherit the new mapping types from the already created index template. since we can't do that right now (waiting for datastreams), they cannot update, and this will cause an error in the update process.

Test
I think we need a test package to test more uses cases and know what to look for, but system can be used at the moment.

POST http://localhost:5603/ssg/api/ingest_manager/epm/packages/system-0.0.1
POST http://localhost:5603/ssg/api/ingest_manager/epm/packages/system-0.9.0
saved objects package version is now at 0.9.0
http://localhost:5601/ssg/api/saved_objects/_find?fields=title&fields=type&fields=version&per_page=10000&type=epm-package
Pipelines are built and referenced in the index template in 0.0.1 and deleted in 0.9.0
Before updating
GET _template/logs-system.auth

After Updating (because pipeline does not exist in 0.9.0)
GET _template/logs-system.auth

This is also reflected in the saved objects for epm package where the ingest pipelines are no longer part of the installed assets

elasticmachine · 2020-04-15T19:15:45Z

Pinging @elastic/ingest-management (Feature:EPM)

jfsiii

I made some comments on names and exported types that I'd like to see , but good to ship as-is from me.

jfsiii · 2020-04-20T21:08:14Z

x-pack/plugins/ingest_manager/common/types/models/epm.ts

@@ -256,10 +256,32 @@ export enum DefaultPackages {
  endpoint = 'endpoint',
 }

+export interface Mappings {


Could we make these more specific? IndexTemplateMapping(s), perhaps?

jfsiii · 2020-04-20T21:10:05Z

x-pack/plugins/ingest_manager/common/types/models/epm.ts

+  indexTemplate: IndexTemplate;
+}
+
+export interface CurrentIndex {


Can you move this to x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts? I think that's the only place it's used and I don't initially see how CurrentIndex is a top-level concept in EPM.

jfsiii · 2020-04-20T21:10:41Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts

@@ -235,9 +241,10 @@ function getBaseTemplate(type: string, templateName: string, mappings: Mappings)
    },
    mappings: {
      // To be filled with interesting information about this specific index
+      /*


Is this intentional? If so, can we delete this and the comment above?

+1 on removing this for now. We can add meta data again if we need it later.

ruflin

Did not check the code in detail but I'm good with getting this in based on your description.

I'm thinking it will simplify things if in the future we can

Install all new assets
Remove old assets which are not needed anymore

I would hope this also helps failure recovery but will require us to keep old and new state. Lets iterate on it.

@neptunian I think we should also have some docs on the exact behaviour on what happens on upgrade.

ruflin · 2020-04-21T11:51:51Z

x-pack/plugins/ingest_manager/server/services/epm/elasticsearch/template/template.ts

@@ -235,9 +241,10 @@ function getBaseTemplate(type: string, templateName: string, mappings: Mappings)
    },
    mappings: {
      // To be filled with interesting information about this specific index
+      /*


+1 on removing this for now. We can add meta data again if we need it later.

neptunian · 2020-04-21T16:39:25Z

@elasticmachine merge upstream

jfsiii · 2020-04-21T16:58:55Z

x-pack/plugins/ingest_manager/common/types/models/epm.ts

-  type: string;
-  properties?: Record<string, Mapping>;
+export interface IndexTemplateMappings {
+  properties: any;


Is this really any? e.g. number, string, etc?

I actually just renamed Mappings to IndexTemplateMappings that @skh created. Trying to type all the mappings and settings is going to be pretty time consuming.

kibanamachine · 2020-04-21T18:35:14Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 0e5624b

History

💚 Build #42155 succeeded aed6968
💚 Build #41989 succeeded 867d62b
💚 Build #41850 succeeded fc41fa7
💚 Build #41052 succeeded 33d457d667792a3f00b6a4c5526d25c66b23a5dd

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* install template after pipeline creation * return installed pkg if this pkg version is already installed * remove pipelines after templates are updated * remove kibana saved objects assets before installing * update current write indices * add back removal of merging previous references lost in rebase * improve some typing names, consolidate, fix bad merges * update query to use aggregate on _index Co-authored-by: Elastic Machine <[email protected]>

ruflin · 2020-04-22T08:29:10Z

@neptunian Could you make sure we have a follow up issue where we can track the open issues around upgrade?

* master: (29 commits) [Dashboard] Deangularize navbar, attempt nr. 2 (elastic#61611) refactor action filter creation utils (elastic#62969) Refresh index pattern list before redirecting (elastic#63329) [APM]fixing custom link unit tests (elastic#64045) [Ingest] EPM & Fleet are enabled when Ingest is enabled (elastic#64103) [Alerting] Fixed bug with no possibility to edit the index name after adding (elastic#64033) [Maps] Map settings: min and max zoom (elastic#63714) [kbn-storybook] Use raw loader for text files (elastic#64108) [EPM] /packages/{package} endpoint to support upgrades (elastic#63629) [SIEM] New Platform Saved Objects Registration (elastic#64029) [Endpoint] Hook to handle events needing navigation via Router (elastic#63863) Fixed small issue in clone functionality (elastic#64085) [Endpoint]EMT-146: use ingest agent for status info (elastic#63921) [SIEM] Server NP Followup (elastic#64010) Register uiSettings on New Platform (elastic#64015) [Reporting] Integration polling config with client code (elastic#63754) [Docs]7.7 SIEM doc updates (elastic#63951) [SIEM] [Cases] Tags suggestions (elastic#63878) Include datasource UUID in agent config yaml, adjust overflow height of yaml view (elastic#64027) [DOCS] Add file size setting for Data Visualizer (elastic#64006) ...

* install template after pipeline creation * return installed pkg if this pkg version is already installed * remove pipelines after templates are updated * remove kibana saved objects assets before installing * update current write indices * add back removal of merging previous references lost in rebase * improve some typing names, consolidate, fix bad merges * update query to use aggregate on _index Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

…ana into task-manager/cancel-logging * 'task-manager/cancel-logging' of github.com:gmmorris/kibana: (28 commits) [Dashboard] Deangularize navbar, attempt nr. 2 (elastic#61611) refactor action filter creation utils (elastic#62969) Refresh index pattern list before redirecting (elastic#63329) [APM]fixing custom link unit tests (elastic#64045) [Ingest] EPM & Fleet are enabled when Ingest is enabled (elastic#64103) [Alerting] Fixed bug with no possibility to edit the index name after adding (elastic#64033) [Maps] Map settings: min and max zoom (elastic#63714) [kbn-storybook] Use raw loader for text files (elastic#64108) [EPM] /packages/{package} endpoint to support upgrades (elastic#63629) [SIEM] New Platform Saved Objects Registration (elastic#64029) [Endpoint] Hook to handle events needing navigation via Router (elastic#63863) Fixed small issue in clone functionality (elastic#64085) [Endpoint]EMT-146: use ingest agent for status info (elastic#63921) [SIEM] Server NP Followup (elastic#64010) Register uiSettings on New Platform (elastic#64015) [Reporting] Integration polling config with client code (elastic#63754) [Docs]7.7 SIEM doc updates (elastic#63951) [SIEM] [Cases] Tags suggestions (elastic#63878) Include datasource UUID in agent config yaml, adjust overflow height of yaml view (elastic#64027) [DOCS] Add file size setting for Data Visualizer (elastic#64006) ...

neptunian added 1sp Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project and removed 1sp labels Apr 15, 2020

neptunian added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 15, 2020

neptunian added 5 commits April 20, 2020 08:48

install template after pipeline creation

80fabe6

return installed pkg if this pkg version is already installed

4f5079e

remove pipelines after templates are updated

263ac83

remove kibana saved objects assets before installing

c8c03bb

update current write indices

fc41fa7

neptunian force-pushed the 59910-upgrade-packages-install-endpoint branch from 33d457d to fc41fa7 Compare April 20, 2020 13:14

neptunian requested a review from ruflin April 20, 2020 14:38

neptunian marked this pull request as ready for review April 20, 2020 15:42

neptunian requested a review from a team April 20, 2020 15:42

neptunian added v8.0.0 v7.8.0 release_note:skip Skip the PR/issue when compiling release notes labels Apr 20, 2020

add back removal of merging previous references lost in rebase

867d62b

jfsiii approved these changes Apr 20, 2020

View reviewed changes

ruflin approved these changes Apr 21, 2020

View reviewed changes

neptunian added 2 commits April 21, 2020 09:03

improve some typing names, consolidate, fix bad merges

aed6968

update query to use aggregate on _index

03f8f54

Merge branch 'master' into 59910-upgrade-packages-install-endpoint

0e5624b

jfsiii reviewed Apr 21, 2020

View reviewed changes

neptunian merged commit adc9b0d into elastic:master Apr 21, 2020

neptunian mentioned this pull request Apr 21, 2020

[7.x] [EPM] /packages/{package} endpoint to support upgrades (#63629) #64117

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[EPM] /packages/{package} endpoint to support upgrades #63629

[EPM] /packages/{package} endpoint to support upgrades #63629

neptunian commented Apr 15, 2020 •

edited

Loading

elasticmachine commented Apr 15, 2020

jfsiii left a comment

jfsiii Apr 20, 2020

jfsiii Apr 20, 2020

jfsiii Apr 20, 2020

ruflin Apr 21, 2020

ruflin left a comment

ruflin Apr 21, 2020

neptunian commented Apr 21, 2020

jfsiii Apr 21, 2020

neptunian Apr 21, 2020

kibanamachine commented Apr 21, 2020

ruflin commented Apr 22, 2020

[EPM] /packages/{package} endpoint to support upgrades #63629

[EPM] /packages/{package} endpoint to support upgrades #63629

Conversation

neptunian commented Apr 15, 2020 • edited Loading

elasticmachine commented Apr 15, 2020

jfsiii left a comment

Choose a reason for hiding this comment

jfsiii Apr 20, 2020

Choose a reason for hiding this comment

jfsiii Apr 20, 2020

Choose a reason for hiding this comment

jfsiii Apr 20, 2020

Choose a reason for hiding this comment

ruflin Apr 21, 2020

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

ruflin Apr 21, 2020

Choose a reason for hiding this comment

neptunian commented Apr 21, 2020

jfsiii Apr 21, 2020

Choose a reason for hiding this comment

neptunian Apr 21, 2020

Choose a reason for hiding this comment

kibanamachine commented Apr 21, 2020

💚 Build Succeeded

History

ruflin commented Apr 22, 2020

neptunian commented Apr 15, 2020 •

edited

Loading