Dump installed objects for a given package #670

jsoriano · 2022-01-26T17:30:46Z

Add the elastic-package dump installed-objects to dump the objects installed for a package in Elasticsearch.

This command is intended to be used as an exploratory tool, to be able to analyze the resources associated to a given package.

For testing, a helper is added that starts a client for a mocked server with recorded responses.

To try it:

make build
./elastic-package stack up
eval "$(./elastic-package stack shellinit)"
./elastic-package dump installed-objects --package system

Fixes #600

jsoriano · 2022-01-26T17:36:21Z

So far, it exports the following objects when run for the system package:

output/component_templates/.fleet_component_template-1.json
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/ilm_policies/metrics.json
output/index_templates/logs-system.application.json
output/index_templates/logs-system.auth.json
output/index_templates/logs-system.security.json
output/index_templates/logs-system.syslog.json
output/index_templates/logs-system.system.json
output/index_templates/metrics-system.cpu.json
output/index_templates/metrics-system.diskio.json
output/index_templates/metrics-system.filesystem.json
output/index_templates/metrics-system.fsstat.json
output/index_templates/metrics-system.load.json
output/index_templates/metrics-system.memory.json
output/index_templates/metrics-system.network.json
output/index_templates/metrics-system.process.json
output/index_templates/metrics-system.process.summary.json
output/index_templates/metrics-system.socket_summary.json
output/index_templates/metrics-system.uptime.json
output/ingest_pipelines/logs-system.application-1.8.0.json
output/ingest_pipelines/logs-system.auth-1.8.0.json
output/ingest_pipelines/logs-system.security-1.8.0.json
output/ingest_pipelines/logs-system.syslog-1.8.0.json
output/ingest_pipelines/logs-system.system-1.8.0.json

elasticmachine · 2022-01-26T17:46:28Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Reason: null
Start Time: 2022-02-01T12:22:39.400+0000
Duration: 31 min 38 sec
Commit: 305deac

Test stats 🧪

Test	Results
Failed	0
Passed	512
Skipped	0
Total	512

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

jsoriano · 2022-01-26T18:47:23Z

Getting the objects starting from the data streams is not reliable, they may not exist if no data has been ingested. Refactored to start discovering from the index templates. For the apache package these objects are exported now:

output/component_templates/.fleet_component_template-1.json
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/component_templates/[email protected]
output/ilm_policies/logs.json
output/ilm_policies/metrics.json
output/index_templates/logs-apache.access.json
output/index_templates/logs-apache.error.json
output/index_templates/metrics-apache.status.json
output/ingest_pipelines/logs-apache.access-1.3.4.json
output/ingest_pipelines/logs-apache.error-1.3.4.json

jsoriano · 2022-01-28T18:10:18Z

Opening for review.

@DJRickyB @tomcallahan it'd be nice if you could give it a try.

You can find sample files exported by this new subcommand in https://github.com/elastic/elastic-package/tree/bbbb70a54495b8b32f42846797733ccdc74ed54d/internal/export/testdata/apache-export-all

mtojek

First review round, mostly focused on API.

mtojek · 2022-01-31T11:32:14Z

cmd/export.go

@@ -33,12 +38,25 @@ func setupExportCommand() *cobraext.Command {
 	exportDashboardCmd.Flags().StringSliceP(cobraext.DashboardIDsFlagName, "d", nil, cobraext.DashboardIDsFlagDescription)
 	exportDashboardCmd.Flags().Bool(cobraext.TLSSkipVerifyFlagName, false, cobraext.TLSSkipVerifyFlagDescription)

+	exportInstalledObjectsCmd := &cobra.Command{
+		Use:   "installed-objects",
+		Short: "Export installed Elasticsearch objects",


Just double-checking: there aren't any Kibana installed object, right?

The command only exports the installed Elasticsearch objects now, as they are the relevant ones for performance testing.

We could consider at least index patterns and dashboards as things to export too. I haven't thought a lot about what would we do if we also want to export them in the same fashion. One option is to just add it here and rephrase this. On the other hand we already have a command to export dashboards, and maybe we could have another one to export index patterns, or add a flag to export dashboards to export the used pattern too.

Same story, if this is a low-hanging fruit, then please consider it.

Not such a low hanging fruit as the source of data would be a different API, and not needed at the moment, I would leave it for a future improvement.

mtojek · 2022-01-31T11:35:20Z

cmd/export.go

+		RunE:  exportInstalledObjectsCmd,
+	}
+	exportInstalledObjectsCmd.Flags().StringP(cobraext.ExportPackageFlagName, "p", "", cobraext.ExportPackageFlagDescription)
+	exportInstalledObjectsCmd.MarkFlagRequired(cobraext.ExportPackageFlagName)


There will be inconsistency with the other export command that uses the current work dir to read the package context. Maybe this command should also follow that rule?

With the approach here you can export the objects without needing the source of the package, I think this will be useful for the current use case, and it actually doesn't need anything about the current package context.

If we want to make all export subcommands to be consistent on this I think I would prefer to add both options to both subcommands (i.e. both work with the package context and with the package flag) :)

If we want to make all export subcommands to be consistent on this I think I would prefer to add both options to both subcommands (i.e. both work with the package context and with the package flag) :)

Yes, I was looking at low hanging fruits :) I will leave you to make the decision.

I have added a test so this continues being a low hanging fruit in the future, but I am not taking it by now :)

cmd/export.go

internal/export/installedobjects.go

cmd/export.go

mtojek

LGTM implementation looks good, not sure if we arguments to discuss the API.

mtojek · 2022-01-31T15:11:54Z

cmd/export.go

+		RunE:  exportInstalledObjectsCmd,
+	}
+	exportInstalledObjectsCmd.Flags().StringP(cobraext.ExportPackageFlagName, "p", "", cobraext.ExportPackageFlagDescription)
+	exportInstalledObjectsCmd.MarkFlagRequired(cobraext.ExportPackageFlagName)


If we want to make all export subcommands to be consistent on this I think I would prefer to add both options to both subcommands (i.e. both work with the package context and with the package flag) :)

Yes, I was looking at low hanging fruits :) I will leave you to make the decision.

mtojek · 2022-01-31T15:14:16Z

cmd/export.go

@@ -33,12 +38,25 @@ func setupExportCommand() *cobraext.Command {
 	exportDashboardCmd.Flags().StringSliceP(cobraext.DashboardIDsFlagName, "d", nil, cobraext.DashboardIDsFlagDescription)
 	exportDashboardCmd.Flags().Bool(cobraext.TLSSkipVerifyFlagName, false, cobraext.TLSSkipVerifyFlagDescription)

+	exportInstalledObjectsCmd := &cobra.Command{
+		Use:   "installed-objects",
+		Short: "Export installed Elasticsearch objects",


Same story, if this is a low-hanging fruit, then please consider it.

cmd/export.go

internal/export/installedobjects.go

jsoriano · 2022-01-31T19:45:53Z

@mtojek moved to a new dump command, let me know what you think.

I would be tempted to make this the only action of this dump subcommand, but I wonder if we may have more things to dump in the future, so I would play safe here and leave it as a subcommand under dump.

mtojek · 2022-02-01T08:20:38Z

internal/common/helpers.go

@@ -12,3 +12,13 @@ func TrimStringSlice(slice []string) {
 		slice[iterator] = strings.TrimSpace(item)
 	}
 }
+
+// StringSliceContains checks if the slice contains the given string.
+func StringSliceContains(slice []string, s string) bool {


nit: I'm wondering to how many places in the source code it can applied :)

We can look for them in a follow up, and perhaps if we wait to https://tip.golang.org/doc/go1.18#generics we can make it generic and apply it even more extensively :)

mtojek · 2022-02-01T08:26:03Z

cmd/dump.go

+		return errors.Wrap(err, "dump failed")
+	}
+	if n == 0 {
+		cmd.Printf("No objects were dumped for package %s, is it installed?", packageName)


nit: do you think we can check if the package is installed apriori? I hope there is a fleet API to make sure it is.

Maybe, but this way this tool can also discover letfovers after uninstalling a package :) Also adding a check against the fleet API adds an additional dependency. As the command is now, it only needs access to Elasticsearch.

Maybe, but this way this tool can also discover letfovers after uninstalling a package

It's a discussion whether it's a bug or feature :) We can keep it as is for now.

mtojek · 2022-02-01T08:28:03Z

cmd/dump.go

+		Long:  dumpInstalledObjectsLongDescription,
+		RunE:  dumpInstalledObjectsCmd,
+	}
+	dumpInstalledObjectsCmd.Flags().StringP(cobraext.DumpPackageFlagName, "p", "", cobraext.DumpPackageFlagDescription)


Shouldn't -p and -o be a command level instead of subcommand?

mtojek · 2022-02-01T08:28:40Z

cmd/dump.go

+		Long:  dumpInstalledObjectsLongDescription,
+		RunE:  dumpInstalledObjectsCmd,
+	}
+	dumpInstalledObjectsCmd.Flags().StringP(cobraext.DumpPackageFlagName, "p", "", cobraext.DumpPackageFlagDescription)


I hope it isn't possible to install multiple package versions at once :)

Well, if some day this is possible, I guess that the worst thing that can happen with current implementation is that the objects of all installed versions will be collected, what I would find fine. We can add an optional version filter if this day arrives.

mtojek · 2022-02-01T08:36:16Z

cmd/dump.go

+	}
+	dumpInstalledObjectsCmd.Flags().StringP(cobraext.DumpPackageFlagName, "p", "", cobraext.DumpPackageFlagDescription)
+	dumpInstalledObjectsCmd.MarkFlagRequired(cobraext.DumpPackageFlagName)
+	dumpInstalledObjectsCmd.Flags().StringP(cobraext.DumpOutputFlagName, "o", "output", cobraext.DumpOutputFlagDescription)


Let's discuss possible dump use cases:

elastic-package dump installed-objects -p hello-world elastic-package dump assets -p hello-world elastic-package dump assets -p hello-world elastic-package dump indices -p hello-world elastic-package dump sample-event -p hello-world

WDYT about storing data in a structured way, for example output-dir can be replaced with package-dump, which may contain installed-objects, indices, etc.

Changing the default from output to package-dump sgtm, but I think I would keep the internal structure flat. If we dump additional resources as indices or sample events at some moment we can store them under package-dump/events and package-dump/indices and so on.

…default output directory

jsoriano · 2022-02-01T11:40:48Z

I have found a couple of places where there can be nested pipelines, adding some logic to collect them too...

mtojek

LGTM

mtojek · 2022-02-01T12:38:10Z

cmd/dump.go

+		return errors.Wrap(err, "dump failed")
+	}
+	if n == 0 {
+		cmd.Printf("No objects were dumped for package %s, is it installed?", packageName)


Maybe, but this way this tool can also discover letfovers after uninstalling a package

It's a discussion whether it's a bug or feature :) We can keep it as is for now.

jsoriano · 2022-02-08T19:49:39Z

@DJRickyB @tomcallahan did you have the chance of trying this new command to dump ES objects related to packages?

DJRickyB · 2022-02-11T13:53:27Z

Hello! I'll get into this today

DJRickyB · 2022-02-11T16:11:36Z

This works great, thank you very much. Interesting, the departure from the existing export command to this dump command, given the similarity of function but as we're only interested in the ES objects anyway, this is great

jsoriano · 2022-02-11T16:44:06Z

This works great, thank you very much. Interesting, the departure from the existing export command to this dump command, given the similarity of function but as we're only interested in the ES objects anyway, this is great

@DJRickyB thanks for trying it!

Yep, both commands are similar, but we decided to leave the export command as a tool for package developers to export resources in a way that can be included directly in packages, for example the export dashboards subcommand does some transformations and stores the dashboards and its elements in the proper paths inside the current package. These commands are expected to be run in the working directory of a package.

On the other hand dump commands will be used to dump resources as they are, without any transformation, or without expectations on the structure of the directory where they are dumped.

DJRickyB · 2022-02-11T16:54:01Z

Thank you for the context, makes total sense

jsoriano added the Team:Ecosystem Label for the Packages Ecosystem team label Jan 26, 2022

jsoriano self-assigned this Jan 26, 2022

jsoriano force-pushed the export-installed-assets branch from ff60144 to b01b737 Compare January 26, 2022 18:44

jsoriano added 10 commits January 27, 2022 21:29

Add export installed-assets subcommand

2b6e42c

Get data streams for package

231a42b

Export ILM policies

cb9be10

Export index templates

b2a50e9

Export logs-related objects too

6a66435

Export ingest pipelines

c4aacc2

Fix export directory for ingest pipelines

bf4e414

Use index templates instead of data streams to discover objects

9565d17

Implement insecure TLS for elasticsearch client

d0432e4

Refactor exporting

38d81a3

jsoriano force-pushed the export-installed-assets branch from d805022 to 38d81a3 Compare January 27, 2022 20:30

jsoriano added 4 commits January 28, 2022 16:18

Handle ILM and pipelines responses with multiple elements

526e687

Centralize JSON formatting in the output

ba3d089

Add godocs

b6c638d

Add test for objects export

bbbb70a

jsoriano marked this pull request as ready for review January 28, 2022 18:04

jsoriano requested a review from a team January 28, 2022 18:04

Merge remote-tracking branch 'origin/main' into export-installed-assets

6a913ea

jsoriano added 2 commits January 28, 2022 19:19

Improve log message

31d65a5

Linting

1018a4a

mtojek reviewed Jan 31, 2022

View reviewed changes

mtojek approved these changes Jan 31, 2022

View reviewed changes

Move StringSliceContains to common helpers

4ff246c

jsoriano added 5 commits January 31, 2022 18:46

Refactor options

4f37572

Add test for exporters of each kind of object

2aab20e

Move subcommand to new dump command

c28f7ce

Update readme

8f6815c

Merge remote-tracking branch 'origin/main' into export-installed-assets

2f8c9db

jsoriano requested a review from mtojek January 31, 2022 19:45

mtojek reviewed Feb 1, 2022

View reviewed changes

Make package and output flags persistent in the dump command, change …

311b58f

…default output directory

jsoriano requested a review from mtojek February 1, 2022 10:21

jsoriano added 4 commits February 1, 2022 12:49

Collect pipelines invoked as processors

64b1726

Dump final pipelines too

c77adc5

Fix godocs

0f2e378

ILM policies can also be defined in index templates

305deac

mtojek approved these changes Feb 1, 2022

View reviewed changes

jsoriano merged commit c8541c5 into elastic:main Feb 1, 2022

jsoriano deleted the export-installed-assets branch February 1, 2022 13:33

jsoriano changed the title ~~Export installed objects for a given package~~ Dump installed objects for a given package Feb 1, 2022

jsoriano mentioned this pull request Feb 3, 2022

Remove wildcards from file names #685

Merged

nicpenning mentioned this pull request Mar 13, 2024

[Feature] elastic-package export pipelines #1722

Open

Dump installed objects for a given package #670

Dump installed objects for a given package #670

Conversation

jsoriano commented Jan 26, 2022 • edited Loading

jsoriano commented Jan 26, 2022 • edited Loading

elasticmachine commented Jan 26, 2022 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

jsoriano commented Jan 26, 2022 • edited Loading

jsoriano commented Jan 28, 2022

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Jan 31, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Feb 1, 2022

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano commented Feb 8, 2022

DJRickyB commented Feb 11, 2022

DJRickyB commented Feb 11, 2022

jsoriano commented Feb 11, 2022

DJRickyB commented Feb 11, 2022

jsoriano commented Jan 26, 2022 •

edited

Loading

jsoriano commented Jan 26, 2022 •

edited

Loading

elasticmachine commented Jan 26, 2022 •

edited

Loading

jsoriano commented Jan 26, 2022 •

edited

Loading