[Meta] Breaking saved object type conversions in 8.0

[jportner]

Overview

In #27004, we implemented a feature to allow users to share saved objects across multiple spaces. As part of that effort, we had to change how saved objects are serialized to raw Elasticsearch documents. We implemented a conversion process to change saved objects to the new format, but this will regenerate IDs of existing saved objects to ensure that they are unique across all spaces.

Regenerating IDs of saved objects is a major breaking change. This will affect objects that reference each other, and it will also affect external references to saved objects (for example, a bookmarked URL to a dashboard). It was decided that the best path forward is to identify all existing "namespace-isolated" object types that need to become shareable, and convert them all at once in the 8.0 release. This process is called making the object types "share-capable".

As such, plugin authors need to take additional steps:

1. Before the 8.0 release (during 7.x): Make sure that existing saved object types are using references correctly to link to other objects. If you are linking to other objects and you aren't using the root-level references field, it will break in 8.0.
2. In the 8.0 release: Change client-side code to use the newly available SavedObjectsClient.resolve API (instead of SavedObjectsClient.get), and handle the new different outcomes (exactMatch, aliasMatch, conflict).

Important note: This conversion will not make any object types fully shareable! From an end-user's perspective, these object types will continue to function the same way as they did before, each one will still be isolated to a single space. Plugin authors will also need to plan additional steps in the 8.x timeframe to make their object types fully shareable. This may entail additional UI elements and user flows as necessitated by their individual use cases.

Affected objects

Each of the objects in Kibana below are currently "isolated" (registered w/ namespaceType: 'single'). Each should be reviewed by plugin authors to determine if they are using references correctly, and if they should be converted to become "share-capable" (registered w/ namespaceType: 'multiple-isolated') in the 8.0 release.

Update: object types are crossed out if we have deemed they should not be converted.

Saved Object Type	Code Owner	GH Issue(s)	Status
action, action_task_params, alert	@elastic/kibana-alerting-services	#100067	✅
graph-workspace, lens, lens-ui-telemetry, search, timelion-sheet, visualization	@elastic/kibana-app	#105808 #105809 #105810 #105812	✅
index-pattern, query, search-session, url	@elastic/kibana-app-services	#106900	✅
config, tag	@elastic/kibana-core	#102870	✅
map	@elastic/kibana-gis	#101177	✅
dashboard, canvas-element, canvas-workpad	@elastic/kibana-presentation	#105967 #105970	✅
infrastructure-ui-source, metrics-explorer-view, inventory-view	@elastic/logs-metrics-ui	#101966	✅
security-solution-signals-migration, siem-detection-engine-rule-actions, siem-detection-engine-rule-status	@elastic/security-detections-response	#105819 #114548	✅
cases, cases-comments, cases-configure, cases-connector-mappings, cases-sub-case, cases-user-actions, siem-ui-timeline, siem-ui-timeline-note, siem-ui-timeline-pinned-event	@elastic/security-threat-hunting	#105677	✅
exception-list	@elastic/security-detections-response & @elastic/security-threat-hunting	#105819 #114548	✅
uptime-dynamic-settings	@elastic/uptime	N/A	✅

Type registration traces -- click to expand

Saved Object Type	Trace
config	UiSettingsService.setup (src/core/server/ui_settings/ui_settings_service.ts:53:18)
url	SharePlugin.setup (src/plugins/share/server/plugin.ts:21:23)
query	QueryService.setup (src/plugins/data/server/query/query_service.ts:14:23)
index-pattern	IndexPatternsServiceProvider.setup (src/plugins/data/server/index_patterns/index_patterns_service.ts:84:23)
visualization	VisualizationsPlugin.setup (src/plugins/visualizations/server/plugin.ts:41:23)
timelion-sheet	TimelionPlugin.setup (src/plugins/timelion/server/plugin.ts:30:23)
graph-workspace	GraphPlugin.setup (x-pack/plugins/graph/server/plugin.ts:38:23)
search	DiscoverServerPlugin.setup (src/plugins/discover/server/plugin.ts:18:23)
dashboard	DashboardPlugin.setup (src/plugins/dashboard/server/plugin.ts:43:23)
tag	SavedObjectTaggingPlugin.setup (x-pack/plugins/saved_objects_tagging/server/plugin.ts:43:18)
lens	setupSavedObjects (x-pack/plugins/lens/server/saved_objects.ts:13:21)
lens-ui-telemetry	setupSavedObjects (x-pack/plugins/lens/server/saved_objects.ts:51:21)
canvas-element	CanvasPlugin.setup (x-pack/plugins/canvas/server/plugin.ts:52:28)
canvas-workpad	CanvasPlugin.setup (x-pack/plugins/canvas/server/plugin.ts:53:28)
exception-list	initSavedObjects (x-pack/plugins/lists/server/saved_objects/init_saved_objects.ts:14:16)
map	MapsPlugin.setup (x-pack/plugins/maps/server/plugin.ts:214:23)
search-session	EnhancedDataServerPlugin.setup (x-pack/plugins/data_enhanced/server/plugin.ts:32:23)
action	setupSavedObjects (x-pack/plugins/actions/server/saved_objects/index.ts:30:16)
action_task_params	setupSavedObjects (x-pack/plugins/actions/server/saved_objects/index.ts:66:16)
alert	setupSavedObjects (x-pack/plugins/alerting/server/saved_objects/index.ts:46:16)
infrastructure-ui-source	InfraServerPlugin.setup (x-pack/plugins/infra/server/plugin.ts:103:23)
metrics-explorer-view	InfraServerPlugin.setup (x-pack/plugins/infra/server/plugin.ts:104:23)
inventory-view	InfraServerPlugin.setup (x-pack/plugins/infra/server/plugin.ts:105:23)
cases-comments	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts:75:23)
cases-configure	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts:76:23)
cases-connector-mappings	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts:77:23)
cases	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts:78:23)
cases-user-actions	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts:79:23)
cases-sub-case (added 2021-09-30, this was hidden behind a feature flag)	CasePlugin.setup (x-pack/plugins/cases/server/plugin.ts)
siem-ui-timeline-note	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
siem-ui-timeline-pinned-event	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
siem-detection-engine-rule-actions	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
siem-detection-engine-rule-status	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
siem-ui-timeline	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
security-solution-signals-migration	forEach (x-pack/plugins/security_solution/server/saved_objects.ts:37:40)
uptime-dynamic-settings (added 2021-08-23)	Plugin.setup (x-pack/plugins/uptime/server/plugin.ts:61:23)

Important considerations

If an object type should be converted, then all object types that it can reference should be converted too. For example, dashboards have references to visualizations, which have references to index-patterns -- all of these object types must be converted to become share-capable.

Developer guide

See the Sharing Saved Objects developer guide for step-by-step instructions on how to prepare your plugin for the 8.0 release. You will need to take 3-5 specific steps (depending on how your plugin uses saved objects).

[elasticmachine]

Pinging @elastic/kibana-security (Team:Security)

[pgayvallet]

Change client-side code to use the newly available SavedObjectsClient.resolve API (instead of SavedObjectsClient.get

Btw, is there an equivalent for bulkGet?

[jportner]

Btw, is there an equivalent for bulkGet?

Good question. There is currently no equivalent for bulkGet. The thinking goes that we likely only need to use resolve when we are loading a page (where a user might have, for example, bookmarked a Dashboard with the old URL before its ID was regenerated). If we identify a use case for bulkResolve we can certainly add it, but we didn't want to build an API that nobody needs.

[oatkiller]

Hi all. I'm hoping to learn about the namespaceType property. I don't see it documented in the Saved Object service docs. Could anyone suggest a reference? Thanks

[legrego]

@oatkiller the description of the different namespaceType values are described here. If you mentally translate namespace to space (as in a Kibana Space), then it should help clarify things. Happy to help answer any other questions you have!

kibana/src/core/server/saved_objects/types.ts

Lines 237 to 249 in 106afd4

	/**
	* The namespace type dictates how a saved object can be interacted in relation to namespaces. Each type is mutually exclusive:
	* * single (default): This type of saved object is namespace-isolated, e.g., it exists in only one namespace.
	* * multiple: This type of saved object is shareable, e.g., it can exist in one or more namespaces.
	* * multiple-isolated: This type of saved object is namespace-isolated, e.g., it exists in only one namespace, but object IDs must be
	* unique across all namespaces. This is intended to be an intermediate step when objects with a "single" namespace type are being
	* converted to a "multiple" namespace type. In other words, objects with a "multiple-isolated" namespace type will be *share-capable*,
	* but will not actually be shareable until the namespace type is changed to "multiple".
	* * agnostic: This type of saved object is global.
	*
	* @public
	*/
	export type SavedObjectsNamespaceType = 'single' | 'multiple' | 'multiple-isolated' | 'agnostic';

[kevinlog]

@paul-tavares @dasansol92 FYI on the above - for exception-lists we should look at trusted apps and event filters, anything else?

[pgayvallet]

If an object type should be converted, then all object types that it can reference should be converted too

@jportner In that case, I guess the conversion of the tag type will be mandatory, as it's potentially referenced by any kind of other SO? Or did you treat when differently when you implemented sharing phase 3, I don't remember exactly?

Regarding config, I don't see why we'll want to have it shareable.

[simianhacker]

@weltenwort @jasonrhodes I looked at the Metrics UI and the Source configuration and it appears we are ready. The Metrics UI saved objects don't have references. The source configuration does for the Logs UI index patterns and they appear to be using the references properly.

[weltenwort]

The Metrics UI saved objects don't have references.

@simianhacker Are the inventoryDefaultView and metricsExplorerDefaultView properties on the source config document references to other saved objects?

The source configuration does for the Logs UI index patterns and they appear to be using the references properly.

indeed

[jportner]

@jportner In that case, I guess the conversion of the tag type will be mandatory, as it's potentially referenced by any kind of other SO? Or did you treat when differently when you implemented sharing phase 3, I don't remember exactly?

Tag is a bit of a special case. Recently (as of last week) I met with Product stakeholders and we decided to exclude tags from sharing for the time being. The biggest reason for this is that we don't currently have a way to reconcile/merge similar tags (two different tags with the same label). So I think we can safely convert tags to namespaceType: multiple-isolated' in 8.0 and leave them like that for the foreseeable future.

Regarding config, I don't see why we'll want to have it shareable.

👍

[simianhacker]

@weltenwort Yes... good catch!

(tracking #101966)

[pgayvallet]

I met with Product stakeholders and we decided to exclude tags from sharing for the time being. The biggest reason for this is that we don't currently have a way to reconcile/merge similar tags (two different tags with the same label). So I think we can safely convert tags to namespaceType: multiple-isolated' in 8.0 and leave them like that for the foreseeable future.

I'm not sure to understand then? What will be the behavior when sharing an object that references tags? What's the point in converting them to multiple-isolated in that case?

[jportner]

I'm not sure to understand then? What will be the behavior when sharing an object that references tags? What's the point in converting them to multiple-isolated in that case?

When sharing an object that references tags: tags will be skipped.
However,

1. We are pretty sure that we eventually want referenced tags to be included in shares, when there is some workflow to reconcile/merge "identical" tags
2. We can still allow users to share tags directly, even before that workflow is available

Any object that will be or may be eventually shareable needs to be converted in 8.0.

[tsullivan]

Regenerating IDs of saved objects is a major breaking change. This will affect objects that reference each other, and it will also affect external references to saved objects (for example, a bookmarked URL to a dashboard).

@jportner Is there any plan to keep external references to saved objects working? Otherwise, this seems to indicate that any bookmarked URL to a dashboard will just be broken, correct?

When a user sets up automated reports, they copy a POST URL and paste it into their own Watcher definition or script that can be run with cron. The POST URL often contains references to objects by their ID. Is there any way to save the user from having to manually fix all of their automated reports?

[jportner]

@jportner Is there any plan to keep external references to saved objects working? Otherwise, this seems to indicate that any bookmarked URL to a dashboard will just be broken, correct?

Each plugin owner needs to update their server-side code to resolve() saved object IDs that are passed in via URL. This ensures that an old saved object ID can still be used to find the correct saved object. I added a basic example of this in afc3490 (PR #95958).
Client-side code should also be updated to handle the resolve() outcome (and potentially redirect the user to the new URL, if the saved object ID has changed).

When a user sets up automated reports, they copy a POST URL and paste it into their own Watcher definition or script that can be run with cron. The POST URL often contains references to objects by their ID. Is there any way to save the user from having to manually fix all of their automated reports?

There should be no intervention needed for Reporting directly; Dashboard, Discover, etc. need to be updated so that these URLs won't actually break for end-users.

This week I'll be working on adding more specific dev guidance in the description of this issue 👍

[jportner]

It looks like saved object consumers have accomplished everything we set out to in this issue.

Some additional work:

• 8.0 blocking bugs:
  • Prevent creating legacy URL conflicts #113335
  • Disable aliases when an object is deleted or unshared #116235
  • Sample datasets with objects namespaceType: multiple or multiple-isolated fail to install on multiple namespaces #116261
  • Handle ID changes to Saved Objects installed by Fleet #108959
• 8.x follow-on work:
  • Investigate risk of performance regression from share-capable saved object types #113743
  • Differentiate load-by-reference for embeddables #111191
  • Improve data plugin handling of resolve #111196

[pgayvallet]

Some additional work:

@jportner do we have a meta issue listing all these points?

[jportner]

@jportner do we have a meta issue listing all these points?

Since these are quite different individual issues, I've added the Feature:Security/Sharing Saved Objects label instead of creating a new meta-issue.