[Security Solution] Rule filters: saved object migrations

[banderror]

Epic: #138606
Related to: #147438

Summary

Currently, query filters stored in rules of selected types are typed as an array of t.unknown values which allows a user to store anything in there via the API.

kibana/x-pack/plugins/security_solution/common/detection_engine/rule_schema/model/common_attributes/misc_attributes.ts

Lines 100 to 105 in 47ad5ed

	/**
	* TODO: Right now the filters is an "unknown", when it could more than likely
	* become the actual ESFilter as a type.
	*/
	export type RuleFilterArray = t.TypeOf<typeof RuleFilterArray>; // Filters are not easily type-able yet
	export const RuleFilterArray = t.array(t.unknown); // Filters are not easily type-able yet

For example, for the Custom Query rules, filters are defined here:

kibana/x-pack/plugins/security_solution/common/detection_engine/rule_schema/model/rule_schemas.ts

Line 303 in 47ad5ed

filters: RuleFilterArray,

kibana/x-pack/plugins/security_solution/server/lib/detection_engine/rule_schema/model/rule_schemas.ts

Line 175 in 47ad5ed

filters: t.union([RuleFilterArray, t.undefined]),

In the UI, filter objects are created by UI components we don't own. These components evolve over time and can change (see example PR), sometimes the model of the filters can change in a backward-incompatible way as well.

Since we store filters in detection rules but don't migrate them, there's a risk that a breaking change in the filters model can break existing rules that users already created, when they upgrade kibana to a new version. Both the UI and the rule executors could be affected by that.

We need to make sure that we migrate filters in rules every time their model changes.

To do

This has already been solved for Lens saved objects (ticket, PR) and we need to replicate this for rules in the Alerting Framework.

Lens is an object that contains filters in the attributes.state.filters field (just like rules contain them in the attributes.alert.params.filters field). It is migrated here given a map of existing migrations for filters:

kibana/x-pack/plugins/lens/server/migrations/common_migrations.ts

Lines 341 to 353 in 47ad5ed

	/**
	* This creates a migration map that applies filter migrations to Lens visualizations
	*/
	export const getLensFilterMigrations = (
	filterMigrations: MigrateFunctionsObject
	): MigrateFunctionsObject =>
	mapValues(filterMigrations, (migrate) => (lensDoc: { attributes: LensDocShape }) => ({
	...lensDoc,
	attributes: {
	...lensDoc.attributes,
	state: { ...lensDoc.attributes.state, filters: migrate(lensDoc.attributes.state.filters) },
	},
	}));

The map of filter migrations is exposed from the data plugin:

kibana/x-pack/plugins/lens/server/plugin.tsx

Lines 63 to 65 in 47ad5ed

	const getFilterMigrations = plugins.data.query.filterManager.getAllMigrations.bind(
	plugins.data.query.filterManager
	);

Finally, own lens migrations are merged with external filter migrations here:

kibana/x-pack/plugins/lens/server/migrations/saved_object_migrations.ts

Lines 565 to 605 in 47ad5ed

	const lensMigrations: SavedObjectMigrationMap = {
	'7.7.0': removeInvalidAccessors,
	// The order of these migrations matter, since the timefield migration relies on the aggConfigs
	// sitting directly on the esaggs as an argument and not a nested function (which lens_auto_date was).
	'7.8.0': (doc, context) => addTimeFieldToEsaggs(removeLensAutoDate(doc, context), context),
	'7.10.0': extractReferences,
	'7.11.0': removeSuggestedPriority,
	'7.12.0': transformTableState,
	'7.13.0': renameOperationsForFormula,
	'7.13.1': renameOperationsForFormula, // duplicate this migration in case a broken by value panel is added to the library
	'7.14.0': removeTimezoneDateHistogramParam,
	'7.15.0': addLayerTypeToVisualization,
	'7.16.0': moveDefaultReversedPaletteToCustom,
	'8.1.0': flow(renameFilterReferences, renameRecordsField, addParentFormatter),
	'8.2.0': flow(
	setLastValueShowArrayValues,
	setIncludeEmptyRowsDateHistogram,
	enhanceTableRowHeight
	),
	'8.3.0': flow(lockOldMetricVisSettings, preserveOldLegendSizeDefault, fixValueLabelsInXY),
	'8.5.0': flow(migrateMetricIds, enrichAnnotationLayers, migratePartitionChartGroups),
	'8.6.0': flow(migrateIndexPatternDatasource, migratePartitionMetrics),
	// FOLLOW THESE GUIDELINES IF YOU ARE ADDING A NEW MIGRATION!
	// 1. Make sure you are applying migrations for a given version in the same order here as they are applied in x-pack/plugins/lens/server/embeddable/make_lens_embeddable_factory.ts
	};
	export const getAllMigrations = (
	filterMigrations: MigrateFunctionsObject,
	dataViewMigrations: MigrateFunctionsObject,
	customVisualizationMigrations: CustomVisualizationMigrations
	): SavedObjectMigrationMap =>
	mergeSavedObjectMigrationMaps(
	mergeSavedObjectMigrationMaps(
	mergeSavedObjectMigrationMaps(
	lensMigrations,
	getLensFilterMigrations(filterMigrations) as unknown as SavedObjectMigrationMap
	),
	getLensCustomVisualizationMigrations(customVisualizationMigrations)
	),
	getLensDataViewMigrations(dataViewMigrations) as unknown as SavedObjectMigrationMap
	);

We need to do something similar for rules migrations in the alerting plugin:

kibana/x-pack/plugins/alerting/server/saved_objects/migrations/index.ts

Lines 38 to 83 in 47ad5ed

	export function getMigrations(
	encryptedSavedObjects: EncryptedSavedObjectsPluginSetup,
	searchSourceMigrations: MigrateFunctionsObject,
	isPreconfigured: (connectorId: string) => boolean
	): SavedObjectMigrationMap {
	return mergeSavedObjectMigrationMaps(
	{
	'7.10.0': executeMigrationWithErrorHandling(
	getMigrations7100(encryptedSavedObjects),
	'7.10.0'
	),
	'7.11.0': executeMigrationWithErrorHandling(
	getMigrations7110(encryptedSavedObjects),
	'7.11.0'
	),
	'7.11.2': executeMigrationWithErrorHandling(
	getMigrations7112(encryptedSavedObjects),
	'7.11.2'
	),
	'7.13.0': executeMigrationWithErrorHandling(
	getMigrations7130(encryptedSavedObjects),
	'7.13.0'
	),
	'7.14.1': executeMigrationWithErrorHandling(
	getMigrations7140(encryptedSavedObjects),
	'7.14.1'
	),
	'7.15.0': executeMigrationWithErrorHandling(
	getMigrations7150(encryptedSavedObjects),
	'7.15.0'
	),
	'7.16.0': executeMigrationWithErrorHandling(
	getMigrations7160(encryptedSavedObjects, isPreconfigured),
	'7.16.0'
	),
	'8.0.0': executeMigrationWithErrorHandling(getMigrations800(encryptedSavedObjects), '8.0.0'),
	'8.0.1': executeMigrationWithErrorHandling(getMigrations801(encryptedSavedObjects), '8.0.1'),
	'8.2.0': executeMigrationWithErrorHandling(getMigrations820(encryptedSavedObjects), '8.2.0'),
	'8.3.0': executeMigrationWithErrorHandling(getMigrations830(encryptedSavedObjects), '8.3.0'),
	'8.4.1': executeMigrationWithErrorHandling(getMigrations841(encryptedSavedObjects), '8.4.1'),
	'8.5.0': executeMigrationWithErrorHandling(getMigrations850(encryptedSavedObjects), '8.5.0'),
	'8.6.0': executeMigrationWithErrorHandling(getMigrations860(encryptedSavedObjects), '8.6.0'),
	},
	getSearchSourceMigrations(encryptedSavedObjects, searchSourceMigrations)
	);
	}

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)