[ML] Transform: support for create/update with API key for secondary credentials

[droberts195]

As part of the work to solve #137278 we need the ability to create or update a transform and supply an API key for secondary credentials.

The API key should be created by taking a snapshot of the credentials of the logged-in user. The ResponseOps code contains an example of how to do this:

kibana/x-pack/plugins/alerting/server/rules_client_factory.ts

Lines 110 to 120 in ff312c9

	async createAPIKey(name: string) {
	if (!securityPluginStart) {
	return { apiKeysEnabled: false };
	}
	// Create an API key using the new grant API - in this case the Kibana system user is creating the
	// API key for the user, instead of having the user create it themselves, which requires api_key
	// privileges
	const createAPIKeyResult = await securityPluginStart.authc.apiKeys.grantAsInternalUser(
	request,
	{ name, role_descriptors: {} }
	);

The role_descriptors: {} part is the part that means "take a snapshot of the credentials of the logged-in user".

The header for supplying secondary credentials is es-secondary-authorization:

kibana/x-pack/plugins/ml/server/lib/request_authorization.ts

Line 18 in f5eab68

headers: { 'es-secondary-authorization': request.headers.authorization },

We'll need to find out how to format the contents of an API key as a suitable string value for that header.

The secondary credentials containing an API key should be provided to all calls that the Fleet package installer makes that create or update transforms.

Additionally there should be a function that takes a transform name as input, generates an API key for the currently logged-in user, and then does a no-op update to the transform with primary credentials being the kibana_system user and secondary credentials being the API key. This function can then be called from either the transform management page or the Fleet packages page, to allow a user with the necessary permissions that the transform requires to authorize a transform that currently has inadequate credentials. Transform credentials may be inadequate because it was originally installed with inadequate credentials or because the API key it was originally using has been invalidated. However, in both scenarios the steps needed to fix the problem are the same, so they should be wrapped up in a reusable function.

[elasticmachine]

Pinging @elastic/ml-ui (:ml)

[droberts195]

The problem that revealed the need for elastic/elasticsearch#95187 also reveals an additional requirement for this piece of work.

Some packaged transforms get their source data from system indices, and hence should run as a special system users, as we do not want to be recommending that normal users need access to system indices.

There is also a backwards compatibility concern. Current packages expected to be installed and run as kibana_system, so if we change the default user for those existing packages then we run the risk of problems.

We can solve all these issues by mandating that to use the new secondary auth functionality a transform must have the following in its _meta:

"_meta" : {
  "run_as_kibana_system": false
}

We'll document this in the next edition of the transform package best practices doc, and it will also need to go somewhere in the public docs.

[qn895]

We can solve all these issues by mandating that to use the new secondary auth functionality a transform must have the following in its _meta

@droberts195 I updated the Fleet transform installation to check for that flag and use it on a per-transform basis here for Fleet. If run_as_kibana_system is explicit set to false, the secondary authorization will be used when creating the transform.

[qn895]

Closing this issue via #154736