[ML] Adds secondary authorization header to Transforms in Fleet #154665

qn895 · 2023-04-10T17:02:59Z

Summary

The PR updates how credentials are created and managed for packages including Transforms. Previously, everything will be installed as kibana_system user, which has limited permissions to a specific set of indices defined internally. This PR changes so that a secondary authorization is passed to the creation of Transforms, making the permissions/privileges dependent on the logged-in user.

Installing a package containing transforms

If the package has transforms assets to be installed, it will show warning/info call out message indicating that the transforms will be created and started with the current user's credentials/roles.

-It will parse the authorization header (schema and credentials) from the Kibana request to the package handlers.

If the package contains transforms, and if run_as_kibana_system: false in the any of the transform yml config , then generate an API key from the above credential (as that Kibana user with the roles and permissions at the time of generation), and use it in transform/_put requests.
If user has sufficient permissions:
- Transforms will be successfully created and started. They will be marked in the saved object reference with deferred: false
- Transform _meta will have installed_by: {username}
- Package will be successfully installed
If user has insufficient permissions:
- Transforms will be successfully created, but fail to start. They will be marked in the saved object reference with deferred: true
- Package will still be successfully installed. It will show warning that the package has some deferred installations.

Deferred installations

If a package has deferred installations (a.k.a assets that were included in the package, but require additional permissions to operate correctly), it will:

Show a warning on the Installed integrations page:
Show a warning badge with explanation on the tab:
Show a new Deferred installations section as well as call out message to prompt user to re-authorize inside the Assets tab:

If the currently logged-in user has sufficient permissions (manage_transform ES cluster privilege/transform_admin Kibana role), the Reauthorize buttons will be enabled:

Reauthorizing installations

For transforms:
- Clicking the Reauthorize button will send an _transform/_update API request with a headers: {es-secondary-authorization: 'ApiKey {encoded_api}' and then a _transform/_start to start operations.
- Transform _meta will be updated with addition of last_authorized_by: {username}
- If order is specified in _meta of the transform, they will be updated and started sequentially. Else, they will be executed concurrently.

Reviewers note:

-For kibana-core: saved object for Fleet's EsAsset was extended with deferred: boolean, thus changing the hash.

Checklist

Delete any items that are not applicable to this PR.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

This was checked for breaking API changes and was labeled appropriately

…fix'

lcawl

Minor suggestions, otherwise UI text LGTM

...c/applications/integrations/sections/epm/screens/detail/assets/deferred_assets_accordion.tsx

...lic/applications/integrations/sections/epm/screens/detail/assets/deferred_assets_warning.tsx

...plications/integrations/sections/epm/screens/detail/assets/deferred_transforms_accordion.tsx

nchaulet · 2023-04-17T17:00:58Z

@qn895 Where I can found the hot risk score integration to test this locally?

nchaulet · 2023-04-17T22:08:30Z

@qn895 I did some local tests and overall it seems to work well. I may have one suggestion one we click on reauthorize all it could be nice to reload the assets tab what do you think?

nchaulet · 2023-04-18T12:12:34Z

...plications/integrations/sections/epm/screens/detail/assets/deferred_transforms_accordion.tsx

+    transformId: string;
+  }> | null>(null);
+
+  useEffect(() => {


nit: do we really need a useEffect here can we call an authorizeTransforms directly from the click events?

Updated here 4e54dee (#154665)

nchaulet

Tested locally and LGTM 🚀

szabosteve

One small suggestion, otherwise UI text LGTM!

szabosteve · 2023-04-18T12:55:27Z

...plications/integrations/sections/epm/screens/detail/assets/deferred_transforms_accordion.tsx

+        'xpack.fleet.epm.packageDetails.assets.deferredTransformInstallationsDescription',
+        {
+          defaultMessage:
+            '{assetCount, plural, one {Transform was installed but requires} other {# transforms were installed but require}} additional permissions to run. You must have the transform_admin built-in role or manage_transform cluster privileges to start operations.',


Based on the discussion here, I suggest slightly modifying this message:

Suggested change

'{assetCount, plural, one {Transform was installed but requires} other {# transforms were installed but require}} additional permissions to run. You must have the transform_admin built-in role or manage_transform cluster privileges to start operations.',

'{assetCount, plural, one {Transform was installed but requires} other {# transforms were installed but require}} additional permissions to run. Contact your administrator to request the required privileges.',

Updated here 4e54dee (#154665)

- Add comment to route - Update settings to swallow error without blocking - Remove redundant update/delete index alias code - Better error handling, and scenario if security is not defined

peteharverson

Tested with your example Host Risk Score package and LGTM

peteharverson

Taking a second look I'm wondering if some of the warnings need changes?

...plications/integrations/sections/epm/screens/detail/assets/deferred_transforms_accordion.tsx

...lic/applications/integrations/sections/epm/screens/detail/assets/deferred_assets_warning.tsx

hop-dev · 2023-04-19T16:44:12Z

@qn895 Where I can found the hot risk score integration to test this locally?

@nchaulet @qn895 same question! I would like to test this locally if possible

nchaulet · 2023-04-19T17:58:52Z

@hop-dev you can find the package here https://github.com/qn895/integrations/tree/host_risk_score_versioned

peteharverson

Text changes for the warnings LGTM

hop-dev

Tested locally, looks good to me!

Were you able to test the scenario where the package is reinstalled as system user? Do we retain the previous auth header?

hop-dev · 2023-04-20T13:15:22Z

x-pack/plugins/fleet/common/http_authorization_header.ts

+
+// Extended version of x-pack/plugins/security/server/authentication/http_authentication/http_authorization_header.ts
+// to prevent bundle being required in security_solution
+// FIXME: Put this in a package


For this FIXME comment do we have an issue for the fix? When do we expect to fix it?

I removed this FIXME for now as the file is modified from the original copy of security_solution's code 0187c5f (#154665)

hop-dev · 2023-04-20T13:21:52Z

x-pack/plugins/fleet/public/components/transform_install_as_current_user_callout.tsx

+  count: number;
+}> = ({ count }) => {
+  return (
+    <EuiCallOut color="primary">


I think callouts should always have a title with an icon , check out the EUI examples https://elastic.github.io/eui/#/display/callout

Updated here 0187c5f (#154665)

hop-dev · 2023-04-20T13:23:11Z

x-pack/plugins/fleet/server/routes/epm/handlers.ts

@@ -432,3 +450,60 @@ export const getVerificationKeyIdHandler: FleetRequestHandler = async (
    return defaultFleetErrorHandler({ error, response });
  }
 };
+
+/**


thanks for the comments here 👍

qn895 · 2023-04-20T15:19:41Z

Were you able to test the scenario where the package is reinstalled as system user? Do we retain the previous auth header?

As discussed in Slack with you and Nicolas, I'll create a smaller follow-up PR to address force reinstallation of packages containing transforms with the new spec. Thanks all for your help testing and reviewing this PR 🥳

kibana-ci · 2023-04-20T16:14:53Z

💚 Build Succeeded

Buildkite Build
Commit: 0187c5f

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`fleet`	798	803	+5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`fleet`	998	1004	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`fleet`	945.5KB	954.9KB	+9.4KB

Public APIs missing exports

Total count of every type that is part of your API that should be exported but is not. This will cause broken links in the API documentation system. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats exports for more detailed information.

id	before	after	diff
`fleet`	27	28	+1

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	126.7KB	127.4KB	+689.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/saved-objects-service.html#_mappings

id	before	after	diff
`epm-packages`	22	23	+1

Unknown metric groups

API count

id	before	after	diff
`fleet`	1103	1109	+6

ESLint disabled line counts

id	before	after	diff
`fleet`	49	48	-1
`securitySolution`	394	397	+3
total			+2

Total ESLint disabled count

id	before	after	diff
`fleet`	59	58	-1
`securitySolution`	474	477	+3
total			+2

History

💛 Build #121325 was flaky c7ad1c7
💚 Build #120891 succeeded 73b1216
💚 Build #120506 succeeded 4d2eb77
💔 Build #119795 failed d9b9da5
💛 Build #119641 was flaky 5bb5690
💔 Build #119629 failed b4112ff8ec9cde1d441926fe104e2bc19905da42

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @qn895

droberts195

LGTM

I'm happy to merge this and fix the inefficiency in a followup.

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts

qn895 added 3 commits April 6, 2023 12:20

Cherry pick only Fleet changes

1056f2f

Cherry pick only Fleet changes

a151080

Fix i18n and types

59b7541

qn895 requested review from droberts195 and przemekwitek April 10, 2023 18:26

qn895 changed the title ~~[ML] Add secondary authorization header to Transforms in Fleet~~ [ML] Adds secondary authorization header to Transforms in Fleet Apr 10, 2023

qn895 force-pushed the ml-fleet-2nd-auth branch from 5c5f862 to 4086164 Compare April 10, 2023 20:02

Fix tests

605b27b

qn895 force-pushed the ml-fleet-2nd-auth branch from 4086164 to 605b27b Compare April 10, 2023 20:09

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

918c9c4

…fix'

qn895 requested review from nchaulet and hop-dev April 11, 2023 14:25

qn895 mentioned this pull request Apr 11, 2023

[ML] Adds option to Reauthorize transform in Management page #154736

Merged

11 tasks

qn895 added 3 commits April 11, 2023 10:42

[ML] Update hash

b74f843

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

bef90f7

[ML] Add username to transform meta

bc55ec8

lcawl approved these changes Apr 11, 2023

View reviewed changes

lcawl added the ui-copy Review of UI copy with docs team is recommended label Apr 11, 2023

kevinlog mentioned this pull request Apr 11, 2023

Failing ES Promotion: X-Pack Endpoint Functional Tests.x-pack/test/security_solution_endpoint/apps/endpoint - endpoint "before all" hook in "endpoint" #154741

Closed

qn895 added 7 commits April 11, 2023 15:54

[ML] Add callout message

40802ff

[ML] Fix messaging

98a01d4

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

2885aa6

[Fleet] Fix test

49562f4

[Fleet] Fix deferred missing due to transform not set to auto start

d11196c

Merge upstream/main into branch

1cf340f

Add run_as_kibana_system check

5bb5690

qn895 force-pushed the ml-fleet-2nd-auth branch from b4112ff to 5bb5690 Compare April 13, 2023 04:55

qn895 marked this pull request as ready for review April 13, 2023 05:41

qn895 requested review from a team as code owners April 13, 2023 05:41

qn895 added 2 commits April 17, 2023 18:11

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

4d2eb77

Remove todo

0870d08

nchaulet reviewed Apr 18, 2023

View reviewed changes

nchaulet approved these changes Apr 18, 2023

View reviewed changes

szabosteve approved these changes Apr 18, 2023

View reviewed changes

qn895 added 2 commits April 18, 2023 16:31

- Add key to fragment to rid of React errors

4e54dee

- Add comment to route - Update settings to swallow error without blocking - Remove redundant update/delete index alias code - Better error handling, and scenario if security is not defined

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

73b1216

peteharverson approved these changes Apr 19, 2023

View reviewed changes

peteharverson reviewed Apr 19, 2023

View reviewed changes

...plications/integrations/sections/epm/screens/detail/assets/deferred_transforms_accordion.tsx Outdated Show resolved Hide resolved

...lic/applications/integrations/sections/epm/screens/detail/assets/deferred_assets_warning.tsx Show resolved Hide resolved

peteharverson self-requested a review April 19, 2023 11:37

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

22b5f9d

Update messaging

c7ad1c7

peteharverson approved these changes Apr 20, 2023

View reviewed changes

hop-dev approved these changes Apr 20, 2023

View reviewed changes

qn895 added 2 commits April 20, 2023 09:54

Merge remote-tracking branch 'upstream/main' into ml-fleet-2nd-auth

57bc06a

Add icon, remove fixme

0187c5f

droberts195 approved these changes Apr 20, 2023

View reviewed changes

x-pack/plugins/fleet/server/services/epm/elasticsearch/transform/install.ts Show resolved Hide resolved

qn895 merged commit 1fe26f3 into elastic:main Apr 20, 2023

kibanamachine added the backport:skip This commit does not require backporting label Apr 20, 2023

kfirpeled mentioned this pull request Apr 25, 2023

[Cloud Security Posture] logs-cloud_security_posture.findings_latest-default mappings fails to update during Kibana version upgrade #153718

Closed

angorayc mentioned this pull request Mar 27, 2024

Add support for secondary auth to Core's ES client #179458

Closed

mgiota mentioned this pull request Aug 1, 2024

[SLO][Fleet] Reauthorize SLO transforms #189756

Closed

mgiota mentioned this pull request Aug 22, 2024

[SLOs] remove manage_transform and manage_ingest_pipeline privilege requirements #190572

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ML] Adds secondary authorization header to Transforms in Fleet #154665

[ML] Adds secondary authorization header to Transforms in Fleet #154665

qn895 commented Apr 10, 2023 •

edited

Loading

lcawl left a comment

nchaulet commented Apr 17, 2023

nchaulet commented Apr 17, 2023

nchaulet Apr 18, 2023

qn895 Apr 19, 2023

nchaulet left a comment

szabosteve left a comment

szabosteve Apr 18, 2023

qn895 Apr 19, 2023

peteharverson left a comment

peteharverson left a comment

hop-dev commented Apr 19, 2023

nchaulet commented Apr 19, 2023

peteharverson left a comment

hop-dev left a comment

hop-dev Apr 20, 2023

qn895 Apr 20, 2023

hop-dev Apr 20, 2023

qn895 Apr 20, 2023

hop-dev Apr 20, 2023

qn895 commented Apr 20, 2023

kibana-ci commented Apr 20, 2023

API count

ESLint disabled line counts

Total ESLint disabled count

droberts195 left a comment

	'{assetCount, plural, one {Transform was installed but requires} other {# transforms were installed but require}} additional permissions to run. You must have the transform_admin built-in role or manage_transform cluster privileges to start operations.',
	'{assetCount, plural, one {Transform was installed but requires} other {# transforms were installed but require}} additional permissions to run. Contact your administrator to request the required privileges.',

[ML] Adds secondary authorization header to Transforms in Fleet #154665

[ML] Adds secondary authorization header to Transforms in Fleet #154665

Conversation

qn895 commented Apr 10, 2023 • edited Loading

Summary

Installing a package containing transforms

Deferred installations

Reauthorizing installations

Reviewers note:

Checklist

Risk Matrix

For maintainers

lcawl left a comment

Choose a reason for hiding this comment

nchaulet commented Apr 17, 2023

nchaulet commented Apr 17, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nchaulet left a comment

Choose a reason for hiding this comment

szabosteve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

peteharverson left a comment

Choose a reason for hiding this comment

hop-dev commented Apr 19, 2023

nchaulet commented Apr 19, 2023

peteharverson left a comment

Choose a reason for hiding this comment

hop-dev left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qn895 commented Apr 20, 2023

kibana-ci commented Apr 20, 2023

💚 Build Succeeded

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Public APIs missing exports

Page load bundle

Saved Objects .kibana field count

API count

ESLint disabled line counts

Total ESLint disabled count

History

droberts195 left a comment

Choose a reason for hiding this comment

qn895 commented Apr 10, 2023 •

edited

Loading