Add transforms creation to the CSP plugin initialization #129905

eyalkraft · 2022-04-11T15:29:06Z

Summary

Following the decision to hold back the integration package-spec change to support transform assets,
We will temporarily create the required transforms required for the CSP plugin as part of the plugin initialization itself.
These transforms will be moved to be part of our integration package as soon as the package-spec will allow it.

Checklist

This change was tested locally

x-pack/plugins/cloud_security_posture/server/create_transforms/benchmark_score_transform.ts

x-pack/plugins/cloud_security_posture/server/create_transforms/latest_findings_transform.ts

x-pack/plugins/cloud_security_posture/server/create_transforms/benchmark_score_transform.ts

x-pack/plugins/cloud_security_posture/server/plugin.ts

…-fix'

eyalkraft · 2022-04-17T14:23:58Z

I was forced to add findings-*

export const PRIVILEGED_CSP_KUBEBEAT_INDEX_PATTERN = 'logs-cis_kubernetes_benchmark.findings-*';

and not use the existing (no hyphen) findings*

export const CSP_KUBEBEAT_INDEX_PATTERN = 'logs-cis_kubernetes_benchmark.findings*';

since this is the index pattern the kibana user has the proper privileges for as a result of this PR to Elasticsearch.

Otherwise the following error occurs:

[2022-04-17T17:07:12.066+03:00][ERROR][plugins.cloudSecurityPosture] Failed to create transform cloud_security_posture.latest-default-0.0.1: security_exception: Cannot create transform [cloud_security_posture.latest-default-0.0.1] because user kibana_system lacks the required permissions [logs-cis_kubernetes_benchmark.findings*:[read], logs-cloud_security_posture.findings_latest-default:[]]

wdyt @kfirpeled?

We can also change the original CSP_KUBEBEAT_INDEX_PATTERN ofcourse but it seems more risky, and given that we rename it soon enough maybe better to postpone this change.

Edit: On second thought just adding the - to the original constant seems more reasonable...

elasticmachine · 2022-04-17T14:27:24Z

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security Posture)

…bana into create-transforms-in-plugin

kfirpeled

as is, it looks good to me.
the index pattern was wrong and I've changed that only on the integration part (adding the - before the astrix).

now I see it is needed here as well.

kfirpeled · 2022-04-17T14:56:31Z

x-pack/plugins/cloud_security_posture/common/constants.ts

@@ -17,7 +17,7 @@ export const LATEST_FINDINGS_INDEX_NAME = 'cloud_security_posture.findings_lates
 export const BENCHMARK_SCORE_INDEX_NAME = 'cloud_security_posture.scores';

 export const AGENT_LOGS_INDEX_PATTERN = '.logs-cis_kubernetes_benchmark.metadata*';
-export const CSP_KUBEBEAT_INDEX_PATTERN = 'logs-cis_kubernetes_benchmark.findings*';
+export const CSP_KUBEBEAT_INDEX_PATTERN = 'logs-cis_kubernetes_benchmark.findings-*';


kfirpeled

added few comments

kfirpeled · 2022-04-17T16:04:29Z

x-pack/plugins/cloud_security_posture/server/create_indices/create_transforms_indices.ts

-    benchmarkScoreMapping,
-    logger
-  );
+  return Promise.all([


x-pack/plugins/cloud_security_posture/server/create_transforms/latest_findings_transform.ts

kfirpeled · 2022-04-17T17:26:11Z

x-pack/plugins/cloud_security_posture/server/create_transforms/latest_findings_transform.ts

+  transform_id: 'cloud_security_posture.latest-default-0.0.1',
+  description: 'Defines findings transformation to view only the latest finding per resource',
+  source: {
+    index: CSP_KUBEBEAT_INDEX_PATTERN,


fyi, when cloudbeat would support namespaces, this transform would take results from other spaces into the default's latest index.

what if the index pattern of this transform would be logs-cis_kubernetes_benchmark.findings-default-*. looking ahead, it might prevent a state that would be harder to fix if leaving it as it is now.

I'm pretty sure the pattern you suggested won't work as the datastream name is logs-cis_kubernetes_benchmark.findings-default and the underlying index name is .ds-logs-cis_kubernetes_benchmark.findings-default-2022.04.18-000001

correct, it looks like the datastream name would achieve the same result though.
anyway, we can fix that later

will re-review on next updates

x-pack/plugins/cloud_security_posture/server/create_transforms/create_transforms.ts

kfirpeled

🚀

eyalkraft · 2022-04-18T12:33:29Z

tested locally

kfirpeled · 2022-04-18T12:37:46Z

x-pack/plugins/cloud_security_posture/server/create_transforms/create_transforms.ts

    });
+    return true;


nit: it returns true although it was not created. consider documentation/ better naming for the function

kfirpeled · 2022-04-18T13:28:29Z

🚢 it

kibana-ci · 2022-04-18T14:07:43Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`cloudSecurityPosture`	407.1KB	407.1KB	+1.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`cloudSecurityPosture`	14	15	+1

Total ESLint disabled count

id	before	after	diff
`cloudSecurityPosture`	15	16	+1

History

💚 Build #38603 succeeded b07a2e6
💚 Build #38590 succeeded 8fb23cc
💚 Build #38556 succeeded 378db45
💚 Build #38552 succeeded 0418617
💛 Build #37313 was flaky 5479505

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @eyalkraft

) * add transforms * [CI] Auto-commit changed files from 'node scripts/eslint --no-cache --fix' * catch 404 * add hyphen * start transforms, promise.all * return is clearer * add tests * transform rename * add test * use exact pattern * only start if created Co-authored-by: kibanamachine <[email protected]>

add transforms

5479505

eyalkraft added release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related v8.3.0 labels Apr 11, 2022

eyalkraft self-assigned this Apr 11, 2022

This was referenced Apr 13, 2022

Move transforms from the CSP plugin to the CSP integration package #130086

Closed

Add transform to spec elastic/package-spec#307

Merged

kfirpeled reviewed Apr 14, 2022

View reviewed changes

JordanSh mentioned this pull request Apr 14, 2022

switched to use latest findings index #130231

Closed

kfirpeled reviewed Apr 14, 2022

View reviewed changes

x-pack/plugins/cloud_security_posture/server/plugin.ts Show resolved Hide resolved

eyalkraft and others added 3 commits April 17, 2022 15:27

Merge branch 'main' into create-transforms-in-plugin

b72fbaf

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

163b2e3

…-fix'

catch 404

175743d

Merge branch 'main' into create-transforms-in-plugin

02b1e66

eyalkraft marked this pull request as ready for review April 17, 2022 14:27

eyalkraft added 2 commits April 17, 2022 17:45

add hyphen

9f7e602

Merge branch 'create-transforms-in-plugin' of github.com:eyalkraft/ki…

a46f415

…bana into create-transforms-in-plugin

kfirpeled previously approved these changes Apr 17, 2022

View reviewed changes

kfirpeled self-requested a review April 17, 2022 15:00

eyalkraft added 2 commits April 17, 2022 18:45

start transforms, promise.all

81f7c6f

return is clearer

0418617

kfirpeled reviewed Apr 17, 2022

View reviewed changes

kfirpeled self-requested a review April 17, 2022 17:28

eyalkraft added 2 commits April 17, 2022 20:35

add tests

83e2948

transform rename

378db45

kfirpeled reviewed Apr 17, 2022

View reviewed changes

x-pack/plugins/cloud_security_posture/server/create_transforms/create_transforms.ts Show resolved Hide resolved

x-pack/plugins/cloud_security_posture/server/create_transforms/create_transforms.ts Outdated Show resolved Hide resolved

add test

8fb23cc

Merge branch 'main' into create-transforms-in-plugin

b07a2e6

kfirpeled approved these changes Apr 18, 2022

View reviewed changes

eyalkraft added 2 commits April 18, 2022 14:53

use exact pattern

3ca2fae

only start if created

a4017cc

eyalkraft enabled auto-merge (squash) April 18, 2022 12:33

kfirpeled approved these changes Apr 18, 2022

View reviewed changes

eyalkraft and others added 2 commits April 18, 2022 15:46

add function doc

da2c894

Merge branch 'main' into create-transforms-in-plugin

4957567

eyalkraft merged commit d55c7b3 into elastic:main Apr 18, 2022

kibanamachine added the backport:skip This commit does not require backporting label Apr 18, 2022

eyalkraft deleted the create-transforms-in-plugin branch April 18, 2022 15:16

eyalkraft mentioned this pull request Apr 19, 2022

Fix transform creation on missing source index #130543

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add transforms creation to the CSP plugin initialization #129905

Add transforms creation to the CSP plugin initialization #129905

eyalkraft commented Apr 11, 2022 •

edited

Loading

eyalkraft commented Apr 17, 2022 •

edited

Loading

elasticmachine commented Apr 17, 2022

kfirpeled left a comment •

edited

Loading

kfirpeled Apr 17, 2022

kfirpeled left a comment

kfirpeled Apr 17, 2022

kfirpeled Apr 17, 2022 •

edited

Loading

eyalkraft Apr 18, 2022

kfirpeled Apr 18, 2022

kfirpeled left a comment

eyalkraft commented Apr 18, 2022

kfirpeled Apr 18, 2022

kfirpeled commented Apr 18, 2022

kibana-ci commented Apr 18, 2022

ESLint disabled line counts

Total ESLint disabled count

Add transforms creation to the CSP plugin initialization #129905

Add transforms creation to the CSP plugin initialization #129905

Conversation

eyalkraft commented Apr 11, 2022 • edited Loading

Summary

Checklist

eyalkraft commented Apr 17, 2022 • edited Loading

elasticmachine commented Apr 17, 2022

kfirpeled left a comment • edited Loading

Choose a reason for hiding this comment

kfirpeled Apr 17, 2022

Choose a reason for hiding this comment

kfirpeled left a comment

Choose a reason for hiding this comment

kfirpeled Apr 17, 2022

Choose a reason for hiding this comment

kfirpeled Apr 17, 2022 • edited Loading

Choose a reason for hiding this comment

eyalkraft Apr 18, 2022

Choose a reason for hiding this comment

kfirpeled Apr 18, 2022

Choose a reason for hiding this comment

kfirpeled left a comment

Choose a reason for hiding this comment

eyalkraft commented Apr 18, 2022

kfirpeled Apr 18, 2022

Choose a reason for hiding this comment

kfirpeled commented Apr 18, 2022

kibana-ci commented Apr 18, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

ESLint disabled line counts

Total ESLint disabled count

History

eyalkraft commented Apr 11, 2022 •

edited

Loading

eyalkraft commented Apr 17, 2022 •

edited

Loading

kfirpeled left a comment •

edited

Loading

kfirpeled Apr 17, 2022 •

edited

Loading