[Cloud Security][Metering] Report all assets details #186777

CohenIdo · 2024-06-23T09:22:01Z

solves:

https://github.com/elastic/security-team/issues/9435

Summary

The Cloud Security solution charges solely for billable assets in a serverless project.

With this PR, we utilize the metadata object from the usage-v2-schema to report the complete list of cloud assets for future debugging and optimization purposes.

Note: the API spec doesn't allow nested objects in metadata, that's the reason the tier remains at the same level as the resource dub type details. In addition the metadata object in both the UsageMetrics and UsageSource schemas can contain a dictionary of key-value pairs where both the keys and the values are strings.

Example

In the payload below, you can find the environment containing 4 unique assets, but we only charge for 3.

{
  "id": "cloud_security_cspm_missing_project_id_2024-07-02T08:30:00.000Z",
  "usage_timestamp": "2024-07-02T05:08:08.350Z",
  "creation_timestamp": "2024-07-02T08:30:00.000Z",
  "usage": {
    "type": "cloud_security",
    "sub_type": "cspm",
    "quantity": 3,
    "period_seconds": 1800,
    "metadata": {
      "aws-s3":  2,
      "aws-rds":  1,
      "aws-not-bllable-asset#1":   1,
      "aws-not-bllable-asset#2":   3,
      "aws-not-bllable-asset#3":   5
    }
  },
  "source": {
    "id": "cloud-security-usage-reporting-task:1",
    "instance_group_id": "missing_project_id",
    "metadata": {
      "tier": "complete"
    }
  }
}

CohenIdo · 2024-06-23T09:32:57Z

...k/plugins/security_solution_serverless/server/cloud_security/cloud_security_metering_task.ts

@@ -212,21 +219,20 @@ export const getCloudSecurityUsageRecord = async ({
  logger,
 }: CloudSecurityMeteringCallbackInput): Promise<UsageRecord[] | undefined> => {
  try {


Following our decision to prevent overbilling and until we implement a recovery mechanism, getSearchStartDate is redundant.

CohenIdo · 2024-06-23T09:34:59Z

x-pack/plugins/security_solution_serverless/server/cloud_security/cloud_security_metering.ts

@@ -21,6 +21,15 @@ export const cloudSecurityMetringCallback = async ({
  lastSuccessfulReport,
  config,
 }: MeteringCallbackInput): Promise<MeteringCallBackResponse> => {


Following FP alerts we had, I found it more effective to skip cloud metering earlier.

elasticmachine · 2024-06-24T06:04:45Z

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

kibana-ci · 2024-07-02T09:25:56Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: c323796
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186777-c323796d1805
Security Deployment

Failed CI Steps

Metrics [docs]

✅ unchanged

History

💔 Build #218645 failed a467e97e547d423e947094a5345ebcbe1ee62474
💔 Build #217957 failed c08f1d5
💔 Build #217648 failed c5e3213
💔 Build #217338 failed 3c86f6f

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

joeypoon

My knowledge of cloud billing isn't that intimate so with that in mind, it looks good overall. I believe we should make one change though. We should use usage.metadata here instead of source.metadata since these are billing details, not source/project details.

CohenIdo · 2024-07-03T06:21:25Z

My knowledge of cloud billing isn't that intimate so with that in mind, it looks good overall. I believe we should make one change though. We should use usage.metadata here instead of source.metadata since these are billing details, not source/project details.

@joeypoon I agree, I moved it to usage.metadata

eyalkraft

Thanks @CohenIdo!

Actually I'd expect the complete deletion of BILLABLE_ASSETS_CONFIG and billable notion whatsoever from the Kibana code.
The list of billable assets should be moved to the glue function where we will be free to change what's billable and what's not without even changing Kibana.

I think that' a better separation of responsibilities and aligns with the value we're trying to achieve with this change.

wdyt?

CohenIdo · 2024-07-03T08:06:40Z

Thanks @CohenIdo!

Actually I'd expect the complete deletion of BILLABLE_ASSETS_CONFIG and billable notion whatsoever from the Kibana code. The list of billable assets should be moved to the glue function where we will be free to change what's billable and what's not without even changing Kibana.

I think that' a better separation of responsibilities and aligns with the value we're trying to achieve with this change.

wdyt?

@eyalkraft, that's was the original plan, together with @kfirpeled we decided to not have breaking change but to achieve most of the goals with the current implementation.

joeypoon

I have one tiny comment which would be great if you can fix before merge. Otherwise 👍.

joeypoon · 2024-07-04T01:18:15Z

x-pack/plugins/security_solution_serverless/server/types.ts

@@ -65,17 +66,13 @@ export interface UsageMetrics {
  quantity: number;
  period_seconds?: number;
  cause?: string;
-  metadata?: unknown;
+  metadata?: ResourceSubtypeCounter;


It feels a bit strange to import from a child file upward. We should move ResourceSubtypeCounter into this file or just use [key: string]: string directly.

elasticmachine · 2024-07-04T08:21:51Z

⏳ Build in-progress

Buildkite Build
Commit: b35d731
Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-186777-b35d73186cff
Security Deployment

History

💔 Build #219551 failed 86ed5b5
💔 Build #219506 failed 99906c4
💛 Build #219404 was flaky 99906c4
💔 Build #219367 failed 3b437e7
💚 Build #219275 succeeded d05233b

report all assets details

279db5e

CohenIdo added release_note:skip Skip the PR/issue when compiling release notes Team:Cloud Security Cloud Security team related ci:project-deploy-security Create a Security Serverless Project v8.15.0 labels Jun 23, 2024

CohenIdo commented Jun 23, 2024

View reviewed changes

CohenIdo requested a review from kfirpeled June 23, 2024 09:35

CohenIdo added 2 commits June 23, 2024 12:57

report all assets details

dd27733

Merge remote-tracking branch 'upstream/main' into report-all-csp-assets

acca3d5

CohenIdo marked this pull request as ready for review June 24, 2024 06:04

CohenIdo requested a review from a team as a code owner June 24, 2024 06:04

CohenIdo added 2 commits June 24, 2024 10:20

types

6cee5ee

Merge remote-tracking branch 'upstream/main' into report-all-csp-assets

3c86f6f

CohenIdo force-pushed the report-all-csp-assets branch from d7ce66b to 3c86f6f Compare June 24, 2024 07:21

CohenIdo added 5 commits June 25, 2024 16:22

unit tests

c5e3213

debug

c08f1d5

debug

223e616

fixed version

61a0002

Merge remote-tracking branch 'upstream/main' into report-all-csp-assets

c323796

CohenIdo force-pushed the report-all-csp-assets branch from a467e97 to c323796 Compare July 2, 2024 08:26

CohenIdo added 2 commits July 2, 2024 19:06

update types

66f58e8

update types

bd1e26f

CohenIdo requested review from joeypoon and removed request for kfirpeled July 2, 2024 19:54

joeypoon requested changes Jul 2, 2024

View reviewed changes

move to usage metadata

d05233b

CohenIdo requested a review from joeypoon July 3, 2024 06:21

eyalkraft reviewed Jul 3, 2024

View reviewed changes

CohenIdo requested a review from eyalkraft July 3, 2024 08:07

eyalkraft approved these changes Jul 3, 2024

View reviewed changes

CohenIdo added 4 commits July 3, 2024 15:12

convert value to string

3b437e7

fix unit test

d647b69

Merge remote-tracking branch 'upstream/main' into report-all-csp-assets

30324c3

working and clean version

99906c4

joeypoon approved these changes Jul 4, 2024

View reviewed changes

fix code review comments

86ed5b5

CohenIdo mentioned this pull request Jul 4, 2024

[Cloud Security] Metering integration tests #187219

Merged

code review comments

b35d731

CohenIdo merged commit 3e9ad41 into elastic:main Jul 4, 2024
39 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Jul 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cloud Security][Metering] Report all assets details #186777

[Cloud Security][Metering] Report all assets details #186777

CohenIdo commented Jun 23, 2024 •

edited

Loading

CohenIdo Jun 23, 2024

CohenIdo Jun 23, 2024

elasticmachine commented Jun 24, 2024

kibana-ci commented Jul 2, 2024 •

edited

Loading

joeypoon left a comment

CohenIdo commented Jul 3, 2024 •

edited

Loading

eyalkraft left a comment

CohenIdo commented Jul 3, 2024

joeypoon left a comment

joeypoon Jul 4, 2024

CohenIdo Jul 4, 2024

elasticmachine commented Jul 4, 2024 •

edited

Loading

[Cloud Security][Metering] Report all assets details #186777

[Cloud Security][Metering] Report all assets details #186777

Conversation

CohenIdo commented Jun 23, 2024 • edited Loading

Summary

Example

CohenIdo Jun 23, 2024

Choose a reason for hiding this comment

CohenIdo Jun 23, 2024

Choose a reason for hiding this comment

elasticmachine commented Jun 24, 2024

kibana-ci commented Jul 2, 2024 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

History

joeypoon left a comment

Choose a reason for hiding this comment

CohenIdo commented Jul 3, 2024 • edited Loading

eyalkraft left a comment

Choose a reason for hiding this comment

CohenIdo commented Jul 3, 2024

joeypoon left a comment

Choose a reason for hiding this comment

joeypoon Jul 4, 2024

Choose a reason for hiding this comment

CohenIdo Jul 4, 2024

Choose a reason for hiding this comment

elasticmachine commented Jul 4, 2024 • edited Loading

⏳ Build in-progress

History

CohenIdo commented Jun 23, 2024 •

edited

Loading

kibana-ci commented Jul 2, 2024 •

edited

Loading

CohenIdo commented Jul 3, 2024 •

edited

Loading

elasticmachine commented Jul 4, 2024 •

edited

Loading