Compliance ingestion for minimal reports #3302

alexpop · 2020-04-07T16:16:37Z

This change allows compliance reports to be ingested without the profile metadata attached to them, as long as the backend has the profile metadata.

A new report has been added in the tests that will be rejected during ingestion because the profile ID being used is unknown to the backend.

test_data/audit_reports/2018-03-04_redhat(2)-beta-non-ingestable.json

This report has been converted into a hybrid, where one profile has the metadata in it, another is missing it:

test_data/audit_reports/2018-03-05_centos(2)-beta-nginx(f)-apache(f)-failed.json

Fixes #3087

I did an end-to-end test of this change by using the audit cookbook code in this branch:
https://github.com/chef-cookbooks/audit/compare/ap/support-metasearch

I created an Automate instance from this code. Rebuilt components/compliance-service and components/automate-gateway. Imported from Compliance > Profiles > Available the DevSec Apache Baseline v2.1.0 (w/ sha256 ID: 73eba2f2c12a6014cf0880c1e7d990bb783346a92d0d46284aaf785b8206cd39) profile.

Ran the audit cookbook with the following profiles:

73eba2f2c12a6014cf0880c1e7d990bb783346a92d0d46284aaf785b8206cd39
09adcbb3b9b3233d5de63cd98a5ba3e155b3aaeb66b5abed379f5fb1ff143988
0704c70df5f251b31b4741e0310d8b69ccf7db05e8a97c3ca0a3422335bdff88

In the first run, the Audit cookbook Info logs printed:

[2020-04-08T10:05:59+00:00] INFO: Reporting to chef-automate
[2020-04-08T10:06:00+00:00] INFO: Automate is missing metadata for the following profile ids: ["09adcbb3b9b3233d5de63cd98a5ba3e155b3aaeb66b5abed379f5fb1ff143988", "0704c70df5f251b31b4741e0310d8b69ccf7db05e8a97c3ca0a3422335bdff88"]

and sent a report where the metadata of the apache-baseline profile was stripped away as the backend already has it. The report can be seen here: report-with-profiles-2-full-1-stripped.json.txt

I verified in Automate and the report was ingested correctly with the three profiles.

In the second run of the Audit cookbook with the same profiles, the report being sent had all three profiles now being stripped of the metadata information. The report can be seen here: report-with-profiles-3-stripped.json.txt

rickmarry

Just did peer review of this with @alexpop.. Awesome work! thanks @alexpop!

alexpop · 2020-04-08T08:49:59Z

components/compliance-service/ingest/events/compliance/helper_methods.go

-			for tKey, tValue := range control.Tags.Fields {
-				if newStringTag := relaxting.StringTagsFromProtoFields(tKey, tValue); newStringTag != nil {
-					stringTags = append(stringTags, *newStringTag)
+			if control.Tags != nil {


Made some small fixes to helper_methods.go after trying to ingest the minimal report without any changes to ingestion. Makes ingestion more resilient to malformed or malicious reports.

alexpop · 2020-04-08T08:53:13Z

components/compliance-service/api/tests/03_reports_spec.rb

@@ -577,9 +577,31 @@ def reporting
              "waived" => {},
              "total" => 18
            }
+          },


When we introduced tags filtering, we manually updated some reports and added tags to them. But we didn't add the tags across all profiles with the same ID. Because of this, the minimal report that is being ingested now picked up the tags from the compliance-profiles store as they were saved there when the profile was ingested first. Because of this I had to update this test.

alexpop · 2020-04-08T08:54:48Z

components/compliance-service/ingest/ingestic/mappings/comp-rep-date.go

@@ -543,6 +543,9 @@ var ComplianceRepDate = Mapping{
 					"type": "keyword",
 					"doc_values": false,
 					"ignore_above": 256
+				},
+				"run_time_limit": {


This is a new field the audit cookbook/inspec is going to provide when minimizing reports and reducing the start_time and run_time for each result.

alexpop · 2020-04-08T10:27:36Z

components/compliance-service/ingest/pipeline/publisher/compliance.go

@@ -116,18 +121,8 @@ func insertInspecProfile(msg message.Compliance, profile *relaxting.ESInspecProf
 			"profile_id": profile.Sha256,
 		}).Debug("Ingesting inspec_profile")
 		start := time.Now()
-		profileExists, err := client.ProfileExists(profile.Sha256)


We used to check for profile existence for each profile coming in the report. Reports can contain one or multiple profiles. The logic has been changed and we only query once elasticsearch asking for all the profiles in the report at once with the ProfilesMissing method

alexpop · 2020-04-08T10:32:07Z

components/compliance-service/reporting/relaxting/reports.go

@@ -284,18 +284,21 @@ func (backend *ES2Backend) GetReports(from int32, size int32, filters map[string
 	reports := make([]*reportingapi.Report, 0)
 	if searchResult.TotalHits() > 0 && searchResult.Hits.TotalHits > 0 {
 		for _, hit := range searchResult.Hits.Hits {
-			item := ESInSpecSummary{}
+			item := ESInSpecReport{}


Spotted this with Rick on peer reviewing and decided to switch to the right Struct for this Unmarshalling to avoid problems in the future.

alexpop · 2020-04-08T10:35:07Z

api/external/compliance/profiles/profiles.proto

@@ -142,6 +142,19 @@ service ProfilesService {
 			action: "compliance:profiles:list"
 		};
 	};
+


Added docs for the /metasearch API used by this improvement

alexpop · 2020-04-08T10:36:28Z

...nce-service/test_data/audit_reports/2018-03-05_centos(2)-beta-nginx(f)-apache(f)-failed.json

  "profiles": [
    {
-      "name": "nginx-baseline",


Converted this report into a hybrid one where one of its profiles is stripped of the metadata.

vjeffrey · 2020-04-08T22:26:51Z

components/compliance-service/ingest/pipeline/processor/profile.go

+}
+
+func complianceProfile(in <-chan message.Compliance, client *ingestic.ESClient) <-chan message.Compliance {
+	//ctx := context.Background()


nitpick: seems like we can remove this comment line

vjeffrey

code looks good!
~~would love to get a link to the audit cookbook branch so i can test all the way thru with kitchen!~~
silly me! you linked the audit cb here https://github.com/chef-cookbooks/audit/compare/ap/support-metasearch -- ill test it out

alexpop · 2020-04-08T22:31:21Z

Thank you for the reviews. Audit cookbook branch to use this is ap/support-metasearch

https://github.com/chef-cookbooks/audit/compare/ap/support-metasearch

vjeffrey

tested 3x with audit cb, with profiles that existed in automate and did not, worked great!!!

Signed-off-by: Alex Pop <[email protected]>

…tion Signed-off-by: Alex Pop <[email protected]>

Signed-off-by: Alex Pop <[email protected]>

alexpop · 2020-04-09T08:36:21Z

Removed //ctx := context.Background() and rebased with latest master. Will merge once green

alexpop added the WIP label Apr 7, 2020

alexpop requested a review from a team April 7, 2020 16:16

alexpop force-pushed the ap/ingestion-4-min-reports branch from 2d6a117 to efaf0fb Compare April 7, 2020 17:12

alexpop mentioned this pull request Apr 7, 2020

Add compliance profile metasearch API endpoint #3288

Merged

alexpop force-pushed the ap/ingestion-4-min-reports branch from efaf0fb to 7da261b Compare April 7, 2020 19:36

rickmarry approved these changes Apr 7, 2020

View reviewed changes

alexpop force-pushed the ap/ingestion-4-min-reports branch from 7da261b to 4e79968 Compare April 7, 2020 22:02

alexpop commented Apr 8, 2020

View reviewed changes

alexpop changed the title ~~WIP: Compliance ingestion for minimal reports~~ Compliance ingestion for minimal reports Apr 8, 2020

alexpop removed the WIP label Apr 8, 2020

vjeffrey reviewed Apr 8, 2020

View reviewed changes

vjeffrey approved these changes Apr 8, 2020

View reviewed changes

alexpop and others added 16 commits April 9, 2020 09:05

Make fixes to avoid crashes on min reports

85e8864

Signed-off-by: Alex Pop <[email protected]>

Give profile pipe es access

ae41cf7

Signed-off-by: Alex Pop <[email protected]>

Almost done with the min profiles ingestion

29a3b48

Signed-off-by: Alex Pop <[email protected]>

update repProfile...

104ef8b

Signed-off-by: Alex Pop <[email protected]>

Profile enrichment from profile store and add run_time_limit

7a9c3c4

Signed-off-by: Alex Pop <[email protected]>

Generate protos for the new run_time_limit field and fix hybrid inges…

c9d964c

…tion Signed-off-by: Alex Pop <[email protected]>

Hybrid ingestion works!

691f1f4

Signed-off-by: Alex Pop <[email protected]>

Add more comments bring control impact in

972d5a6

Signed-off-by: Alex Pop <[email protected]>

Fix reporting on ingestion errors

a1b7e2e

Signed-off-by: Alex Pop <[email protected]>

Add ingestion integration tests for the hybrid ingestion

e49c587

Signed-off-by: Alex Pop <[email protected]>

Check ipaddress and rename bad ingestion node

8a715d8

Signed-off-by: Alex Pop <[email protected]>

Revert apache

6be6c49

Signed-off-by: Alex Pop <[email protected]>

Use hybrid report for 03-05

0966004

Signed-off-by: Alex Pop <[email protected]>

Update test data and tests

01ffc61

Signed-off-by: Alex Pop <[email protected]>

Update metasearch API documentation

6c8a28e

Signed-off-by: Alex Pop <[email protected]>

Don't use refs as we will remove them from comp rep indices

7daf356

Signed-off-by: Alex Pop <[email protected]>

alexpop added 3 commits April 9, 2020 09:05

Rollback renaming of getStructArray

42c6f68

Signed-off-by: Alex Pop <[email protected]>

Move compliance.FixInheritedProfiles after profiles are enritched

13ad289

Signed-off-by: Alex Pop <[email protected]>

Remove comment

ecee466

Signed-off-by: Alex Pop <[email protected]>

alexpop force-pushed the ap/ingestion-4-min-reports branch from 4e79968 to ecee466 Compare April 9, 2020 08:05

alexpop merged commit 02011e9 into master Apr 9, 2020

chef-expeditor bot deleted the ap/ingestion-4-min-reports branch April 9, 2020 08:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compliance ingestion for minimal reports #3302

Compliance ingestion for minimal reports #3302

alexpop commented Apr 7, 2020 •

edited

Loading

rickmarry left a comment

alexpop Apr 8, 2020

alexpop Apr 8, 2020

alexpop Apr 8, 2020

alexpop Apr 8, 2020

alexpop Apr 8, 2020

alexpop Apr 8, 2020

alexpop Apr 8, 2020

vjeffrey Apr 8, 2020

vjeffrey left a comment •

edited

Loading

alexpop commented Apr 8, 2020

vjeffrey left a comment

alexpop commented Apr 9, 2020

Compliance ingestion for minimal reports #3302

Compliance ingestion for minimal reports #3302

Conversation

alexpop commented Apr 7, 2020 • edited Loading

rickmarry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vjeffrey left a comment • edited Loading

Choose a reason for hiding this comment

alexpop commented Apr 8, 2020

vjeffrey left a comment

Choose a reason for hiding this comment

alexpop commented Apr 9, 2020

alexpop commented Apr 7, 2020 •

edited

Loading

vjeffrey left a comment •

edited

Loading