set manager to "chef" on chef managed nodes #3840

vjeffrey · 2020-05-29T21:34:46Z

🔩 Description: What code changed, and why?
this change adds the manager type to the nodemanager process node flow, meaning
we can assign any node which is coming through ingest service the manager "chef" since we know that node is managed by chef
any node that is automate or aws-ec2 or azure-api etc will get the type on the processnode message, meaning that we dont have to look up the type of the manager based on the id (save an extra db call)

⛓️ Related Resources
#3515

👍 Definition of Done
manager type is on process node message
nodes managed by chef are labelled as such

👟 How to Build and Test the Change
rebuild compliance, ingest, nodemanager
load some nodes (chef_load_nodes, load_compliance_reports, chef_load_compliance_nodes, load_scan_jobs)
see the manager type for the nodes is correct!

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey · 2020-05-31T00:21:26Z

components/compliance-service/ingest/pipeline/publisher/node_manager_publisher.go

@@ -90,6 +90,10 @@ func gatherInfoForNode(in message.Compliance) (*manager.NodeMetadata, error) {
 		tags = append(tags, &common.Kv{Key: "environment", Value: in.Report.GetEnvironment()})
 	}

+	mgrType := in.Report.GetAutomateManagerType()
+	if in.Report.GetSourceFqdn() != "" {


if there's a value for the chef server, we know the node is managed by chef

If the value is "localhost" it is not managed by a chef server it is an effortless or chef solo report.

yaa, i was going with the idea that that node is still "managed" by chef in the sense that chef has converged on the node at least once/chef-client is on the node

whaddya think? should we distinguish between chef-server managed and just chef solo/effortless?

I don't think we should for now. But it looks like could when and if we want to.

vjeffrey · 2020-05-31T00:22:07Z

components/nodemanager-service/pgdb/node-managers-process-node.go

@@ -127,20 +127,12 @@ func (db *DB) ProcessIncomingNode(node *manager.NodeMetadata) error {
 		return errors.Wrap(err, "ProcessIncomingNode unable to marshal projects data")
 	}

-	var mgrType string
-	if node.ManagerId != "" {


this is the db call we can now remove since we have the type

vjeffrey · 2020-05-31T00:22:53Z

components/nodemanager-service/pgdb/node-managers.go

 				}
 			}
 		}
 	}
 	return nil
 }

-func isValidManagerType(managerType string) bool {


we had two functions basically looking at the same thing and i kept getting tripped up on it. figured it's better to just use one!

vjeffrey · 2020-05-31T00:23:42Z

components/nodemanager-service/pgdb/nodes.go

@@ -690,7 +690,7 @@ func validateNodeFilters(filters []*common.Filter) error {
 func isValidNodeManagerType(item string) bool {
 	switch item {
 	// empty string covers legacy nodes (nodes created before we started associating all manual nodes with "automate" as a manager and nodes brought over from a1)
-	case "aws-ec2", "aws-api", "azure-api", "azure-vm", "automate", "":
+	case "aws-ec2", "aws-api", "azure-api", "azure-vm", "automate", "gcp-api", "chef", "":


woops! we didn't have gcp-api listed as valid! it has been a valid option for a long time

lancewf · 2020-06-01T17:50:30Z

components/automate-ui/src/app/pages/+compliance/+scanner/state/scanner.effects.ts

@@ -45,7 +45,7 @@ export class ScannerEffects {
      const nodesStatusFilter = { key: 'status', values: ['reachable', 'unreachable', 'unknown'] };
      // ensures that we only collect nodes that belong to a manager


This comment is stale now because there are no nodes that don't have a manager now.

lancewf · 2020-06-01T18:08:50Z

components/nodemanager-service/pgdb/node-managers-process-node.go

-	if node.ManagerId != "" {
-		mgrType, err = db.SelectStr(sqlGetManagerTypeFromId, node.ManagerId)
-		if err != nil {
-			logrus.Warnf("unable to find manager for node")


Should we still warn here if node.GetManagerType() is empty?

hmm, good call ya

lancewf · 2020-06-01T18:11:57Z

components/nodemanager-service/pgdb/node-managers.go

-				if !isValidManagerType(item) {
-					return &errorutils.InvalidError{Msg: fmt.Sprintf("Invalid manager type filter: %s. manager_type must be one of the following: 'aws-ec2', 'aws-api', 'azure-api'", item)}
+				if !isValidNodeManagerType(item) {
+					return &errorutils.InvalidError{Msg: fmt.Sprintf("Invalid manager type filter: %s. manager_type must be one of the following: 'aws-ec2', 'aws-api', 'azure-api', 'chef', '', 'azure-vm', 'gcp-api' ", item)}


So, the manager type can still be empty?

no new ones should be empty, but we need to keep this around for the old nodes. they should get updated once a new run/scan comes thru but if there are no new runs/scans for the node it will be stuck in the empty manager state

lancewf · 2020-06-01T18:13:35Z

e2e/cypress/integration/api/nodemanager/node_manager_type.spec.ts

@@ -0,0 +1,38 @@
+import { uuidv4 } from '../../../support/helpers';


all your helper functions/the way you wrote the tests made it SO easy to write this one up, thank you!!

lancewf

Everything looks good. I just go hung up on after running "chef_load_compliance_nodes" and getting errors from the #3724 problem.

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey force-pushed the vj/mgr-fix branch from 80d29f4 to 68d29e1 Compare May 30, 2020 10:30

Victoria Jeffrey added 13 commits May 30, 2020 04:51

ingest infra reports before inspec reports

33f602b

Signed-off-by: Victoria Jeffrey <[email protected]>

add manager type to interservice proto

b6ef93b

Signed-off-by: Victoria Jeffrey <[email protected]>

ensure resolver includes manager type field

aed91d7

Signed-off-by: Victoria Jeffrey <[email protected]>

put manager type on message to nodemanager

29a2527

Signed-off-by: Victoria Jeffrey <[email protected]>

mgr type

f99f14d

Signed-off-by: Victoria Jeffrey <[email protected]>

expand allowed manager types

403584c

Signed-off-by: Victoria Jeffrey <[email protected]>

ensure we do not allow update of name or tags of chef node

7870f21

Signed-off-by: Victoria Jeffrey <[email protected]>

remove mgrtype lookup from process node code

141985f

Signed-off-by: Victoria Jeffrey <[email protected]>

ui: only get automate and node integration associated nodes

5bb25dc

Signed-off-by: Victoria Jeffrey <[email protected]>

test updates

33a7af3

Signed-off-by: Victoria Jeffrey <[email protected]>

cleanup

0bdbf00

Signed-off-by: Victoria Jeffrey <[email protected]>

add cypress api test

9af7025

Signed-off-by: Victoria Jeffrey <[email protected]>

lint fix

c8943b1

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey force-pushed the vj/mgr-fix branch from 68d29e1 to 0eb68f9 Compare May 30, 2020 10:51

expand allowed manager filter types; remove extra func

ab66e5e

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey force-pushed the vj/mgr-fix branch from 0eb68f9 to ab66e5e Compare May 30, 2020 11:18

set default sort for nodes list to last_contact

64e1a05

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey marked this pull request as ready for review May 31, 2020 00:17

vjeffrey commented May 31, 2020

View reviewed changes

vjeffrey requested a review from a team May 31, 2020 23:19

lancewf reviewed Jun 1, 2020

View reviewed changes

lancewf approved these changes Jun 1, 2020

View reviewed changes

warn if mgr field is empty

c8e93c4

Signed-off-by: Victoria Jeffrey <[email protected]>

vjeffrey merged commit e643c09 into master Jun 2, 2020

chef-expeditor bot deleted the vj/mgr-fix branch June 2, 2020 15:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

set manager to "chef" on chef managed nodes #3840

set manager to "chef" on chef managed nodes #3840

vjeffrey commented May 29, 2020 •

edited

Loading

vjeffrey May 31, 2020

lancewf Jun 1, 2020

vjeffrey Jun 1, 2020

vjeffrey Jun 1, 2020

lancewf Jun 1, 2020

vjeffrey May 31, 2020

vjeffrey May 31, 2020

vjeffrey May 31, 2020

lancewf Jun 1, 2020 •

edited

Loading

lancewf Jun 1, 2020

vjeffrey Jun 2, 2020

lancewf Jun 1, 2020

vjeffrey Jun 1, 2020

lancewf Jun 1, 2020

vjeffrey Jun 1, 2020

lancewf left a comment

		@@ -45,7 +45,7 @@ export class ScannerEffects {
		const nodesStatusFilter = { key: 'status', values: ['reachable', 'unreachable', 'unknown'] };
		// ensures that we only collect nodes that belong to a manager

		@@ -0,0 +1,38 @@
		import { uuidv4 } from '../../../support/helpers';

set manager to "chef" on chef managed nodes #3840

set manager to "chef" on chef managed nodes #3840

Conversation

vjeffrey commented May 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lancewf Jun 1, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lancewf left a comment

Choose a reason for hiding this comment

vjeffrey commented May 29, 2020 •

edited

Loading

lancewf Jun 1, 2020 •

edited

Loading