add projects to processNode

[vjeffrey]

User Story

Both the ingest service and compliance ingestion are now calling NodeManager.ProcessNode on report ingestion.
We need to add projects: []string{} to the ProcessNode message and ensure that both of the ingestion services are passing that information through, so that those nodes will have the correct project definitions.

Also, how do we keep these nodes in sync when there is an ingest rules change that runs through the update process?

Definition of Done

Projects on nodes between ingest, compliance, and nodemanager are in sync.

[lancewf]

The BuildNodeManagerPublisher runs after the BuildRunProjectTagger so the node has the correct projects.

	chefRunPipeline(in,
		processor.BuildTransmogrify(9),
		processor.ChefRunCorrections,
		processor.BuildRunProjectTagger(authzClient), <--- 
		publisher.BuildNodeManagerPublisher(nodeMgrClient),
		processor.BuildRunMsgToBulkRequestTransformer(client),
		publisher.BuildBulkRunPublisher(client),
		processor.CountRuns(&counter),
	)

https://github.com/chef/automate/blob/master/components/ingest-service/pipeline/chef_run.go#L33

For the update process, we can either add the node-manager-service to project update process or have compliance kill off the update process.
We will have to ensure that the node manager stores all the fields necessary for the project ingest rules.

[vjeffrey]

what if there was an endpoint, only exposed internally, SetProjectsOnNode in the manager.proto -- that would take a map of string[]string (i.e. node id[]string{project1, project2}) - that the update process could call - would that work? or no b/c of how the process is executing via painless script?

[lancewf]

How we currently have the ingest-service and compliance-service project update working it could not call that endpoint because the update process is running in elasticsearch with the painless script.

[vjeffrey]

ok, so we'll have to store the info. coooolz....i'll gotta think a bit on how to do this

[vjeffrey]

had some time on the airplane to sort this out.

• add projects ([]string) to process node call
• move the compliance ingestion process node call to publisher to i have those project tags (maybe this means i can also remove the node status?)
• nodes tagged with projects info as part of process node call
• add projects_data to process node call: map[string][]string and store that in db as json
• open a separate ticket for adding the nodemanager to the update manager. can update the nodes with an update nodes set projects = $1 where projects_data ->> 'environment' AND...

[vjeffrey]

https://github.com/chef/automate/compare/vj/process-node-projects-data