Add central management service #8263

exekias · 2018-09-07T09:26:30Z

Implement Central Management service. This service will be in charge of polling new configurations from Kibana and apply the changes across all Reloadable objects. See #8205 for details on that.

Part of #7028

Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong.

exekias · 2018-09-13T20:04:52Z

this is ready for review

sayden · 2018-09-14T09:15:57Z

x-pack/libbeat/management/manager.go

+}
+
+// Start the config manager
+func (cm *ConfigManager) Start() {


It's helpful to name methods and functions which launches new goroutines on their bodies with a specify suffix like StartAsync or StartGoroutine. This way, future code readers won't assume synchronous work

I was also thinking that it's not a bad idea to leave it with Start but make it a blocked call. The effect is similar to the one you get with http.ListenAndServe and the caller gets also a clear idea that he/she have some dancing gouroutines after this call (because he/she will use go Start() in the code)

This implements our Runner interface, with Start/Stop methods. In general I think we follow this style: Start is non blocking, sometimes we use Run as it's blocking counterpart

sayden · 2018-09-14T09:17:31Z

libbeat/management/management.go

+
+// GetFactory retrieves config manager constructor. If no one is registered
+// it will create a nil manager
+func GetFactory() Factory {


It's not idiomatic to use the prefix Get for getter like code in Go. See https://golang.org/doc/effective_go.html#Getters

sayden · 2018-09-14T09:34:16Z

libbeat/common/reload/reload.go

 	if err := r.RegisterList(name, list); err != nil {
 		panic(err)
 	}
 }

 // Get returns the reloadable object with the given name, nil if not found
-func (r *registry) Get(name string) Reloadable {
+func (r *Registry) Get(name string) Reloadable {


Beware that a if a caller uses variable name registry this call will look like registry.Get() and it will look like it's going to return a new Registry. This is one of the cases where GetReloadable would be better or simply Reloadable if the field isn't embedded.

kvch · 2018-09-14T10:55:12Z

x-pack/libbeat/management/manager.go

+}
+
+// CheckRawConfig check settings are correct to start the beat
+func (cm *ConfigManager) CheckRawConfig(cfg *common.Config) error {


If this function does not return error does it mean the whole configuration is correct? Is this check is going to be superior to the current test config?

I think I missed a big TODO comment here 😄

This method is minded to check that the user hasn't configured something that is supposed to be managed my Kibana. In a next PR I will add that code. BTW the test config point is good, will need to make sure this check is run there too.

I'm adding a TODO comment now 👍

pushed dbaf5af

* Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog

* Beats enrollment subcommand (elastic#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of elastic#7028, closes elastic#7032 * Add API client to retrieve configurations from CM (elastic#8155) * Add central management service (elastic#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (elastic#8378) * Also send beat name when enrolling (elastic#8380) * Refactor how configs are stored (elastic#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (elastic#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (elastic#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3)

* Add Central Management feature (#8559) * Beats enrollment subcommand (#7182) This PR implements intial enrollment to Central Management in Kibana. After running the enrollment command, beats will have a valid access token to use when retrieving configurations. To test this: - Use the following branches: - Elasticsearch: https://github.com/ycombinator/elasticsearch/tree/x-pack/management/beats - Kibana: https://github.com/elastic/kibana/tree/feature/x-pack/management/beats - Retrieve a valid enrollment token: ``` curl \ -u elastic \ -H 'kbn-xsrf: foobar' \ -H 'Content-Type: application/json' \ -X POST \ http://localhost:5601/api/beats/enrollment_tokens ``` - Use it: ``` <beat> enroll http://localhost:5601 <enrollment_token> ``` - Check agent is enrolled: ``` curl http://localhost:5601/api/beats/agents | jq ``` This is part of #7028, closes #7032 * Add API client to retrieve configurations from CM (#8155) * Add central management service (#8263) * Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong. * Register output for reloading (#8378) * Also send beat name when enrolling (#8380) * Refactor how configs are stored (#8379) * Refactor configs storage to avoid YAML issues * Refactor manager loop to avoid repeated code * Use beat name var when registering confs (#8435) This should make Auditbeat or any other beat based on Metricbeat have their own namespace for confs * Allow user/passwd based enrollment (#8524) * Allow user/passwd based enrollment This allows to enroll using the following workflow: ``` $ <beat> enroll http://kibana:5601 --username elastic Enter password: Enrolled and ready to retrieve settings from Kibana ``` It also allows to pass the password as an env variable: ``` PASS=... $ <beat> enroll http://kibana:5601 --username elastic --password env:PASS Enrolled and ready to retrieve settings from Kibana ``` * Fix some strings after review comments * Add changelog (cherry picked from commit 4247bc3) * Fix monitoring registry usage

* Add config manager initial skeleton Config manager will poll configs from Kibana and apply them locally. It must be started with the beat. In order to check the user is not trying to override configurations provided by central management, the Config Manager can check the exisitng configuration and return errors if something is wrong.

Carlos Pérez-Aradros Herce added 7 commits September 6, 2018 17:34

Implement configs polling mechanism

3d4192a

Register metricbeat.modules config

1515ddc

Apply config changes when a new configuration is retrieved

0ae4c8a

Fix defaults

79ede93

Update to latest changes in the API

6cfcb83

Adapt to latest changes in API and master

cf81bd5

exekias added in progress Pull request is currently in progress. libbeat labels Sep 7, 2018

Carlos Pérez-Aradros Herce added 4 commits September 11, 2018 18:37

Ignore kibana version

7d14711

Pass reloadable registry as a variable

a547ecb

Add config comparison tests

95299fb

Add tests

5a3bf53

exekias force-pushed the cm-manager branch from 40e5052 to 5a3bf53 Compare September 13, 2018 18:24

exekias added review and removed in progress Pull request is currently in progress. labels Sep 13, 2018

kvch self-requested a review September 14, 2018 08:40

sayden reviewed Sep 14, 2018

View reviewed changes

kvch reviewed Sep 14, 2018

View reviewed changes

Carlos Pérez-Aradros Herce added 3 commits September 14, 2018 13:30

Add TODO message

dbaf5af

Use better name for Factory getter

c1b20dc

Better names for reloadable registry getters

598298a

exekias mentioned this pull request Sep 14, 2018

Central Management #7028

Closed

26 tasks

kvch approved these changes Sep 19, 2018

View reviewed changes

exekias merged commit 9876164 into elastic:feature/centralmgmt Sep 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add central management service #8263

Add central management service #8263

exekias commented Sep 7, 2018 •

edited

Loading

exekias commented Sep 13, 2018

sayden Sep 14, 2018 •

edited

Loading

sayden Sep 14, 2018 •

edited

Loading

exekias Sep 14, 2018

sayden Sep 14, 2018

sayden Sep 14, 2018 •

edited

Loading

kvch Sep 14, 2018

exekias Sep 14, 2018

exekias Sep 14, 2018

Add central management service #8263

Add central management service #8263

Conversation

exekias commented Sep 7, 2018 • edited Loading

exekias commented Sep 13, 2018

sayden Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

sayden Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

exekias Sep 14, 2018

Choose a reason for hiding this comment

sayden Sep 14, 2018

Choose a reason for hiding this comment

sayden Sep 14, 2018 • edited Loading

Choose a reason for hiding this comment

kvch Sep 14, 2018

Choose a reason for hiding this comment

exekias Sep 14, 2018

Choose a reason for hiding this comment

exekias Sep 14, 2018

Choose a reason for hiding this comment

exekias commented Sep 7, 2018 •

edited

Loading

sayden Sep 14, 2018 •

edited

Loading

sayden Sep 14, 2018 •

edited

Loading

sayden Sep 14, 2018 •

edited

Loading