[Ingest Manager] [BUG] Using namespace with capital letters causes agent to not install beats/endpoint

[roncohen]

Kibana version: 7.9 BC(?)
Filed by @crowens

If you either add a new integration either through the 'add integration workflow' or through the 'add configuration' workflow and use a non-default namespace with capital letters, the agent does not finish installing the beats/endpoint and has the following logs messages:

{"log.level":"debug","@timestamp":"2020-08-04T17:31:16.049-0400","log.origin":{"file.name":"application/action_dispatcher.go","file.line":93},"message":"Failed to dispatch action 'action_id: 48199371-80e8-478a-8bfe-9013a6e9c356, type: CONFIG_CHANGE', error: failed to filter configuration: provided namespace is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[17]: unknown error","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-08-04T17:31:16.049-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":159},"message":"failed to dispatch actions, error: failed to filter configuration: provided namespace is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[17]: unknown error","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:16.049-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":162},"message":"FleetGateway is sleeping, next update in 1s","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:17.416-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":142},"message":"FleetGateway calling Checkin API","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:17.417-0400","log.origin":{"file.name":"kibana/client.go","file.line":170},"message":"Request method: POST, path: /api/ingest_manager/fleet/agents/6ede0f1a-9993-47bc-83b6-9c428649b49f/checkin","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.066-0400","log.origin":{"file.name":"application/action_dispatcher.go","file.line":81},"message":"Dispatch 1 actions of types: *fleetapi.ActionConfigChange","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.066-0400","log.origin":{"file.name":"application/handler_action_policy_change.go","file.line":23},"message":"handlerConfigChange: action 'action_id: 48199371-80e8-478a-8bfe-9013a6e9c356, type: CONFIG_CHANGE' received","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.067-0400","log.origin":{"file.name":"application/handler_action_policy_change.go","file.line":34},"message":"handlerConfigChange: emit configuration for action action_id: 48199371-80e8-478a-8bfe-9013a6e9c356, type: CONFIG_CHANGE","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.067-0400","log.origin":{"file.name":"application/emitter.go","file.line":39},"message":"Transforming configuration into a tree","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.067-0400","log.origin":{"file.name":"application/action_dispatcher.go","file.line":93},"message":"Failed to dispatch action 'action_id: 48199371-80e8-478a-8bfe-9013a6e9c356, type: CONFIG_CHANGE', error: failed to filter configuration: provided namespace is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[17]: unknown error","ecs.version":"1.5.0"}
{"log.level":"error","@timestamp":"2020-08-04T17:31:18.067-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":159},"message":"failed to dispatch actions, error: failed to filter configuration: provided namespace is invalid: /go/src/github.com/elastic/beats/x-pack/elastic-agent/pkg/agent/application/filters/stream_checker.go[17]: unknown error","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:18.067-0400","log.origin":{"file.name":"application/fleet_gateway.go","file.line":162},"message":"FleetGateway is sleeping, next update in 1s","ecs.version":"1.5.0"}
{"log.level":"debug","@timestamp":"2020-08-04T17:31:19.126-0400","log.logger":"service","log.origin":{"file.name":"service/service.go","file.line":54},"message":"Received sigterm/sigint, stopping","ecs.version":"1.5.0"}
{"log.level":"info","@timestamp":"2020-08-04T17:31:19.126-0400","log.origin":{"file.name":"application/managed_mode.go","file.line":253},"message":"Agent is stopped","ecs.version":"1.5.0"}

Steps to recreate:

1. Create a new agent configuration and choose a new namespace name that has capital letters:
   

2. Deploy an agent with that configuration, you will get the errors above. Beats never get installed/run. No logs in fleet view of agent:
   

I tried this 3 times. Two times with capital letters (namespace names of Newnamespace and TryAgain) and it failed. I then did a namespace of "again" and it worked. So, I am guessing capital letters.

I discussed with @jen-huang the option to limit the namespace input box to only allow lowercase. We also agree this was not a blocker for 7.9

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ruflin]

I missed that Elasticsearch only supports lower case index names. It is something we should also document around the indexing strategy and enforce on the agent side (if we don't already do it).

[jen-huang]

@ruflin I opened #75381 that implements relevant ES index name rules to the namespace string. We can add additional rules too, for example not allowing the string to start with -. I'm happy to make changes in this PR or we can iterate later.

[ruflin]

@jen-huang Sounds great.

@ph We need the same rules on the Agent side if they don't already exist.

@ycombinator More thing to enforce on the dataset name part.

[ph]

Added elastic/beats#20693 to add validation on top of the elastic agent.