[Alerting] Log more descriptive error messages when runtime field mappings are updated to be incompatible with original query

[ymao1]

Based on this investigation, we have determined that when runtime field mappings are updated to be incompatible with the originally defined type (date to keyword for a field used in a time range query for example, the index-threshold and es-query stack rule executions will fail with a search_phase_execution_exception that is logged in the event log. The actual error coming back from ES is usually more descriptive. It would be helpful to capture these more descriptive error messages to aid in debugging rule execution failures.

Some examples of the errors coming back from ES:

{
  "type": "query_shard_exception",
  "reason": "failed to create query: For input string: \"2021-03-25T18:35:54.545Z\"",
  "index_uuid": "znV1kqQrTEuOWgad1KOBBw",
  "index": "es-apm-sys-sim",
  "caused_by": {
  "type": "number_format_exception",
    "reason": "For input string: \"2021-03-25T18:35:54.545Z\""
  }
}

{
  "type": "illegal_argument_exception",
  "reason": "Field [second_timestamp] of type [keyword] does not support custom formats",
  "caused_by": {
    "type": "illegal_argument_exception",
    "reason": "Field [second_timestamp] of type [keyword] does not support custom formats"
  }
}

{
  "type": "illegal_argument_exception",
  "reason": "Field [free_memory] of type [keyword] is not supported for aggregation [avg]",
  "caused_by": {
    "type": "illegal_argument_exception",
    "reason": "Field [free_memory] of type [keyword] is not supported for aggregation [avg]"
  }
}

[elasticmachine]

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

[mikecote]

+1. I've seen this issue for other types of rules and it's hard to debug (example).