[Response Ops][Alerting] ES query rule error Executing Rule default:.es-query has resulted in Error: Cannot set properties of undefined (setting 'order') - TypeError: Cannot set properties of undefined (setting 'order')

[ymao1]

Seen in the logs of the staging overview cluster. Looks like it's happening in the build_agg function

kibana/x-pack/plugins/triggers_actions_ui/common/data/lib/build_agg.ts

Line 117 in 0c0cf0a

aggParent.aggs.groupAgg.terms!.order = {

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[pmuellr]

Nice! I could have sworn I saw a message like in the logs recently, and had remembered while opening a different issue, so tried to find it again. But couldn't. I think I looked in qa and prod, but not staging 🤦🏻‍♂️

OHHHH .... I don't see it in serverless, only in ESS!

I checked prod, and it's happening with other fields as well. For searching, use message: "Cannot read properties of undefined" AND tags:".es-query" AND tags:"rule-run-failed"

[pmuellr]

Some of the fields/locations I've found over the last 90 days:

• type
• gte
• timeFieldName
• index
• actionId
• getFlapping
• scheduledAt
• meta
• fields
• title

I wouldn't be surprised if some of these were caused by calls to ES that went bad somehow - which are likely not as critical as when it's not an ES call that goes bad. So we may be able to defer fixing some of these, if needed.

type:

Executing Rule default:.es-query:0e2d43dd-27d2-4e69-bbac-5a9a0ebd1c45 has resulted in Error: Cannot read properties of undefined (reading 'type') - TypeError: Cannot read properties of undefined (reading 'type')
    at /usr/share/kibana/node_modules/@kbn/es-query/src/filters/build_filters/range_filter.js:103:58
    at /usr/share/kibana/node_modules/lodash/lodash.js:13469:38
    at /usr/share/kibana/node_modules/lodash/lodash.js:4967:15
    at baseForOwn (/usr/share/kibana/node_modules/lodash/lodash.js:3032:24)
    at mapValues (/usr/share/kibana/node_modules/lodash/lodash.js:13468:7)
    at buildRangeFilter (/usr/share/kibana/node_modules/@kbn/es-query/src/filters/build_filters/range_filter.js:103:34)
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:68:50)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:43:7)

gte:

Executing Rule default:.es-query:55024590-a989-11ee-8c16-dbbdc60ea0be has resulted in Error: Cannot read properties of undefined (reading 'gte') - TypeError: Cannot read properties of undefined (reading 'gte')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:70:130)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:41:7)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:84:8)

timeFieldName:

xecuting Rule default:.es-query:93138bc0-e76c-11ee-8563-83a10873f2c2 has resulted in Error: Cannot read properties of undefined (reading 'timeFieldName') - TypeError: Cannot read properties of undefined (reading 'timeFieldName')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:59:51)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:40:7)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:87:8)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:180:14)

index:

Executing Rule default:.es-query:74b1a3e2-f7f0-4d72-8d2f-74b996eba09f has resulted in Error: Cannot read properties of undefined (reading 'index') - TypeError: Cannot read properties of undefined (reading 'index')
    at /usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:151:21
    at Array.map (<anonymous>)
    at updateFilterReferences (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:150:26)
    at generateLink (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:131:26)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:46:16)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:82:26)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:168:14)

actionId:

Executing Rule default:.es-query:fa369db0-143c-11ed-8815-654215629985 has resulted in Error: Cannot read properties of undefined (reading 'actionId') - TypeError: Cannot read properties of undefined (reading 'actionId')
    at /usr/share/kibana/x-pack/plugins/actions/server/create_execute_function.js:150:38
    at Array.map (<anonymous>)
    at ActionsClient.execute [as bulkExecutionEnqueuer] (/usr/share/kibana/x-pack/plugins/actions/server/create_execute_function.js:149:65)

getFlapping:

Executing Rule default:.es-query:f99cfd60-e85a-11ee-ad8d-2120ea18016e has resulted in Error: Cannot read properties of undefined (reading 'getFlapping') - TypeError: Cannot read properties of undefined (reading 'getFlapping')
    at getAlertsForNotification (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/lib/get_alerts_for_notification.js:26:30)
    at LegacyAlertsClient.processAndLogAlerts (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_client/legacy_alerts_client.js:114:54)
    at AlertsClient.processAndLogAlerts (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/alerts_client/alerts_client.js:183:29)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:385:20

scheduledAt:

Executing Rule default:.es-query:2eed01d0-fca2-11ee-8f3a-2f515cdd4944 has resulted in Error: Cannot read properties of undefined (reading 'scheduledAt') - TypeError: Cannot read properties of undefined (reading 'scheduledAt')
    at savedObjectToConcreteTaskInstance (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:423:50)
    at /usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:116:49
    at Array.map (<anonymous>)
    at TaskStore.bulkSchedule (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_store.js:116:39)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at TaskScheduling.bulkSchedule (/usr/share/kibana/node_modules/@kbn/task-manager-plugin/server/task_scheduling.js:94:12)
    at ActionsClient.execute [as bulkExecutionEnqueuer] (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/create_execute_function.js:167:5)
    at ExecutionHandler.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/execution_handler.js:245:11)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:376:37

meta:

Executing Rule default:.es-query:2eed01d0-fca2-11ee-8f3a-2f515cdd4944 has resulted in Error: Cannot read properties of undefined (reading 'meta') - TypeError: Cannot read properties of undefined (reading 'meta')
    at /usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:47:54
    at Array.reduce (<anonymous>)
    at getBulkAuthorizationModeBySource (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/authorization/get_authorization_mode_by_source.js:43:47)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at ActionsClient.bulkEnqueueExecution (/usr/share/kibana/node_modules/@kbn/actions-plugin/server/actions_client.js:570:24)
    at ExecutionHandler.run (/usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/execution_handler.js:245:11)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:376:37

fields:

Executing Rule default:.es-query:9fdd3cdc-cd87-4846-89bf-0bf76bfa46c5 has resulted in Error: Cannot read properties of undefined (reading 'fields') - TypeError: Cannot read properties of undefined (reading 'fields')
    at updateSearchSource (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:61:23)
    at fetchSearchSourceQuery (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/lib/fetch_search_source_query.js:38:24)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/executor.js:78:26)
    at Object.executor (/usr/share/kibana/node_modules/@kbn/stack-alerts-plugin/server/rule_types/es_query/rule_type.js:162:14)
    at /usr/share/kibana/node_modules/@kbn/alerting-plugin/server/task_runner/task_runner.js:318:26

title:

Executing Rule default:.es-query:ce83a050-dcb8-11ee-8cfd-776c9fe84c58 has resulted in Error: Cannot read properties of undefined (reading 'title') - TypeError: Cannot read properties of undefined (reading 'title')
    at getSearchParamsFromRequest (/usr/share/kibana/src/plugins/data/common/search/search_source/fetch/get_search_params.js:44:32)
    at SearchSource.fetchSearch$ (/usr/share/kibana/src/plugins/data/common/search/search_source/search_source.js:431:58)
    at /usr/share/kibana/src/plugins/data/common/search/search_source/search_source.js:298:19
    at /usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/switchMap.js:17:35
    at OperatorSubscriber._this._next (/usr/share/kibana/node_modules/rxjs/dist/cjs/internal/operators/OperatorSubscriber.js:33:21)

[ymao1]

Error counts from the production logs

Bug with ES query multi-terms
Cannot set properties of undefined (setting 'order') - TypeError: Cannot set properties of undefined (setting 'order')
24,200 instances of this error across 2 clusters in the last week

Related to search source
Cannot read properties of undefined (reading 'type') - TypeError: Cannot read properties of undefined (reading 'type')
833,075 instances of this error spread out over hundreds of clusters in the last week

Cannot read properties of undefined (reading 'gte') - TypeError: Cannot read properties of undefined (reading 'gte')
10,355 instances of this error over 3 clusters in the last week

Cannot read properties of undefined (reading 'timeFieldName') - TypeError: Cannot read properties of undefined (reading 'timeFieldName')
8,006 instances of this error over 2 clusters in the last week

Cannot read properties of undefined (reading 'index') - TypeError: Cannot read properties of undefined (reading 'index')
5,993 instances of this error over 2 clusters in the last week

Cannot read properties of undefined (reading 'fields') - TypeError: Cannot read properties of undefined (reading 'fields')
0 instances of this error in the last week

Cannot read properties of undefined (reading 'title') - TypeError: Cannot read properties of undefined (reading 'title')
0 instances of this error in the last week

Accessing field from attributes returned by saved objects client - points to possible ES issues
Cannot read properties of undefined (reading 'actionId') - TypeError: Cannot read properties of undefined (reading 'actionId')
50 instances of this error over 3 clusters in the last week

Cannot read properties of undefined (reading 'scheduledAt') - TypeError: Cannot read properties of undefined (reading 'scheduledAt')
0 instances of this error in the last week

Cannot read properties of undefined (reading 'meta') - TypeError: Cannot read properties of undefined (reading 'meta')
0 instances of this error in the last week

Other
Cannot read properties of undefined (reading 'getFlapping') - TypeError: Cannot read properties of undefined (reading 'getFlapping')
0 instances of this error in the last week
Looking back over the last 90 days, we only see this error show up in version 8.11.1 so it seems it may have been fixed at some point

[ymao1]

ES query bug fixed with #182865
Search source timefield bug fixed with #182883

Created followup issues for the remaining search source errors #183068 and for the ES related errors #183072

Closing this issue in favor of those