Infinite loop trying to check alerts index patterns after reinstalling Wazuh indexer

[vicferpoy]

Wazuh
4.3.5-rc1

Browser
Chrome

Description
During the manual tests for the Wazuh indexer (v4.3.5-rc1), I noticed a strange error when reinstalling the Wazuh indexer. The Wazuh dashboard would try to check alerts index patterns until its service would restart.

Preconditions

1. To have a Wazuh stack installed. In my case, Wazuh dashboard, single-node indexer and single-node server.

Steps to reproduce

1. I had a Wazuh stack deployed and working.
2. I reinstalled the Wazuh indexer.
3. The Wazuh dashboard could not check the alerts index patterns and would loop forever. No additional information could be retrieved.
4. I restarted the Wazuh dashboard service. Everything went back to normal.

Screenshots

Screenshare.-.2022-06-28.1_24_59.PM.mp4

This issue was detected here: wazuh/wazuh#13994 (comment)

[Desvelao]

Research

How to replicate

1. Using a Wazuh stack deployment (Wazuh dashboard, Wazuh indexer, Wazuh server), configure and start de the components. The Wazuh indexer should be in an independent machine.
2. Access to Wazuh dashboard and navigate to Wazuh plugin, configure the API host to work as expected. Secure the index pattern related to Wazuh are created. Keep the browser tab in the Wazuh plugin.
3. Remove, reinstall and start the Wazuh indexer using the Wazuh install assistant following the next steps:

sudo bash wazuh-install.sh -u
sudo bash wazuh-install.sh --wazuh-indexer indexer
sudo bash wazuh-install.sh --start-cluster

4. Returns to the browser tab that has the Wazuh plugin and navigates to another plugin section. The plugin health check should run and it should finish each check suscessfully, then it should redirect to /overview plugin path and instantly should redirect to the /health-check path again.

Digging the cause

Remove and reinstall the Wazuh indexer causes the previous data is removed.

When the plugin health check finish and redirects to /overview endpoint, the getIp route resolver is executed and there is a problem when getting the saved objects of index-pattern type using the saved objects frontend client that doesn't return any data despite the index patterns were created in the health check. This causes a redirection to the /health-check route because of this.

It is a little strange that any data is returned from find from the saved object client but other related endpoints used by the alerts index pattern check in the plugin health check are getting the expected data without problems.

The find's method of the saved object does a request to:

GET <PLUGIN_PLATFORM>/api/saved_objects/_find?type=index-pattern&perPage=10000&fields=title

Other side effects caused by this situation

Plugin related to the plugin platform fails:

• Stack management/Index patterns: no index pattern data
  
  
  

• Stack management/Saved objects: has an error
  

• Discover: accessing to this plugin redirects to Stack management/Index patterns.

Solutions

• Restart the Wazuh dashboard service fixes the Wazuh plugin loop and the problems in other plugins.

[Desvelao]

Researching a fix

I was researching because the find's method of the saved object client can't return any data related to the index pattern but I could find any hit of problem or if there is a bug or intentional.

We could replace or adapt the use of find's method of the saved object client by:

• the get method of the saved object client:

// something like:
savedObjectsClient.get('index-pattern', patternID, ['title']);

• use another endpoint how is done in the health check's checks:

GET /api/saved_objects/index-pattern/{patternID}?fields=title&fields=fields

but this doesn't solve the problem with the malfunction of other plugins. To fix them, it is required to restart the Wazuh dashboard service.

Moreover, the Wazuh plugin could have more problems because of trusting to find's method of the saved object client.

Because of this issue appears in a situation whose use flow is not expected (by others core plugins), and this causes an exceptional malfunction of the find's method of the saved objects client, it is hard to capture this situation and display some type of error that informs to the user.

[Desvelao]

Research

The find's method of the saved object's client does a request to GET <PLUGIN_PLATFORM>/api/saved_objects/_find endpoint. This returns the result of this method. Reviewing the source code of endpoint, I saw that there is a hardcoded response when there is an error with status code 404. I was researching and when the issue happens, the endpoint replies with a status code of 200, not 404, so the response could be related to the search query in the index pattern.

Digging the search query in GET <PLUGIN_PLATFORM>/api/saved_objects/_find

The query is:

{
  "index": [
    ".kibana"
  ],
  "size": 10000,
  "from": 0,
  "_source": [
    "index-pattern.title",
    "namespace",
    "namespaces",
    "type",
    "references",
    "migrationVersion",
    "updated_at",
    "originId",
    "title"
  ],
  "rest_total_hits_as_int": true,
  "preference": undefined,
  "body": {
    "seq_no_primary_term": true,
    "query": {
      "bool": {
        "filter": [
          {
            "bool": {
              "must": undefined,
              "should": [
                {
                  "bool": {
                    "must": [
                      {
                        "term": {
                          "type": "index-pattern"
                        }
                      }
                    ],
                    "should": [
                      {
                        "bool": {
                          "must_not": [
                            {
                              "exists": {
                                "field": "namespace"
                              }
                            }
                          ]
                        }
                      }
                    ],
                    "minimum_should_match": 1,
                    "must_not": [
                      {
                        "exists": {
                          "field": "namespaces"
                        }
                      }
                    ]
                  }
                }
              ],
              "minimum_should_match": 1
            }
          }
        ]
      }
    }
  }
}

Note that there is a term query in the type field. This field should have the mapping of type keyword so the query could match.

The reason the endpoint return an empty list is because after reinstalling Wazuh indexer, all the indices are removed. If the user navigates in the Wazuh plugin, a new .kibana<tenant> index is created, but this doesn't have the expected field mappings to the search query work as expected. This does the above search query doesn't matches with the documents despite the health check of Wazuh plugin could have created the index patterns' saved objects in the .kibana<tenant> index.

Indices after reinstalling Wazuh indexer:

$ curl -k -u admin:<password> https://<wazuh_indexer_ip>:9200/_cat/indices
green open .opendistro_security bY8RZSn1RXGw5otbRQSV7Q 1 0 9 9 90.2kb 90.2kb

If the user navigates to the Wazuh plugin after reinstalling Wazuh indexer, the getIp route resolver gets an empty list of index patterns.

Indices after navigating in the Wazuh plugin, and it runs the health check passing (the index patterns's saved objects are created suscessfully):

$ curl -k -u admin:<password> https://<wazuh_indexer_ip>:9200/_cat/indices
green open .kibana_92668751_admin_1  KUYj4WPSQFKxmhMdrrZEXQ 1 0 4 7 105.2kb 105.2kb
green open wazuh-statistics-2022.30w Yj6III8WQG6xb8AcCfeX6Q 1 0 4 0  24.5kb  24.5kb
green open .opendistro_security      bY8RZSn1RXGw5otbRQSV7Q 1 0 9 9  90.2kb  90.2kb

We could see that a .kibana<tenant> index exists (.kibana_92668751_admin_1`). If we see the information about that index:

.kibana_92668751_admin_1 information

$ curl -k -u admin:f4PS7CUa6Lp0bke+3TlbjC?4AVu+6Opp https://192.168.56.4:9200/.kibana_92668751_admin_1?pretty
{
  ".kibana_92668751_admin_1" : {
    "aliases" : {
      ".kibana_92668751_admin" : { }
    },
    "mappings" : {
      "properties" : {
        "config" : {
          "properties" : {
            "buildNum" : {
              "type" : "long"
            },
            "defaultIndex" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "metaFields" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timelion:max_buckets" : {
              "type" : "long"
            },
            "timepicker:timeDefaults" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "index-pattern" : {
          "properties" : {
            "fieldFormatMap" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "fields" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "sourceFilters" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timeFieldName" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "title" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "migrationVersion" : {
          "properties" : {
            "config" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "index-pattern" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "type" : {
          "type" : "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        },
        "updated_at" : {
          "type" : "date"
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".kibana_92668751_admin_1",
        "creation_date" : "1659088153015",
        "number_of_replicas" : "0",
        "uuid" : "KUYj4WPSQFKxmhMdrrZEXQ",
        "version" : {
          "created" : "135238227"
        }
      }
    }
  }
}

The field mapping for type is:

"type" : {
  "type" : "text",
  "fields" : {
    "keyword" : {
      "type" : "keyword",
      "ignore_above" : 256
    }
  }
}

This means there are these fields:

• type: type text
• type.keyword: type keyword

If we reviews the query done by the endpoint, we could see to do a term query in the type field, but if this index is used, it won't match the result. The term query applies to the field of type keyword not text as it is the case. This causes the endpoint returns an empty list and the Wazuh plugin [redirects](https://github.com/wazuh/wazuh-kibana-app/blob/v4.3.6-1.2.0-wzd/public/services/resolves/get-ip.js#L59-L70) to the /health-check` because can not find the saved objects of the index patterns.

If we restart the Wazuh dashboard service, the indices are:

$ curl -k -u admin:<password> https://<wazuh_indexer_ip>:9200/_cat/indices
green open wazuh-monitoring-2022.30w SYP9bV7sS5WU16b66KIKkQ 1 0 1 0    17kb    17kb
green open .kibana_92668751_admin_1  KUYj4WPSQFKxmhMdrrZEXQ 1 0 4 3 106.2kb 106.2kb
green open wazuh-statistics-2022.30w Yj6III8WQG6xb8AcCfeX6Q 1 0 4 0  24.5kb  24.5kb
green open .kibana_92668751_admin_2  7V1pxjOmS8uVOQcB85V3Jg 1 0 4 0  14.1kb  14.1kb
green open .kibana_1                 cS1_vKEgQci2JSIC471u3A 1 0 0 0    208b    208b
green open .opendistro_security      bY8RZSn1RXGw5otbRQSV7Q 1 0 9 8    91kb    91kb

A new tenant .kibana<tenant> index is created: .kibana_92668751_admin_2. The field mapping for this index is:

.kibana_92668751_admin_2 information

$ curl -k -u admin:<password> https://<wazuh_indexer_ip>:9200/.kibana_92668751_admin_2?pretty
{
  ".kibana_92668751_admin_2" : {
    "aliases" : {
      ".kibana_92668751_admin" : { }
    },
    "mappings" : {
      "dynamic" : "strict",
      "_meta" : {
        "migrationMappingPropertyHashes" : {
          "application_usage_daily" : "43b8830d5d0df85a6823d290885fc9fd",
          "visualization" : "f819cf6636b75c9e76ba733a0c6ef355",
          "references" : "7997cf5a56cc02bdc9c93361bde732b0",
          "query" : "11aaeb7f5f7fa5bb43f25e18ce26e7d9",
          "ui-metric" : "0d409297dc5ebe1e3a1da691c6ee32e3",
          "application_usage_transactional" : "3d1b76c39bfb2cc8296b024d73854724",
          "type" : "2f4316de49999235636386fe51dc06c1",
          "url" : "c7f66a0df8b1b52f17c28c4adb111105",
          "migrationVersion" : "4a1746014a75ade3a714e1db5763276f",
          "sample-data-telemetry" : "7d3cfeb915303c9641c59681967ffeb4",
          "index-pattern" : "45915a1ad866812242df474eb0479052",
          "search" : "43012c7ebc4cb57054e0a490e4b43023",
          "originId" : "2f4316de49999235636386fe51dc06c1",
          "application_usage_totals" : "3d1b76c39bfb2cc8296b024d73854724",
          "updated_at" : "00da57df13e94e9d98437d13ace4bfe0",
          "dql-telemetry" : "d12a98a6f19a2d273696597547e064ee",
          "search-telemetry" : "3d1b76c39bfb2cc8296b024d73854724",
          "namespace" : "2f4316de49999235636386fe51dc06c1",
          "timelion-sheet" : "9a2a2748877c7a7b582fef201ab1d4cf",
          "config" : "c63748b75f39d0c54de12d12c1ccbc20",
          "dashboard" : "40554caf09725935e2c02e02563a2d07",
          "tsvb-validation-telemetry" : "3a37ef6c8700ae6fc97d5c7da00e9215",
          "namespaces" : "2f4316de49999235636386fe51dc06c1"
        }
      },
      "properties" : {
        "application_usage_daily" : {
          "dynamic" : "false",
          "properties" : {
            "timestamp" : {
              "type" : "date"
            }
          }
        },
        "application_usage_totals" : {
          "type" : "object",
          "dynamic" : "false"
        },
        "application_usage_transactional" : {
          "type" : "object",
          "dynamic" : "false"
        },
        "config" : {
          "dynamic" : "false",
          "properties" : {
            "buildNum" : {
              "type" : "keyword"
            }
          }
        },
        "dashboard" : {
          "properties" : {
            "description" : {
              "type" : "text"
            },
            "hits" : {
              "type" : "integer",
              "index" : false,
              "doc_values" : false
            },
            "kibanaSavedObjectMeta" : {
              "properties" : {
                "searchSourceJSON" : {
                  "type" : "text",
                  "index" : false
                }
              }
            },
            "optionsJSON" : {
              "type" : "text",
              "index" : false
            },
            "panelsJSON" : {
              "type" : "text",
              "index" : false
            },
            "refreshInterval" : {
              "properties" : {
                "display" : {
                  "type" : "keyword",
                  "index" : false,
                  "doc_values" : false
                },
                "pause" : {
                  "type" : "boolean",
                  "doc_values" : false,
                  "index" : false
                },
                "section" : {
                  "type" : "integer",
                  "index" : false,
                  "doc_values" : false
                },
                "value" : {
                  "type" : "integer",
                  "index" : false,
                  "doc_values" : false
                }
              }
            },
            "timeFrom" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
            "timeRestore" : {
              "type" : "boolean",
              "doc_values" : false,
              "index" : false
            },
            "timeTo" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
            "title" : {
              "type" : "text"
            },
            "version" : {
              "type" : "integer"
            }
          }
        },
        "dql-telemetry" : {
          "properties" : {
            "optInCount" : {
              "type" : "long"
            },
            "optOutCount" : {
              "type" : "long"
            }
          }
        },
        "index-pattern" : {
          "dynamic" : "false",
          "properties" : {
            "title" : {
              "type" : "text"
            },
            "type" : {
              "type" : "keyword"
            }
          }
        },
        "migrationVersion" : {
          "dynamic" : "true",
          "properties" : {
            "config" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "index-pattern" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "namespace" : {
          "type" : "keyword"
        },
        "namespaces" : {
          "type" : "keyword"
        },
        "originId" : {
          "type" : "keyword"
        },
        "query" : {
          "properties" : {
            "description" : {
              "type" : "text"
            },
            "filters" : {
              "type" : "object",
              "enabled" : false
            },
            "query" : {
              "properties" : {
                "language" : {
                  "type" : "keyword"
                },
                "query" : {
                  "type" : "keyword",
                  "index" : false
                }
              }
            },
            "timefilter" : {
              "type" : "object",
              "enabled" : false
            },
            "title" : {
              "type" : "text"
            }
          }
        },
        "references" : {
          "type" : "nested",
          "properties" : {
            "id" : {
              "type" : "keyword"
            },
            "name" : {
              "type" : "keyword"
            },
            "type" : {
              "type" : "keyword"
            }
          }
        },
        "sample-data-telemetry" : {
          "properties" : {
            "installCount" : {
              "type" : "long"
            },
            "unInstallCount" : {
              "type" : "long"
            }
          }
        },
        "search" : {
          "properties" : {
            "columns" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
            "description" : {
              "type" : "text"
            },
            "hits" : {
              "type" : "integer",
              "index" : false,
              "doc_values" : false
            },
            "kibanaSavedObjectMeta" : {
              "properties" : {
                "searchSourceJSON" : {
                  "type" : "text",
                  "index" : false
                }
              }
            },
            "sort" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
            "title" : {
              "type" : "text"
            },
            "version" : {
              "type" : "integer"
            }
          }
        },
        "search-telemetry" : {
          "type" : "object",
          "dynamic" : "false"
        },
        "timelion-sheet" : {
          "properties" : {
            "description" : {
              "type" : "text"
            },
            "hits" : {
              "type" : "integer"
            },
            "kibanaSavedObjectMeta" : {
              "properties" : {
                "searchSourceJSON" : {
                  "type" : "text"
                }
              }
            },
            "timelion_chart_height" : {
              "type" : "integer"
            },
            "timelion_columns" : {
              "type" : "integer"
            },
            "timelion_interval" : {
              "type" : "keyword"
            },
            "timelion_other_interval" : {
              "type" : "keyword"
            },
            "timelion_rows" : {
              "type" : "integer"
            },
            "timelion_sheet" : {
              "type" : "text"
            },
            "title" : {
              "type" : "text"
            },
            "version" : {
              "type" : "integer"
            }
          }
        },
        "tsvb-validation-telemetry" : {
          "properties" : {
            "failedRequests" : {
              "type" : "long"
            }
          }
        },
        "type" : {
          "type" : "keyword"
        },
        "ui-metric" : {
          "properties" : {
            "count" : {
              "type" : "integer"
            }
          }
        },
        "updated_at" : {
          "type" : "date"
        },
        "url" : {
          "properties" : {
            "accessCount" : {
              "type" : "long"
            },
            "accessDate" : {
              "type" : "date"
            },
            "createDate" : {
              "type" : "date"
            },
            "url" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 2048
                }
              }
            }
          }
        },
        "visualization" : {
          "properties" : {
            "description" : {
              "type" : "text"
            },
            "kibanaSavedObjectMeta" : {
              "properties" : {
                "searchSourceJSON" : {
                  "type" : "text",
                  "index" : false
                }
              }
            },
            "savedSearchRefName" : {
              "type" : "keyword",
              "index" : false,
              "doc_values" : false
            },
            "title" : {
              "type" : "text"
            },
            "uiStateJSON" : {
              "type" : "text",
              "index" : false
            },
            "version" : {
              "type" : "integer"
            },
            "visState" : {
              "type" : "text",
              "index" : false
            }
          }
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "auto_expand_replicas" : "0-1",
        "provided_name" : ".kibana_92668751_admin_2",
        "creation_date" : "1659088624931",
        "number_of_replicas" : "0",
        "uuid" : "7V1pxjOmS8uVOQcB85V3Jg",
        "version" : {
          "created" : "135238227"
        }
      }
    }
  }
}

The field mapping for type is:

"type" : {
  "type" : "keyword"
}

This field mapping is valid to work with the search query of GET <PLUGIN_PLATFORM>/api/saved_objects/_find endpoint. This causes the index patterns' saved objects are found and the Wazuh plugin doesn't loop due to the redirection because could not find the idnex patterns' saved objects. This index is used for the logged user to do the request in the plugin platform and this was a migration of .kibana_92668751_admin_1, but the field mapping for the type field is different.

Conclusion

The problem happens due to search query is done in an index that has not expected field mapping for the type field doing the query doesn't match despite the health check of Wazuh plugin creating the index patterns' saved objects successfully.

Restart the Wazuh dashboard, it causes the index could be migrated and it could be created with the expected field mapping to work with the search query of the endpoint.

[Desvelao]

Fixing the loop in the Wazuh plugin

Knowing the loop is caused by exceptional circumstances with the field mapping of the index that has the saved objects and the index patterns' saved objects are created successfully in the health check when the user navigates in the Wazuh plugin, the current logic seems to be valid.

Anyways we could include some mechanism to stop the loop in the Wazuh plugin under this circumstance.

The health check of Wazuh plugin seems to be working without problems. There are checks that validate the existence of index patterns related to Wazuh. These validations pass without problems. For another hand, the getIp route resolver can't get the expected to be created index pattern due to the problem mentioned in the previous comment. It means the way to validate the existence of index patterns' saved objects is different.

Possible approaches

1. Due to both methods to check the existence of the saved objects could get different results, it seems to make sense, only use a method to check the existence of an index pattern' saved object.

2. If we are unwilling or unable to use only one method to check the existence of the saved objects. We could add, at some point of the Wazuh plugin, the validation with both methods and check that they get a similar result, if not, then display some error. This could be done in the health check of the Wazuh plugin. In this way, we secure both methods will work as expected in the circumstance of the issue. This approach seems overmeasured but it should stop the loop.

[Desvelao]

Solution

I added a verification in each check of health check related to the index patterns.

Solved in #4373.

[Desvelao]

I tested in an Elastic stack environment and I could replicate the issue. I marked the PR to be backported to the development branches of the plugin for Kibana.