[system tests] Add new options to wait for documents before validating fields/mappings

[mrodm]

Currently, system tests wait for at least one document being ingested into Elasticsearch before validating the fields or mappings.

elastic-package/internal/testrunner/runners/system/tester.go

Line 1396 in 93c7a8d

return hits.size() > 0, nil

This behavior can be tuned thanks to the setting assert.hit_count. This setting changes the process to wait for an exact number of documents being ingested into Elasticsearch.

elastic-package/docs/howto/system_testing.md

Line 470 in 93c7a8d

To add an assertion on the number of hits in a given system test, consider this example from the `httpjson/generic` data stream's `test-expected-hit-count-config.yml`, shown below.

These two methods solve most of the usages, but there are some scenarios where packages need to wait for a unknown number of documents:

• This would help to validate a better representation of the fields being ingested by elastic-agent (and therefore by the service).

This was observed for instance that would be interesting for envoyproxy package from integrations repository. Some fields (that were failing while testing the validation based on mappings) just were ingested after some docs (buildkite link).

test case failed: one or more errors found in mappings in metrics-envoyproxy.stats index template: [0] field "envoy.thread_local_cluster_manager.main_thread_clusters_inflated.value" is undefined: field definition not found
[1] field "envoy.thread_local_cluster_manager.worker_0_clusters_inflated.value" is undefined: field definition not found
[2] field "envoy.thread_local_cluster_manager.worker_1_clusters_inflated.value" is undefined: field definition not found

It would be great to add new settings for the system tests to wait for documents being ingested into Elasticsearch:

• Wait for a minimum number of documents, but not required to be an exact number.
• Wait for some expected fields being present in the hits found.
• Any other option?