Added max/min Scores to Pecentage Heuristic

[nirmalc]

Adds two params to Percentage heuristic; min_score and max_score.

minScore and maxScore to Percentile allows interesting aggregations like exclusive terms . Might be solution for #23818

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[rjernst]

@markharwood Can you take a look at this? I don't know a lot about percentage heuristic, but this seems pretty hacky to me. I would rather a solution be more composable?

[markharwood]

From a quick look at the code and the motivation behind adding it (finding distinct terms) I'm not sure it will work as intended. The mistake is in assuming that there is a single shard/index in play and the heuristic sees all data. In a distributed system the significance heuristic is invoked at shard-level and then again at the reducer node.
Even if we could bypass the min/max constraints at a shard level this sort of "distinct term" calculation will typically require a global view of the data and we can't assume it is practical to stream all stats for all terms back to a central reducer node when we have distributed indices and high cardinality fields.

I can't see a way around that limitation so I'm reluctant to accept this as a solution to the problem as it stands.

[nirmalc]

Thanks for taking a look , yes - i couldnt think of an inexpensive way to truly be distinct across cluster, so this is more or less "Approximately between x% - y%" for percent heuristics.

[markharwood]

Approximately between x% - y%" for percent heuristics.

For your use case and on time-based indices the error margin is unbounded. Consider a query for new IPs today across 7 single-sharded daily indexes. 6 of the indexes have an empty set for the foreground (they are not today's content) so will supply no background stats for any IPs because there are no hits. In contrast the index for today will have a foreground set that is exactly the same as its background set and score every IP a perfect "1" without any consideration of the other 6 days.