Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move scripted-metrics agg to plugin #63096

Open
polyfractal opened this issue Sep 30, 2020 · 3 comments
Open

Move scripted-metrics agg to plugin #63096

polyfractal opened this issue Sep 30, 2020 · 3 comments
Labels
:Analytics/Aggregations Aggregations >refactoring Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)

Comments

@polyfractal
Copy link
Contributor

We'd like to move the scripted-metrics aggregation to a plugin. This can be included by default so that users can continue to use it without disruption, but will prevent "stack" components from using it.

The rationale behind this is that scripted-metrics is very powerful and offers a lot of flexibility... but it is also a very sharp tool and difficult to use well. Because scripted-metrics is essentially a scripted map-reduce job, the user needs to be aware of how to build a robust distributed algorithm, not leak memory or take excessive runtime, and do it inside a scripting language. It is very easy to (accidentally) write bad scripted-metrics for these reasons. Worse yet, many of these "bad" implementations will work fine at small scale but can blow up catastrophically when encountering adversarial situations in large clusters.

For these reasons, we wish to discourage our own stack features from using scripted-metric. If there is functionality that is only doable with scripted-metric, it's a good indicator that we are missing important functionality in regular aggregations and we can instead work on adding "core" functionality. This will ensure that it is properly tested for performance and stability, and maintained by the analytics team.

By moving scripted-metric to a plugin, we can add a forcing function for "internal" uses to find (or build) alternate solutions, while still allowing users of Elasticsearch to use scripted-metric unchanged.

@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (:Analytics/Aggregations)

@Esduard
Copy link
Contributor

Esduard commented Jun 10, 2021

Hello, I am interested in this issue. Is It being worked on by someone at the moment?

@imotov
Copy link
Contributor

imotov commented Jun 10, 2021

@Esduard welcome! I think the best place to start would to familiarize yourself with contributing guidelines. Nobody is assigned to this issue at the moment, but it might not be the best issue to start with especially as a class assignment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Aggregations Aggregations >refactoring Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo)
Projects
None yet
Development

No branches or pull requests

4 participants