Adaptive Load metrics evaluator library #495
Conversation
update from master
merge from upstream
merge from upstream
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
…double Signed-off-by: eric846 <[email protected]>
…variable_setter Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
merge from upstream
…ent.Output turns out not to include the status Signed-off-by: eric846 <[email protected]>
…plugin names, log thresholds only once per session Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
| */ | ||
| virtual void | ||
| ExtractMetricSpecs(const nighthawk::adaptive_load::AdaptiveLoadSessionSpec& spec, | ||
| std::vector<const nighthawk::adaptive_load::MetricSpec*>& metric_specs, |
There was a problem hiding this comment.
Alternatively, could we not just return the map, and retrieve the keys specifically if we need that? I also don't like having two separate data structures that are supposed to be in sync with each other implicitly.
| return response; | ||
| } | ||
|
|
||
| TEST(EvaluateMetric, SetsMetricId) { |
There was a problem hiding this comment.
I think we prefer the ConditionAWillResultinB format for naming these.
There was a problem hiding this comment.
Added some more detail throughout.
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
eric846
added
waiting-for-review
Signed-off-by: eric846 <[email protected]>
| metric_spec_list_map.second; | ||
|
|
||
| std::vector<std::string> errors; | ||
| for (const MetricSpec* metric_spec : metric_specs) { |
There was a problem hiding this comment.
Reading this, I'm still wondering; it seems to me that having ExtractMetricSpecs return a vector<std::pair<MetricSpec*, ThresholdSpec*>> could simplify that interface as well as the code here; wouldn't that avoid the map lookups? The map usage would be in implementation detail in ExtractMetricSpecs, and not propagate outside of it. Of course, that might thwart re-use for other purposes perhaps.
There was a problem hiding this comment.
Oh that's a good idea, it preserves the order and is simpler. I definitely don't need the map. If someone did need a map later, I think they could pass the vector of pairs into a map constructor.
There was a problem hiding this comment.
Updated now to return a vector of pairs.
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
Signed-off-by: eric846 <[email protected]>
|
LGTM |
| namespace Nighthawk { | ||
|
|
||
| /** | ||
| * An interface with utilities for translating between metrics definitions, thresholds, scores, |
There was a problem hiding this comment.
Can we add a usage example here? I have other confusions about the API here, but if you feel like your example will resolve them, feel free to ignore other points.
There was a problem hiding this comment.
I added a better explanation of what this library does. Only one of the methods is expected to be called directly.
| * it according to a threshold and scoring function. | ||
| * | ||
| * @param metric_spec The metric spec identifying the metric by name and plugin name. | ||
| * @param metrics_plugin An already activated MetricsPlugin used by the metric_spec. |
There was a problem hiding this comment.
The word activated here is confusing to me. I'm not sure it means: Two questions:
- Would someone using this file usually already know what this means, making this less of a concern?
- Is the fact that the plugin would already need to be "activated" more or less obvious if you know what it means, making this over-specified?
There was a problem hiding this comment.
Dropped that wording, it should be clearer now.
|
|
||
| /** | ||
| * Given a MetricSpec, obtains a single metric value from the MetricPlugin and optionally scores | ||
| * it according to a threshold and scoring function. |
There was a problem hiding this comment.
I think this would be easier to follow if we said according to a ThresholdSpec. Then one would look for the threshold spec in the parameters, and see that it is a threshold with a scoring function.
| namespace Nighthawk { | ||
|
|
||
| /** | ||
| * An interface with utilities for translating between metrics definitions, thresholds, scores, |
There was a problem hiding this comment.
This comment seems to be focused on what it works with, rather than what it actually does? It's possible this will be obvious after the usage is included, but can we try to clarify its purpose if not
There was a problem hiding this comment.
This should be clearer now.
|
|
||
| /** | ||
| * Analyzes a Nighthawk Service benchmark against configured MetricThresholds. Queries | ||
| * outside MetricsPlugins for current metric values, and/or uses "nighthawk.builtin" plugin to |
There was a problem hiding this comment.
These feel like implementation details to me.
Is nighthawk.builtin a MetricsPlugin? If so, I don't think this sentence ("Queries outside MetricsPlugins") is necessary. The documentation for nighthawk.builtin should rest within that plugin itself, and this would imply we only have one internal MetricsPlugin, which I don't think is a good assumption to make for the future.
There was a problem hiding this comment.
Dropping this discussion of the builtin plugin. Mentioned the nighthawk.builtin in some of the parameter descriptions.
| * Analyzes a Nighthawk Service benchmark against configured MetricThresholds. Queries | ||
| * outside MetricsPlugins for current metric values, and/or uses "nighthawk.builtin" plugin to | ||
| * extract stats and counters from the latest Nighthawk Service output. The Nighthawk benchmark is | ||
| * assumed to have finished recently so values from MetricsPlugins will be relevant. |
There was a problem hiding this comment.
If I understand this, I think it might be better phrased as the following. (Please feel free to edit or push back):
Assumes that the values from MetricsPlugins correspond timewise with the nighthawk benchmark.
| * @param nighthawk_response Proto returned from Nighthawk Service describing the latest single | ||
| * benchmark session. | ||
| * @param spec Top-level proto defining the adaptive load session. | ||
| * @param name_to_custom_metrics_plugin_map Common map from plugin names to MetricsPlugins, loaded |
There was a problem hiding this comment.
the second clause of this is describing what the caller does, right? I don't think we should be documenting the calling code from this function. (What if it's called by something else in the future, especially since this is an interface, rather than the impl file)
If there are constraints around this, it might be better to describe it from this function's perspective:
"Should not change between calls within one adaptive load session."
But if that's a constraint here, that sounds like an architectural oddity I'd have questions about.
There was a problem hiding this comment.
I see, dropped the information about how the caller operates and and added a description of the true constraints.
dubious90
added
waiting-for-changes
Signed-off-by: eric846 <[email protected]>
eric846
added
waiting-for-review
A library that combines the latest Nighthawk Service response, metric and threshold spec configuration, and results from MetricsPlugins to produce metric scores.
Part 5 of splitting PR #483.