Add support for Elasticsearch datasource #99 #100

fho · 2017-12-21T16:29:10Z

The PR adds partial support for creating graphs against an elasticsearch datasource.
It only adds support for basic query and aggregators.
Grafana supports more aggregators and query options that are not implemented yet.

fho · 2018-01-04T17:12:07Z

@jml any chance to get the PR merged?
It would be great to have the elasticsearch support in the upstream version

jml · 2018-01-04T17:15:19Z

I've scheduled tomorrow for doing grafanalib stuff.

…

On Thu, 4 Jan 2018 at 17:12 Fabian Holler ***@***.***> wrote: @jml <https://github.com/jml> any chance to get the PR merged? It would be great to have the elasticsearch support in the upstream version — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHq6owDouGTc6BQnzF0aNTEDDIpNF8cks5tHQZogaJpZM4RKDWq> .

jml

Thanks for the PR! I've got a couple of bigger changes I'd like you to make and a few smaller things. Let me know if you've got questions.

grafanalib/elasticsearch.py

+
+@attr.s
+class DateTimeAgg(object):
+    field = attr.ib(default="time_iso8601", validator=instance_of(str))


grafanalib/elasticsearch.py

+class DateTimeAgg(object):
+    field = attr.ib(default="time_iso8601", validator=instance_of(str))
+    id = attr.ib(default=0, validator=instance_of(int))
+    settings = attr.ib(default=DateTimeAggSettings())


grafanalib/elasticsearch.py

+                'id': str(self.id),
+                'settings': self.settings,
+                'type': self.type
+                }


grafanalib/elasticsearch.py

+
+@attr.s
+class CountMetric(object):
+    field = attr.ib(default="select field", validator=instance_of(str))


grafanalib/elasticsearch.py

+@attr.s
+class CountMetric(object):
+    field = attr.ib(default="select field", validator=instance_of(str))
+    type = attr.ib(default="count", validator=instance_of(str))


grafanalib/elasticsearch.py

+    """
+
+    alias = attr.ib(default=None)
+    bucket_aggs = attr.ib(default=[DateTimeAgg()])


grafanalib/elasticsearch.py

+            if agg.id:
+                return agg
+
+            agg.id = next(auto_ids)


grafanalib/elasticsearch.py

+    def to_json_data(self):
+        return {
+            'alias': self.alias,
+            'bucketAggs': self.auto_bucket_aggs_id(),


grafanalib/elasticsearch.py

+    refId = attr.ib(default="", validator=instance_of(str))
+
+    def auto_bucket_aggs_id(self):
+        ids = set([agg.id for agg in self.bucket_aggs if agg.id])


docs/example-elasticsearch.dashboard.py

@@ -0,0 +1,65 @@
+from grafanalib.core import *


fho · 2018-01-05T12:28:16Z

@jml thanks a lot for you feedback. I'll address your remarks soon.

Add an ElasticsearchTarget and query related helper classes to create graphs with queries from an elasticsearch datasource. This commit only adds partial support for elasticsearch queries. Grafana supports a lot more Metric and Aggregators for elasticsearch.

fho · 2018-02-14T18:04:13Z

@jml
I'm indecisive regarding the naming:

Elasticsearch's bucket aggregators are called "Group by" in the Grafana Webinterface
Elasticsearch's metric aggregators are called "Metric" in the Grafana Webinterface

In the current PR version metric aggregators have the postfix MetricAgg (like MaxMetricAgg), bucket aggregators have the postfix GroupBy (like DateHistogramGroupBy).
That is inconsistent but calling them GroupBy might be more clear if you are accustomed to the webinterface.

What naming would you prefer?

- add documentation - improve naming - remove Settings objects, move the fields to the aggregator objects - make fields that don't change constants in to_json_data()

fho · 2018-02-14T18:11:45Z

@jml I addressed all your comments and did some additional refactoring.
Please review it again

jml · 2018-02-15T17:10:34Z

Will do, thanks!

…

On Wed, 14 Feb 2018 at 18:11 Fabian Holler ***@***.***> wrote: @jml <https://github.com/jml> I addressed all your comments and did some additional refactoring. Please review it again — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#100 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHq6vJ0ieKyh9eIJI2v0gWFphi_IDcVks5tUyHigaJpZM4RKDWq> .

jml

Awesome. Thank you so much for sticking with this.

All my comments are very minor, so I'll fix them up myself and then merge this.

docs/example-elasticsearch.dashboard.py

@@ -0,0 +1,75 @@
+"""
+This is an exemplary Grafana board that uses an elastichsearch datasource.
+The graph shows the followinging metrics for HTTP requests to the URL path "/login":


docs/example-elasticsearch.dashboard.py

+"""
+This is an exemplary Grafana board that uses an elastichsearch datasource.
+The graph shows the followinging metrics for HTTP requests to the URL path "/login":
+- number of succesful requests resulted in a HTTP response code between 200-300


docs/example-elasticsearch.dashboard.py

+    )
+
+dashboard = Dashboard(title="HTTP dashboard",
+                      rows=[Row(panels=[g])])


grafanalib/elasticsearch.py

@@ -0,0 +1,207 @@
+"""Helpers to create Elasticsearch specific Grafana Queries."""


jml suggested changes Jan 5, 2018

View reviewed changes

fho added 2 commits February 14, 2018 18:40

docs: add an elasticsearch dashboard example

88fcf7e

elasticsearch: adapt implementation to review comments

de63d60

- add documentation - improve naming - remove Settings objects, move the fields to the aggregator objects - make fields that don't change constants in to_json_data()

jml approved these changes Feb 19, 2018

View reviewed changes

jml added 4 commits February 19, 2018 14:19

Correct spelling in elasticsearch example

14d07b9

More spelling fixes to elasticsearch example

f11b340

Consistent indentation for elasticsearch example

8d3be7a

Minor grammar tweaks to elasticsearch docstring

ac7d481

jml merged commit f295d76 into weaveworks:master Feb 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for Elasticsearch datasource #99 #100

Add support for Elasticsearch datasource #99 #100

fho commented Dec 21, 2017

fho commented Jan 4, 2018

jml commented Jan 4, 2018 via email

jml left a comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

fho commented Jan 5, 2018

fho commented Feb 14, 2018

fho commented Feb 14, 2018

jml commented Feb 15, 2018 via email

jml left a comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

		@@ -0,0 +1,207 @@
		"""Helpers to create Elasticsearch specific Grafana Queries."""

Add support for Elasticsearch datasource #99 #100

Add support for Elasticsearch datasource #99 #100

Conversation

fho commented Dec 21, 2017

fho commented Jan 4, 2018

jml commented Jan 4, 2018 via email

jml left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

fho commented Jan 5, 2018

fho commented Feb 14, 2018

fho commented Feb 14, 2018

jml commented Feb 15, 2018 via email

jml left a comment

Choose a reason for hiding this comment

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.

This comment was marked as abuse.