[WIP] Generate dashboard code from json

[atopuzov]

This is still WIP. Use for discussion on direction.

Tries to solve: #75
Tested with https://github.com/percona/grafana-dashboards
Had some success at "beautifying" the output with https://pythoniter.appspot.com/

[jml]

Thanks! I'm excited about this. I think the approach of parsing JSON into Grafanalib objects and then pretty-printing those is the right one.

I'm going to insist that we have Hypothesis-powered roundtripping tests for this code. That's going to be a lot of work, I'm afraid. I'm happy to chip in, but I'm unlikely to go as fast as you'd like.

[atopuzov]

@jml Can you provide a small example of hypotestis testing and I can take from there (I haven't used it before).

[jml]

from hypothesis import given
from hypothesis import strategies as st

def colors():
    """Generate arbitrary value from 0 to 255, for RGB color."""
    return st.integers(min_value=0, max_value=255)

def rgbas():
    """Generate arbitrary, valid RGBA objects."""
    return st.build(RGBA, r=colors(), g=colors(), b=colors(), a=st.floats(min_value=0, max_value=1)

@given(x=rbgas())
def test_rgbas_roundtrips(x):
    """RGBA can be encoded and then decoded."""
    json_dict = x.to_json_data()
    y = RGBA.parse_json_data(json_dict)
    assert x == y

I think if you use classmethod rather than staticmethod you can do x.parse_json_data, which would allow you to extract a re-usable roundtripping tester, but my knowledge of Python minutiae is a little stale.

Does that help?

[atopuzov]

@jml Awesome! Thanks. Will take it for here as time allows.

[atopuzov]

@jml Managed to do some hypotesis testing (which is awesome, just like haskell quickcheck).
Needs a bit of cleanup and making the things generated a bit stricter. How does it look to you?

[jml]

Oh wow, this is great, thank you.

Some thoughts:

• I'm kind of fussy about Python formatting. If you could make this match the rest of grafanalib, lovely. If not, I'll happily do the re-indenting myself.
• I'd rather we not use destructive operations like pop on things that get passed to us, reserving it for values that only live inside a single scope.
• Rather than using @staticmethod and then manually stating the class name (e.g. return Template(*args, **kwargs)), why not use @classmethod and then return cls(*args, **kwargs)?

I'll follow up with some specific in-line comments.

(Also, yes, Hypothesis is awesome. I love Haskell and I love QuickCheck, but Hypothesis is way better, IMHO.)

[jml]

So grateful that you're tackling this.

[dogopupper]

Can this be tested / instructions for doing so? @atopuzov

[atopuzov]

@dogopupper you can checkout the branch from my repo and pip install it.
eg.

git clone [email protected]:atopuzov/grafanalib.git
git checkout from_json
pip install . # I suggest you do it in a virtualenv
parse-dashboard <json-file> -o <python-file>

Obviously it is dependent on what is implemented in grafanalib atm.

[atopuzov]

@jml I think I have addressed all the concerns you had. Can you give it another look? Thanks!

[jml]

Will do.

…

On Fri, 1 Dec 2017 at 16:22 Aleksandar Topuzović ***@***.***> wrote: @jml <https://github.com/jml> I think I have addressed all the concerns you had. Can you give it another look? Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#91 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHq6hqcBGVAc2o0dwojXWHU3LDVXGV8ks5s8CfdgaJpZM4QOGpM> .

[dogopupper]

@atopuzov @jml having trouble with importlib.machinery SourceFileLoader when running parse-dashboard

https://docs.python.org/3/library/importlib.html#importlib.machinery.SourceFileLoader

  File "/usr/local/bin/parse-dashboard", line 9, in <module>
    load_entry_point('grafanalib==0.3.0', 'console_scripts', 'parse-dashboard')()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 565, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2697, in load_entry_point
    return ep.load()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2370, in load
    return self.resolve()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/pkg_resources/__init__.py", line 2376, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Library/Python/2.7/site-packages/grafanalib/_gen.py", line 9, in <module>
    from importlib.machinery import SourceFileLoader
ImportError: No module named machinery```

[dogopupper]

when using withn python 3.6:

  File "/Users/dogopupper/work/grafanalib/p35/bin/parse-dashboard", line 11, in <module>
    load_entry_point('grafanalib==0.3.0', 'console_scripts', 'parse-dashboard')()
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 159, in parse_dashboard_script
    run_script(parse_dashboard)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 120, in run_script
    sys.exit(f(sys.argv[1:]))
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 151, in parse_dashboard
    dashboard = grafanalib.core.Dashboard.parse_json_data(json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 1213, in parse_json_data
    transform=foreach(DashboardLink.parse_json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 54, in transform_dict
    new_data[new_key] = transform(old_value)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in inner
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in <listcomp>
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 642, in parse_json_data
    dicttransform('height', transform=Pixels.parse_json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 54, in transform_dict
    new_data[new_key] = transform(old_value)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in inner
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in <listcomp>
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 640, in <lambda>
    transform=foreach(lambda pnl: parse_object(pnl, PANEL_TYPES))
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 1756, in parse_object
    return mapping[object_type].parse_json_data(obj)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 1663, in parse_json_data
    transform=foreach(ValueMap.parse_json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 54, in transform_dict
    new_data[new_key] = transform(old_value)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in inner
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in <listcomp>
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 1420, in parse_json_data
    return cls(**data)
TypeError: __init__() got an unexpected keyword argument 'from'```

[atopuzov]

@dogopupper Found the bug, will push a fix. Thanks.

[dogopupper]

Thank you @atopuzov . New error (3.6):

  File "/Users/dogopupper/work/grafanalib/p35/bin/parse-dashboard", line 11, in <module>
    load_entry_point('grafanalib==0.3.0', 'console_scripts', 'parse-dashboard')()
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 159, in parse_dashboard_script
    run_script(parse_dashboard)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 120, in run_script
    sys.exit(f(sys.argv[1:]))
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/_gen.py", line 151, in parse_dashboard
    dashboard = grafanalib.core.Dashboard.parse_json_data(json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 1213, in parse_json_data
    transform=foreach(DashboardLink.parse_json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 54, in transform_dict
    new_data[new_key] = transform(old_value)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in inner
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 60, in <listcomp>
    return [func(arg) for arg in args]
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 642, in parse_json_data
    dicttransform('height', transform=Pixels.parse_json_data)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 54, in transform_dict
    new_data[new_key] = transform(old_value)
  File "/Users/dogopupper/work/grafanalib/p35/lib/python3.6/site-packages/grafanalib/core.py", line 133, in parse_json_data
    match = Pixels.REGEX.match(data)
TypeError: expected string or bytes-like object```

Would it be best if I passed my 3k lines JSON with multiple graphs to you so you can further develop the solution until it works?

[atopuzov]

@dogopupper If you can drop it somewhere would be great. Eg. a github gist, pastebin.

[dogopupper]

https://pastebin.com/QGbYPL6m 🍰

[atopuzov]

@dogopupper I tested with your input and now the only issue is when the row height is null or empty string.
Looking at 1 for Row it has to be an instance of Pixels. There is also DEFAULT_ROW_HEIGHT which is 250. height is also defined as an attribute on Graph, SingleStat, Text and Table but it can be None there.I made it so that for Row it can be None as well through att.validators.optional (i should also make it that other object do the same).
At the moment you need to remove __requires from the JSON as grafanalib does not have necessary objects to represent that. Unfortunately your dashboard is not accessible anymore and I edited the one I downloaded.

[1] https://github.com/weaveworks/grafanalib/blob/master/grafanalib/core.py#L400

[dogopupper]

@atopuzov oooooo it works!!!!!!!!!!!! however..... the python result is 20% longer (800 extra lines, around 4100)...... it's like a huge yaml file with the indentation mess (after using the pythoniter beautify)...... is the resulting grafanalib code what is actually needed to produce the dashboard or is there a lot of redundant information that can be further worked on?

[atopuzov]

@dogopupper Yes there is a lot of redundant information (eg. defaults). The object is just pretty printed to a file (eg. you would get the same result for any dashboard you define with the minimum of python code and pretty print it to file, it would also be huge).
For the moment I haven't actually thought of a way to deal with that.

[jml]

Yes there is a lot of redundant information (eg. defaults). The object is just pretty printed to a file (eg. you would get the same result for any dashboard you define with the minimum of python code and pretty print it to file, it would also be huge). For the moment I haven't actually thought of a way to deal with that.

FWIW, I'm OK with this still being the case when we merge, and am also OK with doing a release in this state, as long as the command-line tool is marked as experimental.

Once we've done the hard work of turning the JSON into Python objects (thanks again @atopuzov!), manipulating those objects is relatively easy.

[atopuzov]

@jml I need to rebase first and then it's ready for a final review.

[atopuzov]

@jml This should be good to go.

[jml]

Thanks @atopuzov. I'll take a look as soon as I can. Might not be for a couple of weeks though.

[jml]

Hi @atopuzov

I've got a local rebase of this on my laptop. Is there some way I can share this with you so you don't have to do the laborious work of rebasing yourself?

I had a brief look over the changes and I think I'd like to have a play around with the code you've written, rather than doing a traditional review.

Here's what I'll be looking into:

• can commit history be cleaned up a bit?
• can we add the Hypothesis stuff (or a simplified version of Hypothesis stuff) first in one PR and then the parser in another?

As well as general exploration to make sure I understand what's going on.

[anandsinghkunwar]

Any updates on this?

[atopuzov]

Unfortunately I don't have the time ATM. I'll pick it up when time allows.
Sorry :-(

[jakewarr8]

What's the status on this?

[atopuzov]

Sorry I can't commit any time to this.

[jml]

No worries. Thanks for letting us know, and thank you for your efforts!

[psmgeelen]

Hi there, I would still love to have this feature, can I be supportive to this?