Patch Jedi to get completions for compiled objects in Numpy and Matplotlib #3808

bcolsen · 2016-12-09T00:30:57Z

This pull request uses code from davidhalter/jedi#796 to add completions of returns that are in numpydoc doc format. This pull fixes #2162 for numpy and matplotlib.

In order for this to work you need the "numpydoc" module installed

Here is some "test" code to copy into the Spyder editor.

import numpy as np
import matplotlib.pyplot as plt

x = np.array([1,2,3])
y = np.ones()
y.                        # works
x.                        # works
w = x + y
w.                        # works
v = y.transpose()         # finds the docstring with Ctrl-i and params pop-up
v.                        # works

fig = plt.figure()
fig.                      # works
fig.add_axes()            # finds the docstring with Ctrl-i and params pop-up

ax = plt.subplot(111)
ax.                       # fails...this will have to wait for numpydoc in matplotlib
ax.set_title()            # fails...this will have to wait for numpydoc in matplotlib

ccordoba12 · 2016-12-10T13:50:10Z

spyder/docstrings.py

@@ -0,0 +1,286 @@
+"""


Let's move this file to the spyder/utils/introspection folder

ccordoba12 · 2016-12-10T13:50:38Z

spyder/docstrings.py

+    def _search_return_in_numpydocstr(docstr):
+        r"""
+        Search `docstr` (in numpydoc format) for type(-s) of `param_str`.
+


Remove this blank line

ccordoba12 · 2016-12-10T13:51:10Z

spyder/docstrings.py

+    False
+    >>> _search_param_in_docstr(':param int param: some description', 'param')
+    ['int']
+


Remove this blank line

ccordoba12 · 2016-12-10T13:51:26Z

spyder/docstrings.py

+
+    See also:
+    http://sphinx-doc.org/domains.html#cross-referencing-python-objects
+


Remove this blank line

ccordoba12 · 2016-12-10T14:04:30Z

Thanks so much for working on this! I left some minor style comments for now :-)

To fix our tests on CircleCI you need to rebase your work on top of the latest 3.x branch.

Besides, it'd be nice if you could add some tests for this new addition and also move our tests at the end of jedi_plugin.py inside spyder/utils/introspection/test/test_jedi_plugin.py. You could use as inspiration this file

https://github.com/spyder-ide/spyder/pull/3715/files#diff-4e3108d8800283797868af8e23210200

bcolsen · 2016-12-11T21:29:29Z

@ccordoba12 I can make all these changes, but I think I need to make slightly larger changes for the compiled objects returns.

I'll patch Jedi like rope is currently with a function in a separate file (like jedi_patch.py) so I can patch some Jedi methods.

I can also set a limit on the patch to Jedi == 0.9.0 because I hope all this will be included for the 0.10.0 release.

Should I make these changes in this pull request or make a new one? I think I can do it here without making a mess.

ccordoba12 · 2016-12-11T21:37:54Z

I think I can do it here without making a mess.

Sure, please make them here :-)

bcolsen · 2016-12-12T09:03:55Z

Numpy completes in Spyder with this commit but I still need to finish the testing and something else is wrong in matplotlib.

Testing is incomplete and breaking stuff. I just wanted to push to test it at work.

@ccordoba12 Where do I see the output of the debug_print() functions in jedi_plugin.py? I'm currently testing spyder with "bootstrap.py --debug"

ccordoba12 · 2016-12-12T15:47:54Z

Where do I see the output of the debug_print()

In the terminal where you started Spyder, but for that you need to start Spyder like this

SPYDER_DEBUG=3 python bootstrap.py

(assuming your using our bootstrap.py script to start Spyder from your git clone :-).

bcolsen · 2016-12-14T08:40:14Z

@ccordoba12 I've had some good progress with this. I now have mpl Axes objects returning without modifying matplotlib. I edited the first comment to show what works now.

I added a numpydoc dependancy to try to fix the CircleCl test but it still doesn't pass.

I think I'm ready for review.

goanpeca · 2016-12-14T17:24:51Z

@bcolsen awesome. Thanks for this work.

I think you need to add numpydoc here

This is a pain right now, so we are moving to use a requirements.txt file so that we only change stuff there and all CI pick up the right thing.

bcolsen · 2016-12-14T21:19:07Z

Thanks @goanpeca. I'll add numpydoc to the circle.yml

Since both numpy and matplotlib are optional, I'm going to change numpydoc to an optional dependency as well. I'll change the test_jedi_plugin to make the numpy and matplotlib tests skippable using pytest.skipif if they are not installed

TODO: Testing is incomplete

FIX: matplotlib Axes completions ADD: numpydoc dependency

FIX: added numpydocs and matplotlib to circle.yml

ccordoba12 · 2016-12-15T03:07:49Z

I have to say you've done a terrific job here, congrats!

Your tests are all passing in Python 3.5, but unfortunately failing in 2.7 (as you can see in AppVeyor and CircleCI). Please take a look at them :-)

bcolsen · 2016-12-15T04:35:46Z

OK green lights....2.7 that's still a thing :-) It was a unicode problem in the docstrings. Unfortunatly, there is still a problem in the matplotlib Axes return for python 2.7, but I'm pretty amazed it worked in 3.5 without docstrings. I guess I have to stop this PR somewhere.

I'll start looking at the upstream for jedi 0.10.0 and see if I can get some of these patches in there.

Now this is ready for review.

ccordoba12 · 2016-12-15T04:51:47Z

spyder/utils/introspection/jedi_patch.py

+            types = docstrings.find_return_types(evaluator, self)
+            if types:
+                for result in types:
+                    debug.dbg('BC!!!!!!!!!!!!!! wow types?: %s in %s', result, self)


Could you improve this debug message?

Or maybe not :-) This comes from Jedi source code, right?

Yes...oops...I needed something easy to see. I'll make it better

ccordoba12 · 2016-12-15T04:54:04Z

spyder/utils/introspection/jedi_patch.py

+    # [2] Adding type returns for compiled objects in jedi
+    # Patching jedi.evaluate.compiled.CompiledObject...
+    from jedi.evaluate.compiled import \
+        CompiledObject, builtin, _create_from_name, debug


Please parenthesis here

from jedi.evaluate.compiled import ( CompiledObject, builtin, _create_from_name, debug)

ccordoba12 · 2016-12-15T04:54:35Z

spyder/utils/introspection/jedi_patch.py

+    # [4] Fixing introspection for matplotlib Axes objects
+    # Patching jedi.evaluate.precedence...
+    from jedi.evaluate.representation import \
+        tree, InstanceName, Instance, compiled, FunctionExecution, InstanceElement


Parenthesis here too

goanpeca · 2016-12-15T04:57:06Z

spyder/utils/introspection/docstrings.py

+from jedi.parser.tree import Class
+from jedi.common import indent_block
+from jedi.evaluate.iterable import Array, FakeSequence, AlreadyEvaluated
+


Could we please arrange the imports as

# Standard library imports from ast import literal_eval from itertools import chain from textwrap import dedent import re # Third party imports from jedi import debug from jedi.common import indent_block from jedi.evaluate.cache import memoize_default from jedi.evaluate.iterable import AlreadyEvaluated, Array, FakeSequence from jedi.parser import Parser, load_grammar from jedi.parser.tree import Class

ccordoba12 · 2016-12-15T04:58:00Z

I left some minor, style comments. After that, I'd say this one is ready :-)

Thanks a lot for this great contribution!

goanpeca · 2016-12-15T04:57:27Z

spyder/utils/introspection/docstrings.py

+        return []
+
+    def _search_return_in_numpydocstr(docstr):
+        r"""


why the r here?

goanpeca · 2016-12-15T04:57:59Z

spyder/utils/introspection/docstrings.py

+
+def _expand_typestr(p_type):
+    """
+    Attempts to interpret the possible types


This should be

"""Attempts to interpret the possible types."""

goanpeca · 2016-12-15T04:58:47Z

spyder/utils/introspection/docstrings.py

+
+
+def _evaluate_for_statement_string(evaluator, string, module):
+    if string is None:


goanpeca · 2016-12-15T05:00:16Z

spyder/utils/introspection/docstrings.py

+    for element in re.findall('((?:\w+\.)*\w+)\.', string):
+        # Try to import module part in dotted name.
+        # (e.g., 'threading' in 'threading.Thread').
+        string = 'import %s\n' % element + string


We prefer the use of format and like

string = 'import {0}\n'.format(element) + string

goanpeca · 2016-12-15T05:00:22Z

spyder/utils/introspection/docstrings.py

+    # Take the default grammar here, if we load the Python 2.7 grammar here, it
+    # will be impossible to use `...` (Ellipsis) as a token. Docstring types
+    # don't need to conform with the current grammar.
+    p = Parser(load_grammar(), code % indent_block(string))


goanpeca · 2016-12-15T05:00:58Z

spyder/utils/introspection/docstrings.py

+
+def _execute_array_values(evaluator, array):
+    """
+    Tuples indicate that there's not just one return value, but the listed


goanpeca · 2016-12-15T05:01:15Z

spyder/utils/introspection/docstrings.py

+    if isinstance(array, Array):
+        values = []
+        for types in array.py__iter__():
+            objects = set(chain.from_iterable(_execute_array_values(evaluator, typ) for typ in types))


too long line?

goanpeca · 2016-12-15T05:01:22Z

spyder/utils/introspection/docstrings.py

+@memoize_default(None, evaluator_is_first_arg=True)
+def follow_param(evaluator, param):
+    """
+    Determines a set of potential types for `param` using docstring hints


period at the end

goanpeca · 2016-12-15T05:02:10Z

spyder/utils/introspection/docstrings.py

+        return set(
+            [p for string in _search_param_in_docstr(docstr, param_str)
+                for p in _evaluate_for_statement_string(evaluator, string, module)]
+        )


Isn't it better to just use two normal loops?

Using a doubly nested list comp. does not make things clear.

goanpeca · 2016-12-15T05:02:16Z

spyder/utils/introspection/docstrings.py

+@memoize_default(None, evaluator_is_first_arg=True)
+def find_return_types(evaluator, func):
+    """
+    Determines a set of potential return types for `func` using docstring hints


period at the end

ccordoba12 · 2016-12-15T05:08:30Z

@bcolsen took utils/instropection/docstrings.py from an open PR in Jedi, so I think we shouldn't demand him to fix style errors in that file.

goanpeca · 2016-12-15T05:15:16Z

@bcolsen took utils/instropection/docstrings.py from an open PR in Jedi, so I think we shouldn't demand him to fix style errors in that file

Ok!

bcolsen · 2016-12-15T05:49:22Z

I made the changes to jedi_patch.py. I really don't want to mess around with docstrings.py too much. I hope to upstream some changes to jedi after the docstrings patch gets merged there.

Thanks for your encouragement on this PR and of coarse the great work you do on Spyder!
3.1 is going to be a great release

goanpeca · 2016-12-15T05:52:13Z

Thanks for this great work @bcolsen :-)

ccordoba12 · 2016-12-15T16:27:35Z

3.1 is going to be a great release

Sure it is!

ccordoba12 added this to the v3.1 milestone Dec 9, 2016

ccordoba12 reviewed Dec 10, 2016

View reviewed changes

spyder/docstrings.py

@@ -0,0 +1,286 @@

"""

Copy link

Member

ccordoba12 Dec 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move this file to the spyder/utils/introspection folder

ccordoba12 reviewed Dec 10, 2016

View reviewed changes

spyder/docstrings.py

See also:

http://sphinx-doc.org/domains.html#cross-referencing-python-objects

Copy link

Member

ccordoba12 Dec 10, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this blank line

bcolsen force-pushed the 3.x branch from 3616ecb to 4ed8cfb Compare December 12, 2016 08:54

bcolsen added 8 commits December 14, 2016 18:11

ADD: docstrings.py to patch jedi

150f5ef

ADD: CompiledObject support

40e78ac

TODO: Testing is incomplete

FIX: PatchedCompiledObject namespace fixes

7bc3dbf

FIX: jedi_plugin tests

94f19fb

FIX: matplotlib Axes completions ADD: numpydoc dependency

ADD: Comments to jedi_patch

c8ec138

FIX: test_jedi_plugin test_get_path

d622cf9

FIX: made tests for matplotlib and numpy optional

ca9d6a9

FIX: added numpydocs and matplotlib to circle.yml

FIX: testing with python 2.7

0afc404

bcolsen force-pushed the 3.x branch from 69d20ba to 0afc404 Compare December 15, 2016 01:12

FIX: python 2.7 unicode fix

b32d7a4

ccordoba12 reviewed Dec 15, 2016

View reviewed changes

goanpeca reviewed Dec 15, 2016

View reviewed changes

FIX: style changes in jedi_patch

cae69c2

ccordoba12 changed the title ~~ADD: docstrings.py to patch jedi~~ Patch Jedi to get completions for compiled objects in Numpy and Matplotlib Dec 15, 2016

ccordoba12 merged commit 07ae16b into spyder-ide:3.x Dec 15, 2016

ccordoba12 added a commit that referenced this pull request Dec 15, 2016

Merge from 3.x: PR #3808

539cb5c


		See also:
		http://sphinx-doc.org/domains.html#cross-referencing-python-objects



		def _evaluate_for_statement_string(evaluator, string, module):
		if string is None:

Patch Jedi to get completions for compiled objects in Numpy and Matplotlib #3808

Patch Jedi to get completions for compiled objects in Numpy and Matplotlib #3808

Conversation

bcolsen commented Dec 9, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccordoba12 commented Dec 10, 2016

bcolsen commented Dec 11, 2016

ccordoba12 commented Dec 11, 2016

bcolsen commented Dec 12, 2016 • edited Loading

ccordoba12 commented Dec 12, 2016

bcolsen commented Dec 14, 2016

goanpeca commented Dec 14, 2016

bcolsen commented Dec 14, 2016 • edited Loading

ccordoba12 commented Dec 15, 2016 • edited Loading

bcolsen commented Dec 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccordoba12 commented Dec 15, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ccordoba12 commented Dec 15, 2016 • edited Loading

goanpeca commented Dec 15, 2016

bcolsen commented Dec 15, 2016

goanpeca commented Dec 15, 2016 • edited Loading

ccordoba12 commented Dec 15, 2016

bcolsen commented Dec 9, 2016 •

edited

Loading

bcolsen commented Dec 12, 2016 •

edited

Loading

bcolsen commented Dec 14, 2016 •

edited

Loading

ccordoba12 commented Dec 15, 2016 •

edited

Loading

ccordoba12 commented Dec 15, 2016 •

edited

Loading

goanpeca commented Dec 15, 2016 •

edited

Loading