Fix "go to definition" and resolving imports

[busykoala]

I have "python.analysis.downloadChannel": "daily" activated.

The update two days ago broke the 'go to definition' feature for my setup.
I'm using vscode and it's regarding a large plone project using the omelette as described below.

To avoid language server load infinitely (which would be another issue, maybe related to this), I excluded the parts/omelette with search.exclude.
Below that in the settings, I added the parts/omelette with: "python.autoComplete.extraPaths": ["${workspaceFolder}/parts/omelette", ], which seemed to work for the last few days now.
By 'it seemed to work' I mean it could resolve the omelette imports and jump to definitions.
Now it sais 'no definition found for...' and the imports can't be resolved anymore.

For what I can see in the output, before the update, it used to analyse from autoComplete.extraPath, which it does not anymore since it updated the language server today.

(parts/omelette contains symlinks to cross-project sources in a local directory.)

[jakebailey]

@busykoala A new build with #467 is in progress and should show up in daily in a bit.

[busykoala]

Thank you very much!

The version I tried now, could resolve imports again with given settings.

Not yet solved in my case is 'go to definition' which once was working properly (about three days ago).
The output I get when trying to jump to a definition is following:

[Error - 12:09:05 AM] Request textDocument/definition failed.
  Message: PriorityProducerConsumer`1 instance is disposed
Object name: 'PriorityProducerConsumer`1'.
  Code: -32000 
   at Microsoft.PythonTools.Analysis.Infrastructure.DisposeToken.ThrowIfDisposed()
   at Microsoft.PythonTools.Analysis.Infrastructure.PriorityProducerConsumer`1.Produce(T value, Int32 priority)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ExecuteInQueueAsync(Func`2 function, AnalysisPriority priority)
   at Microsoft.Python.LanguageServer.Implementation.Server.WaitForCompleteAnalysisWorker(CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.FindReferences(ReferencesParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.Server.GotoDefinition(TextDocumentPositionParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.GotoDefinition(JToken token, CancellationToken cancellationToken)

Rather than this file structure some/other/package_x only is a symlink. This is pretty handy to have shared packages and is common use in plone projects.

root
│   main.py
│
├───project
│       __init__.py
│
└───some
    └───other
        └───package_x -> Users/username/.buildout/eggs/package_x

As I mentioned to avoid infinit loading on 'go to definition' it yet (and still) is necessary to search exclude **/some/other.
The folder is added again with "python.autoComplete.extraPaths":["${workspaceFolder}/some/other", ],

[jakebailey]

Is there an error message before that disposed message? That should only occur after an exception occurs in a different part of the code (#448), but the initial error should be in the same output panel.

[busykoala]

Well I think it's not really an error message except of 'no definition found for...' in the vscode gui.

Getting hover for package_name
Waiting for parsing to complete.
Parsing complete. Waiting for analysis entries to enqueue.
Enqueue complete. Waiting for analysis to complete.
[Error - 12:39:21 AM] Request textDocument/definition failed.

But it doesn't seem to scan the additional files initially. So the output makes sense.

[AlexanderSher]

@busykoala , I've tried to recreate the file structure with symlink, but for me go to definition works as expected. If you have minimal repro, can you pack it into zip or tarball and attach to the bug?

[busykoala]

I've just tried different repositories with the same structure.

In smaller project it worked having these two lines in my settings:

"python.autoComplete.extraPaths": ["${workspaceFolder}/parts/omelette", ],
"python.autoComplete.typeshedPaths": ["${workspaceFolder}/parts/omelette", ],

With bigger projects it's not working. Therefore I guess it is regarding the size of projects. I'm sorry I didn't figure that out before.

Thank you for the help and let me try python language server when it advanced a little more.

[busykoala]

I thought I'd give vscode a shot again.

My language server relevant settings are:

"python.jediEnabled": false,
"python.analysis.downloadChannel": "daily",

using microsoft language server in vscode 1.30.2 and the correct python environment.

I tested with the repository https://github.com/4teamwork/opengever.core/ which has the setup described above and reliably doesn't work (going to definition, completion suggestions).

After indexing (I guess indexing has finished because there are no new logs of indexing anymore) when trying to use go to definition the blue loading bar at the top moves around and the log shows:

Waiting for parsing to complete.
Parsing complete. Waiting for analysis entries to enqueue.
Enqueue complete. Waiting for analysis to complete.

It stays like that for minutes (I only waited around 5') until one quits the program.

I'm not totally sure that the settings are correct, but I couldn't figure out why it shouldn't work like that.

[MikhailArkhipov]

To verify perf with #546.

[busykoala]

It looks like the omelette is being indexed and also does not show any error of not found imports but I'm still not able to make use of go to definition.
It does not seems to use much ram anymore as it did previously though.

[MikhailArkhipov]

@busykoala - so what is the current problem? Is there particular code/workspace we can use for repro? LS analysis was rewritten so I'd like to verify whatever didn't work for you now does.

[busykoala]

I tested the issue again now. The repo I used is https://github.com/4teamwork/opengever.core/

Steps to get there:

git clone [email protected]:4teamwork/opengever.core.git
path/to/python/env/2.7/without/setuptools/installed bootstrap.py
bin/buildout

Now the project should be setup. I open it in code with the python relevant settings and using the same environment as used to bootstrap:

"python.analysis.logLevel": "Trace",
"python.jediEnabled": false,
"python.analysis.downloadChannel": "daily",

The version of vscode is 1.31.1 and because of "python.analysis.downloadChannel": "daily", the version of language server should be the most recent.

A use case which does not work:

If I open the file opengever/api/notifications.py and rightclick/Go to Definition on interface https://github.com/4teamwork/opengever.core/blob/447c6e9828afe300bfd3a64165c355136f780d85/opengever/api/notifications.py#L9 I get the output:

Hover in file:///Users/busykoala/Documents/Github/opengever.core/opengever/api/notifications.py at (8, 13)
Waiting for parsing to complete.
Parsing complete. Waiting for analysis entries to enqueue.
Enqueue complete. Waiting for analysis to complete.

It does not go to the definition and the blue line on top starts going from left to right (indefinitely until closing the application).

[aniforprez]

Just adding that I too am facing this same issue as @busykoala has detailed so well. The blue bar appears on top, keeps "loading" but never stops until you reload the window or quit the appllication. The "ctrl+click" method works fine until some unknown point in time when it stops turning certain symbols into links and then this happens

[contactr2m]

This happens in both cases ie. whether jedi is enabled or disabled.
When jedi is enabled:
It takes times hower tooltip to appear also sometimes it doe not even appear and justs loops.

When jedi is disabled: (MS python language server kicks in)
Keeps loading.

[jakebailey]

We released a new version of the language server to the beta download channel (0.2.16). You can set this in your VS Code settings.json to get it:

"python.analysis.downloadChannel": "beta"

I'd appreciate it if you could retest with your setup.

[bcroq]

@jakebailey I just updated to 0.2.16, "Go to definition" works.

[busykoala]

The update was a major improvement. For some imports I can go to definition now and it stopped the looping thing of the loading bar.

• While some can now be followed, many others still won't work. I'll update if I have a better idea of which work and which don't but strangely I tested some which I thought would act exactly the same.
• Once in a file from the omelette I couldn't go to definition at all (for my tested cases).
• Some of the import autocomplete works, some won't.

[contactr2m]

After upgrading to beta build.

[Error - 11:56:47 AM] Server initialization failed. Message: Object reference not set to an instance of an object. Code: -32000 1 at Microsoft.Python.Analysis.Modules.Resolution.MainModuleResolution.CreateModule(String name) at Microsoft.Python.Analysis.Modules.Resolution.ModuleResolutionBase.ModuleRef.GetOrCreate(String name, ModuleResolutionBase mrb) at Microsoft.Python.Analysis.Modules.Resolution.ModuleResolutionBase.GetOrLoadModule(String name) at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.LoadMissingDocuments(IPythonInterpreter interpreter, ImmutableArray1 missingKeys) at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.AnalyzeDocumentAsync(ModuleKey key, PythonAnalyzerEntry entry, CancellationToken cancellationToken) at Microsoft.Python.Core.TaskExtensions.DoNotWait(Task task) at Microsoft.Python.Analysis.Analyzer.PythonAnalyzer.EnqueueDocumentForAnalysis(IPythonModule module, PythonAst ast, Int32 bufferVersion, CancellationToken cancellationToken) at Microsoft.Python.Analysis.Modules.PythonModule.Parse(CancellationToken cancellationToken) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location where exception was thrown ---
at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot)
--- End of stack trace from previous location where exception was thrown ---
at Microsoft.Python.Analysis.Modules.PythonModule.GetAstAsync(CancellationToken cancellationToken) at Microsoft.Python.Analysis.Modules.PythonModule.LoadAndAnalyzeAsync(CancellationToken cancellationToken) at Microsoft.Python.Analysis.Modules.Resolution.MainModuleResolution.LoadBuiltinTypesAsync(CancellationToken cancellationToken) at Microsoft.Python.Analysis.Analyzer.PythonInterpreter.LoadBuiltinTypesAsync(String root, IServiceManager sm, CancellationToken cancellationToken) at Microsoft.Python.Analysis.Analyzer.PythonInterpreter.CreateAsync(InterpreterConfiguration configuration, String root, IServiceManager sm, CancellationToken cancellationToken) at Microsoft.Python.LanguageServer.Implementation.Server.InitializeAsync(InitializeParams params, CancellationToken cancellationToken) at Microsoft.Python.LanguageServer.Implementation.LanguageServer.Initialize(JToken token, CancellationToken cancellationToken)

PS : I set PYTHONPATH before opening vs code from command line.

Sorry for bad formatting

[jakebailey]

That is #719. It'd be better for you to file issues if they differ from the original issue in this thread.

[busykoala]

With these settings it is working perfectly fine for some time (around a minute) and then something breaks (I think this is the initial error):

Goto Definition in file:///Users/busykoala/Documents/PythonProjects/opengever.core/opengever/core/hooks.py at (1, 39)
[Error - 3:06:16 PM] Unable to find document file:///Users/busykoala/Documents/PythonProjects/opengever.core/opengever/core/hooks.py
[Error - 3:06:16 PM] Request textDocument/definition failed.
  Message: Object reference not set to an instance of an object.
  Code: -32000 
   at Microsoft.Python.LanguageServer.Sources.DefinitionSource.FindDefinition(IDocumentAnalysis analysis, SourceLocation location)
   at Microsoft.Python.LanguageServer.Implementation.Server.GotoDefinition(TextDocumentPositionParams params, CancellationToken cancellationToken)
   at Microsoft.Python.LanguageServer.Implementation.LanguageServer.GotoDefinition(JToken token, CancellationToken cancellationToken)

settings

workspace settings (in ${workspace}/.vscode/settings.json):

{
    // python virtual environment
    "python.pythonPath": "/absolute/path/to/venv/bin/python",
    // add the omelette folder
    "python.autoComplete.extraPaths": [
        "${workspaceFolder}/parts/omelette",
    ],
    // for my project python language server would crash and stop restarting
    // without raising the default memory max
    "files.maxMemoryForLargeFilesMB": 8192,
}

user settings (cmd+p then Preferences: Open Settings (JSON)):

{
    // exclude omelette (in parts folder)
    "search.exclude": {
        "**/parts": true,
    },
    // to use ms python language server
    "python.jediEnabled": false,
    // for some nice output
    "python.analysis.logLevel": "Trace",
    // to get the most current stuff
    "python.analysis.downloadChannel": "daily",
    "python.analysis.downloadChannel": "beta",
}

[MikhailArkhipov]

@busykoala - the exception should be fixed in 0.2.36 that is coming out ~ March 29th
Once in a file from the omelette I couldn't go to definition at all (for my tested cases). - #840.

Since the item is evolved with many cases fixed, I am closing it. Please open new issues for the cases you still discover. Thanks.

[erihanse]

Has this stopped working again? It works with python.jediEnabled: true

[jakebailey]

What is the message being squiggled? If the import isn't resolved, then it won't be able to find the definition.

[erihanse]

I believe it's a linter complaining about lack of module documentation. The import should be valid.

[jakebailey]

A linter meaning an external linter, or a message generated by the language server? (It'll say (Python) in the hover if it's from the LS, otherwise the name of the linter.)

Also, what's in your .env? Are you setting PYTHONPATH? Similarly, are you setting extraPaths in your VS Code settings?

[erihanse]

It's an external, pydocstyle.

.env:

PYTHONPATH="."

I'm not setting extraPaths in vs code settings.

[erihanse]

When I hover it says it sees it as a module.

[jakebailey]

You're at the latest version, correct? I recently fixed some things related to setting a path to . (which should be redundant, as the opened folder itself is a search path). Are variables from that imported module accessible, or are they not listed?

Since I think it's getting resolved, it's probably not related to the PYTHONPATH=. fixes. I'd appreciate it if you could open a new issue with that project attached (zipped, if you're willing to upload it).

[erihanse]

I'm running 2019.3.6558 (8 April 2019) of vs code python extension. Not sure if there are any steps I need to do to upgrade LS? I'll create a new issue.

[jakebailey]

The LS version is different from the extension; you can find the language server version printed in the Python output panel (or in the directory name for the download inside the Python extension's folder).