Uninstalling python package from virtual env crashed language server

[DonJayamanne]

@DonJayamanne commented on Tue Jan 08 2019

• Load workspace with local virtual env and LS
• Do some work
• Uninstall python package (pip uninstall)
• Here's the stack trace

Reloading modules... 
done.
[Error - 9:17:21 AM] System.AggregateException: One or more errors occurred. (Could not find a part of the path '/Users/donjayamanne/Desktop/Development/PythonStuff/extensionTesting/forTests/venv/lib/python2.7/site-packages/_pytest/config/__init__.py'.) ---> System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/donjayamanne/Desktop/Development/PythonStuff/extensionTesting/forTests/venv/lib/python2.7/site-packages/_pytest/config/__init__.py'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Microsoft.PythonTools.Interpreter.Ast.PythonModuleLoader.FromFile(IPythonInterpreter interpreter, String sourceFile, PythonLanguageVersion langVersion, String moduleFullName)
   at Microsoft.PythonTools.Interpreter.Ast.AstModuleResolution.ImportFromSearchPaths(String name, PathResolverSnapshot pathResolver)
   at Microsoft.PythonTools.Interpreter.Ast.AstModuleResolution.TryImportModuleAsync(String name, PathResolverSnapshot pathResolver, IReadOnlyList`1 typeStubPaths, Boolean mergeTypeStubPackages, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Interpreter.Ast.AstPythonInterpreter.ImportModuleAsync(String name, CancellationToken token)
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Interpreter.Ast.AstPythonInterpreter.ImportModule(String name)
   at Microsoft.PythonTools.Interpreter.Ast.AstNestedPythonModule.GetModule()
   at Microsoft.PythonTools.Interpreter.Ast.AstNestedPythonModuleMember.Get()
   at Microsoft.PythonTools.Interpreter.Ast.AstPythonModule.GetMember(IModuleContext context, String name)
   at Microsoft.PythonTools.Analysis.Values.BuiltinNamespace`1.GetTypeMember(Node node, AnalysisUnit unit, String name)
   at Microsoft.PythonTools.Analysis.Values.BuiltinModule.GetMember(Node node, AnalysisUnit unit, String name)
   at Microsoft.PythonTools.Analysis.AnalysisValueSetExtensions.GetMember(IAnalysisSet self, Node node, AnalysisUnit unit, String name)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateCall(ExpressionEvaluator ee, Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.ExpressionEvaluator.EvaluateWorker(Node node)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(ExpressionStatement node)
   at Microsoft.PythonTools.Parsing.Ast.ExpressionStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Walk(SuiteStatement node)
   at Microsoft.PythonTools.Parsing.Ast.SuiteStatement.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Parsing.Ast.PythonAst.Walk(PythonWalker walker)
   at Microsoft.PythonTools.Analysis.AnalysisUnit.AnalyzeWorker(DDG ddg, CancellationToken cancel)
   at Microsoft.PythonTools.Analysis.Analyzer.DDG.Analyze(Deque`1 queue, CancellationToken cancel, Action`1 reportQueueSize, Int32 reportQueueInterval)
   at Microsoft.PythonTools.Analysis.PythonAnalyzer.AnalyzeQueuedEntries(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.GroupAnalysis.Analyze(CancellationToken cancel)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.HandleAnalyzable(IAnalyzable item, AnalysisPriority priority, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Intellisense.AnalysisQueue.ConsumerLoop()
---> (Inner Exception #0) System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/donjayamanne/Desktop/Development/PythonStuff/extensionTesting/forTests/venv/lib/python2.7/site-packages/_pytest/config/__init__.py'.
   at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
   at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at Microsoft.PythonTools.Interpreter.Ast.PythonModuleLoader.FromFile(IPythonInterpreter interpreter, String sourceFile, PythonLanguageVersion langVersion, String moduleFullName)
   at Microsoft.PythonTools.Interpreter.Ast.AstModuleResolution.ImportFromSearchPaths(String name, PathResolverSnapshot pathResolver)
   at Microsoft.PythonTools.Interpreter.Ast.AstModuleResolution.TryImportModuleAsync(String name, PathResolverSnapshot pathResolver, IReadOnlyList`1 typeStubPaths, Boolean mergeTypeStubPackages, CancellationToken cancellationToken)
   at Microsoft.PythonTools.Interpreter.Ast.AstPythonInterpreter.ImportModuleAsync(String name, CancellationToken token)<---

[jakebailey]

The DirectoryNotFoundException and FileNotFoundException exceptions from FromFile seem to be the bulk of the analysis queue exceptions (with a few generic IOExceptions and UnauthorizedAccessExceptions thrown in). I think FromStream needs to be handling those types of exceptions, maybe returning an empty/unknown analysis if something goes wrong. @AlexanderSher thoughts?

With 0.1.75 promoted to stable, all of these exceptions will now crash the server instead of silently breaking it, so these are likely to become much more visible.

[AlexanderSher]

Ideally, we should get rid of FromStream and load file directly into memory. For now, we can catch exception in the AstModuleResolution.TryImportModuleAsync:

            // Do normal searches
            if (!string.IsNullOrEmpty(_configuration?.InterpreterPath)) {
                try {
                    module = ImportFromSearchPaths(name, pathResolver);
                } catch (OperationCanceledException) {
                    _log?.Log(TraceLevel.Error, "ImportTimeout", name, "ImportFromSearchPaths");
                    return TryImportModuleResult.Timeout;
                } catch (IOException) {
                    return TryImportModuleResult.ModuleNotFound;
                }
            }

[jakebailey]

To also catch UnauthorizedAccessException, we can probably reuse what I added here:

https://github.com/Microsoft/python-language-server/blob/23f6d67276bb7376c26abc6dbc95504ec2d6eba9/src/LanguageServer/Impl/LanguageServer.cs#L172-L181

[jakebailey]

Doing some logging (like the timeout does) may be a good idea as well.

[AlexanderSher]

Yep

[qubitron]

Seems like something we should fix before going GA

[jakebailey]

Fixed in #525.