Pytest discovery fails if something is printed to stderr #18431
Comments
1st
added
triage-needed
|
Hi @kimadeline Please let me know if I can provide any useful details that may help to identify the issue. Thanks |
|
What I may say, in my current project it's two apps in one codebase. Depending on the env variable (that is set in the When I run the tests, I do it like this I did not find how make the test discovery that is used by VS Code to specify env variables. All what I need from VS Code discovery - just to show me a list of all tests I have in the codebase. And also to have a "run" icon near each test function. Currently I do run tests manually. I have defined in my It works to me fine. Only thing that I do often - opening the Maybe it's a way to make Test Discovery works fine with pytest for such a project? |
|
Getting a similar error, however it seems like it's because the environment location is messed up |
|
Perhaps I'm getting something similar which seems to stem from the discovery process trying to run my module itself with bad command line options. I'm really new to setting up a new project here so very unsure if this is my problem or similar to what others have reported here. This bit puzzles me, why is discovery trying to run Running |
|
I'm getting a very similar issue - pytest discovery not working in vscode while pytest from cmd line works fine. Not sure if something in conda is able to interpret If I try to copy that command (and replace the "~" with userprofile path), it fails as follows (I've added -v flags to conda run to try to get more info): I also tried removing the This is the line I'm referring to - although not sure if this is a anything to do with the problem - it should be f:\anaconda\envs\msticpy\python.exe The result is the same with or without the |
karthiknadig
added
area-testing
bug
|
Hi everyone, a couple of things here:
@1st and @brianoflondon, please provide a sample repo that I can use to try and replicate your issues. |
Thanks for taking a look @kimadeline ! This is the repo I've got which I can't get discovery working on. It was set up with Poetry and I think I can trace the problem to the point where I ran poetry install and it installed this project within itself (does that make sense?). My problem distinctly comes from the discovery process trying to run
I'm somewhat new to packaging and trying to publish on Pypi. https://github.com/brianoflondon/podping-hivewatcher |
|
I'm running into issues that might be related to the conda run problems. This is on a mac with mambaforge's anaconda; conda version 4.11. I created the python environment with the following yaml file ( Running I have no idea why it's pulling files from This is with 2022.0.1814523869 |
|
Sorry for the delay! @brianoflondon I cloned your repo, initialized a virtual environment with Does @pstjohn when you created your virtual environment, does it appear in the list of available interpreters when you run the |
|
It does, and that's actually the currently selected interpreter as well; if that's relevant: |
|
Do you have a list repro steps (and ideally, a sample repo as well) I could use to try and replicate this issue? |
|
I don't expect it's a repo issue -- I'm using zsh as my default mac shell. I was able to get this to work by
Oddly the command worked in either a zsh terminal outside of vscode, and in the zsh terminal inside vscode. It was only during pytest discovery that the |
|
ok, looks like it might have been because I specified a Apparently its broken again. Seems related to microsoft/vscode#70248, since when I open a new terminal, it doesn't set the python path correctly
Edit, it seems like even if I set up my vscode terminal such that and the command run by the testing framework works in the integrated terminal; it still fails in the pytest discovery, clearly picking up the system's python binaries |
|
Is there a way of forcing vscode to operate in the conda environment for pytest discovery? VS code insiders works though - even not started explicitly from a conda activated terminal |
|
Similar problem, but it appears that I don't have any of the errors mentioned by others. I am trying to move to VSCODE from PyCharm. So far I'm unimpressed. I have a project where my code is in a folder named 'api' and my tests are in a folder named 'tests'. Pytest runs in PyCharm with no trouble. Pytest runs on the command line with no trouble. PyTest discovery in VSCODE produces the output below. I'm running on a Mac M1 box with macOS 12.2. |
|
There’s a fundamental issue here in that the extension is always trying to do full discovery before anything is usable/runnable. In large repositories this is impractical, unhelpful, time-consuming, and unnecessary. With e.g. Eclipse and PyDev I can always select a test file or folder and “just run it” regardless of the state of the rest of the repo. This doesn’t seem that unreasonable, or that unachievable. With a large repo, I largely don’t care about the state of the rest of the repo, I just want to run the test/tests I’m currently working on. Ultimately, if the the file I’m viewing is called test.py and the function I’m looking at is called test I should be able to run it, without caring about anything else, shouldn’t I? |
|
I am still running into @calebho 's issue. I'm not clear how to resolve this.
|
|
@1vecera For now I "fixed" the issue by reverting to |
|
Hi everyone
We recently made some changes to how we discover and use conda environments, please take the newest release of the extension for a spin and report back! If it still doesn't work, please include a sample repo and repro steps that I could use to replicate the issue you're facing.
You can try specifying a specific folder in |
|
Seems to be working for me as of Edit, seems to be broken again for me. Downgrading to v2022.4.0 doesn't seem to solve the problem. Again, it seems to have to do with how $PATH is set in the integrated terminal. seems to solve it for me Edit2: well, that did solve it for a few hours, but now it's back to not working with the same error... |
|
I tried with the lates version, it's the same issue. Note: I run my VS Code in the Dev Container (Docker container). When I run this: pytest --rootdir . -s --cache-clear --no-cov --collect-onlyit shows me successful result: ================================================== 1422 tests collected in 2.99s ==================================================When VS Code runs the test discovery, it fails (taken from the Python output panel): > /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --no-cov --collect-only
cwd: .
[ERROR 2022-3-2 15:55:31.771]: Error discovering pytest tests:
[r [Error]: Graceful shutdown completed in 0.000 seconds.
at ChildProcess.<anonymous> (/root/.vscode-server/extensions/ms-python.python-2022.4.0/out/client/extension.js:32:38969)
at Object.onceWrapper (node:events:510:26)
at ChildProcess.emit (node:events:390:28)
at maybeClose (node:internal/child_process:1064:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)]Tried to run the command manually, and it printer the correct list of objects output: /usr/local/bin/python ~/.vscode-server/extensions/ms-python.python-2022.4.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear --no-cov --collect-only
[
....
{"id": "./my_app/tests/test_settings.py::test_one", "name": "test_one", "source": "./my_app/tests/test_settings.py:4", "markers": [], "parentid": "./my_app/tests/test_settings.py"}]}
]
Graceful shutdown completed in 0.000 seconds.Also I need to pass env variables to this test discovery process. Because depending on the env variable, my app is run in the "Testing" mode or a normal app mode. Didn't find how to pass the env variable to the test discovery process. |
|
Finally we found the root of the issue. Our project has such a file from api_b2c2 import configure_settings
pytest_plugins = (
"common.tests.plugin",
)
configure_settings()And also this file import atexit
import sys
def shutdown():
# Some code is omitted from this function. But interestingly - the next piece of code,
# that prevented VS Code from picking the tests tree during discovery (it writes to the STD ERR)
print(
f"Graceful shutdown completed.",
file=sys.stderr,
)
atexit.register(shutdown)When the Please take this edge case from our experience into consideration about how to improve the VS Code to handle such cases, or at least show some helpful debug information about why test discovery can't be done. It may help other projects as well. |
|
If I add
I get an assertion error when I try to do pytest discovery, since it's clearly using the system's python 2.7: interestingly I can run flake8 no problem with that assert statement in place, |
|
Still stuck on the @calebho issue: Can someone suggest a version to revert to? Version: 1.66.2 (system setup) Best Daniel |
kimadeline
changed the title
|
Update:
Thanks! |
|
Mine is working again seemingly... I'll open up a new issue if it returns Edit: ah ok, I think I've discovered the source of the error, will open a new issue. |
|
I struggle with a similar issue and I think it depends on a similar problem with printing to stderr. At the moment one of my tests is failing and terminal output looks as following: ❯ pytest
================================================== test session starts ===================================================
platform linux -- Python 3.10.4, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /run/media/makanu/Daten/GIT/boxsup_pytorch/.venv/bin/python
cachedir: .pytest_cache
rootdir: /run/media/makanu/Daten/GIT/boxsup_pytorch, configfile: pyproject.toml
plugins: cov-2.12.1
collected 4 items
tests/test_boxsup_pytorch.py::test_project_info PASSED [ 25%]
tests/test_losses.py::test_overlapping_1 PASSED [ 50%]
tests/test_losses.py::test_inter_o_union_even PASSED [ 75%]
tests/test_losses.py::test_inter_o_union_uneven FAILED [100%]
======================================================== FAILURES ========================================================
_______________________________________________ test_inter_o_union_uneven ________________________________________________
bounding_box = array([[0, 1],
[0, 0]])
full_overlap_cands = array([[[0, 1],
[0, 0]],
[[0, 1],
[0, 0]],
[[0, 1],
[0, 0]]])
def test_inter_o_union_uneven(bounding_box:np.array, full_overlap_cands:np.array):
"""test 1: uneven overlapping"""
> print(inter_o_union(bounding_box, full_overlap_cands))
tests/test_losses.py:135:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pred = array([[0, 1],
[0, 0]])
target = array([[[0, 1],
[0, 0]],
[[0, 1],
[0, 0]],
[[0, 1],
[0, 0]]])
def inter_o_union(pred:np.array, target:np.array):
"""calculates Intersection over Union
Args:
pred (np.array): The prediction(s) px array
target (np.array): The target(s) px array
Returns:
float: the divison of sum of intersection px by union px
"""
axis = None
if len(pred.shape) == 3 or len(target.shape) == 3: # More than 1 pred
axis = (1,2)
inter = np.sum(np.logical_and(pred, target), axis=axis)
union = np.sum(np.logical_or(pred, target), axis=axis)
print('inter: ', inter)
print('union: ', union)
> return union and inter / union or 0
E ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
boxsup_pytorch/losses.py:27: ValueError
-------------------------------------------------- Captured stdout call --------------------------------------------------
inter: [1 1 1]
union: [1 1 1]
---------- coverage: platform linux, python 3.10.4-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------------------
boxsup_pytorch/__init__.py 5 0 0 0 100%
boxsup_pytorch/boxsup_pytorch.py 0 0 0 0 100%
boxsup_pytorch/losses.py 12 0 2 0 100%
------------------------------------------------------------------------------
TOTAL 17 0 2 0 100%
Required test coverage of 100% reached. Total coverage: 100.00%
================================================ short test summary info =================================================
FAILED tests/test_losses.py::test_inter_o_union_uneven - ValueError: The truth value of an array with more than one ele...
============================================== 1 failed, 3 passed in 0.38s ===============================================The issue happend as I started with pytest.fixtures decorators. I deleted a test which used yielding fixture. but the discover still tracked it. So deleted all caches in project directory but now the discovery fails. The Output for python: > ./.venv/bin/python ~/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir . -s --cache-clear tests
cwd: .
[ERROR 2022-4-6 23:20:19.95]: Error discovering pytest tests:
[n [Error]: �[1m============================= test session starts ==============================�[0m
platform linux -- Python 3.10.4, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /run/media/makanu/Daten/GIT/boxsup_pytorch/.venv/bin/python
cachedir: .pytest_cache
rootdir: /run/media/makanu/Daten/GIT/boxsup_pytorch, configfile: pyproject.toml
plugins: cov-2.12.1
�[1mcollecting ... �[0mcollected 4 items
<Package tests>
tests namespace.
<Module test_boxsup_pytorch.py>
Test module for boxsup_pytorch.
<Function test_project_info>
Test __author__ value.
<Module test_losses.py>
Test Module for losses.
test scenario for overlapping_loss:
The loss formula:
$\Epsilon_o = {1\over N} \sum (1 - IoU(B,S)\delta (l_B, l_S)$
we test with following inputs:
Grid size is always 2x2
test 1:
- bounding_box: count 1, label 1
- candidates: count 3, label 1, no overlapping
expected result = 0
test 2:
- bounding_box: count 1, label 1
- candidates: count 1, label 1, no overlapping
- candidates: count 1, label 1, overlapping
- candidates: count 1, label 1, full overlapping
expected result = 1/2
test 3:
- bounding_box: count 1, label 1
- candidates: count 3, label 1, full overlapping
expected result = 1
test 4:
- bounding_box: count 1, label 1
- candidates: count 3, label 2, full overlapping
expected result = 0
<Function test_overlapping_1>
test 1: no overlapping
<Function test_inter_o_union_even>
test 1: even overlapping
<Function test_inter_o_union_uneven>
test 1: uneven overlapping
---------- coverage: platform linux, python 3.10.4-final-0 -----------
Name Stmts Miss Branch BrPart Cover Missing
------------------------------------------------------------------------------
boxsup_pytorch/__init__.py 5 0 0 0 100%
boxsup_pytorch/boxsup_pytorch.py 0 0 0 0 100%
boxsup_pytorch/losses.py 12 9 2 0 21% 6, 18-27
------------------------------------------------------------------------------
TOTAL 17 9 2 0 42%
�[31m�[1mFAIL Required test coverage of 100% not reached. Total coverage: 42.11%
�[0m�[32m========================== �[32m4 tests collected�[0m�[32m in 0.41s�[0m�[32m ==========================�[0m
Traceback (most recent call last):
File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
main(tool, cmd, subargs, toolargs)
File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
parents, result = run(toolargs, **subargs)
File "/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 1)
at ChildProcess.<anonymous> (/home/makanu/.vscode/extensions/ms-python.python-2022.6.0/out/client/extension.js:2:227541)
at Object.onceWrapper (node:events:510:26)
at ChildProcess.emit (node:events:390:28)
at maybeClose (node:internal/child_process:1064:16)
at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)]The project is completely managed by poetry no conda involved. I have a different repo with also build with poetry and there the tests run without any problem. |
|
I discovered the Problem but it is mind blowing: As soon as I import
I try to downgrade numpy until it is working again and share informations. EDIT: Update: I managed to downgrade to 1.22.0 with not running into other dependency issues. I also could install the newest release 1.21.6. For all versions I rerun pytest discovery in the test section of vscode, all with the same result. EDIT: A work around from this post solved the issue for the moment: https://stackoverflow.com/a/62197127/10985257 |
|
I digged in this problem because I was using the the problem semms to be to me that that Then In this case, I guess is conda's fault since it writes to the Btw, if vscode would activate the environment before, instead of running: The problem would be fixed |
|
I believe this is fixed on the rewrite per #18547 but please reopen if not. Thanks |
Can I get any help with pytest discovery failure?
Tried many different options with "python.testing.cwd" and "python.testing.pytestArgs" but without luck.
I used unit tests in the past and they work fine in VS Code.
Also I use VS Code with "Remote Container" and work inside the Docker container and run tests inside it.
Hope that you can solve this issue soon, because with pytest it doesn't work for me. And I have no clear vision why. Thanks.
The text was updated successfully, but these errors were encountered: