Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

xdist adds current directory to sys.path #421

Closed
ndevenish opened this issue Mar 19, 2019 · 4 comments
Closed

xdist adds current directory to sys.path #421

ndevenish opened this issue Mar 19, 2019 · 4 comments

Comments

@ndevenish
Copy link

ndevenish commented Mar 19, 2019

With 1.26.1.

import sys

def test_path():
    assert "" not in sys.path
$ pytest test_path.py
<OK>
$ pytest test_path.py -n 1
E     AssertionError....

Obviously this can cause problems/change behaviour depending on where you are running the tests from.

Possibly related to #376?

@ndevenish
Copy link
Author

ndevenish commented Mar 19, 2019

In fact, the diagnosis on my end is almost exactly the same as #376 - the second entry changes from the python executable dir to the empty string. So I guess a regression rather than a new bug?

@nicoddemus
Copy link
Member

Hi @ndevenish,

Are you on Python 2.7? I ask because you might have the same issue as #414.

@ndevenish
Copy link
Author

Yes, Initially, but the error (path turning into '' therefore current directory import) happens in 3 also, either a virtualenv or a simple docker:

FROM python:3
RUN pip3 install -U pytest-xdist==1.26.1
RUN printf "import sys\ndef test_path():\n  assert not '' in sys.path\n" >> test_path.py
RUN pytest test_path.py
RUN pytest test_path.py -n 1

So, no conftest.py or site.py or anything interfering. Results in:

Step 4/5 : RUN pytest test_path.py
============================= test session starts ==============================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /, inifile:
plugins: xdist-1.26.1, forked-1.0.2
collected 1 item

test_path.py .                                                           [100%]

=========================== 1 passed in 0.01 seconds ===========================
Step 5/5 : RUN pytest test_path.py -n 1
============================= test session starts ==============================
platform linux -- Python 3.7.2, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
rootdir: /, inifile:
plugins: xdist-1.26.1, forked-1.0.2
gw0 I
gw0 [1]

F                                                                        [100%]
=================================== FAILURES ===================================
__________________________________ test_path ___________________________________
[gw0] linux -- Python 3.7.2 /usr/local/bin/python

    def test_path():
>     assert not '' in sys.path
E     AssertionError: assert not '' in ['/', '', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages']
E      +  where ['/', '', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/site-packages'] = sys.path

test_path.py:3: AssertionError
=========================== 1 failed in 0.32 seconds ===========================
The command '/bin/sh -c pytest test_path.py -n 1' returned a non-zero code: 1

It looks to be the same root cause though.

@graingert
Copy link
Member

graingert commented Jun 3, 2021

the problem is the way execnet runs python, vs setuptools console_scripts entry_points

when you run with /home/graingert/projects/pytest-bdd/.tox/py39-pytestlatest-xdist/bin/pytest the sys.path is set to:

E         +  '/home/graingert/projects/pytest-bdd/.tox/py39-pytestlatest-xdist/bin',
E         +  '/usr/lib/python39.zip',
E         +  '/usr/lib/python3.9',
E         +  '/usr/lib/python3.9/lib-dynload',
E         +  '/home/graingert/projects/pytest-bdd/.tox/py39-pytestlatest-xdist/lib/python3.9/site-packages',

however execnet runs with

/home/graingert/projects/pytest-bdd/.tox/py39-pytestlatest-xdist/bin/python

and so the path is:

['', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/home/graingert/projects/pytest-bdd/.tox/py39-pytestlatest-xdist/lib/python3.9/site-packages']

graingert added a commit to graingert/pytest-xdist that referenced this issue Jun 3, 2021
RonnyPfannschmidt added a commit that referenced this issue Jun 16, 2021
fix sys.path for local workers  Fixes #421
gcf-merge-on-green bot referenced this issue in GoogleCloudPlatform/python-docs-samples Aug 3, 2021
[![WhiteSource Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [pytest-xdist](https://github.com/pytest-dev/pytest-xdist) | `==2.2.1` -> `==2.3.0` | [![age](https://badges.renovateapi.com/packages/pypi/pytest-xdist/2.3.0/age-slim)](https://docs.renovatebot.com/merge-confidence/) | [![adoption](https://badges.renovateapi.com/packages/pypi/pytest-xdist/2.3.0/adoption-slim)](https://docs.renovatebot.com/merge-confidence/) | [![passing](https://badges.renovateapi.com/packages/pypi/pytest-xdist/2.3.0/compatibility-slim/2.2.1)](https://docs.renovatebot.com/merge-confidence/) | [![confidence](https://badges.renovateapi.com/packages/pypi/pytest-xdist/2.3.0/confidence-slim/2.2.1)](https://docs.renovatebot.com/merge-confidence/) |

---

### Release Notes

<details>
<summary>pytest-dev/pytest-xdist</summary>

### [`v2.3.0`](https://github.com/pytest-dev/pytest-xdist/blob/master/CHANGELOG.rst#pytest-xdist-230-2021-06-16)

[Compare Source](https://github.com/pytest-dev/pytest-xdist/compare/v2.2.1...v2.3.0)

\===============================

## Deprecations and Removals

-   `#&#8203;654 <https://github.com/pytest-dev/pytest-xdist/issues/654>`\_: Python 3.5 is no longer supported.

## Features

-   `#&#8203;646 <https://github.com/pytest-dev/pytest-xdist/issues/646>`\_: Add `--numprocesses=logical` flag, which automatically uses the number of logical CPUs available, instead of physical CPUs with `auto`.

    This is very useful for test suites which are not CPU-bound.

-   `#&#8203;650 <https://github.com/pytest-dev/pytest-xdist/issues/650>`\_: Added new `pytest_handlecrashitem` hook to allow handling and rescheduling crashed items.

## Bug Fixes

-   `#&#8203;421 <https://github.com/pytest-dev/pytest-xdist/issues/421>`\_: Copy the parent process sys.path into local workers, to work around execnet's python -c adding the current directory to sys.path.

-   `#&#8203;638 <https://github.com/pytest-dev/pytest-xdist/issues/638>`\_: Fix issue caused by changing the branch name of the pytest repository.

## Trivial Changes

-   `#&#8203;592 <https://github.com/pytest-dev/pytest-xdist/issues/592>`\_: Replace master with controller where ever possible.

-   `#&#8203;643 <https://github.com/pytest-dev/pytest-xdist/issues/643>`\_: Use 'main' to refer to pytest default branch in tox env names.

</details>

---

### Configuration

📅 **Schedule**: At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Renovate will not automatically rebase this PR, because other commits have been found.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box.

---

This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#github/GoogleCloudPlatform/python-docs-samples).
LucasLeRay pushed a commit to LucasLeRay/pytest-xdist that referenced this issue Sep 20, 2021
LucasLeRay pushed a commit to LucasLeRay/pytest-xdist that referenced this issue Sep 20, 2021
stanislavlevin added a commit to stanislavlevin/mypy that referenced this issue Mar 3, 2022
Description:
`run_stubtest` creates temp directory and prepend `sys.path` with
relative path (dot `.`) wrongly assuming that the dot will be
resolved to absolute path on *every* import attempt.

But in Python dot(`.`) in sys.path is actually resolved by
PathFinder and cached in `sys.path_importer_cache` like:
```
sys.path_importer_cache['.']
FileFinder('/somepath/.')
```
later calls for `find_module` return None and import of
`test_module` fails.

This resulted in only the first test in stubtest's suite passed in
non-pytest-xdist environments.

This issue was hidden with bug or feature in pytest-xdist < 2.3.0:
pytest-dev/pytest-xdist#421

It was fixed in pytest-xdist 2.3.0:
pytest-dev/pytest-xdist#667

- sys.path for pytest-xdist < 2.3.0
  `'.', 'project_path', ''`

- sys.path for pytest-xdist >= 2.3.0 or without xdist
  `'.', 'project_path'`

Fix:
In Python for denoting cwd the empty path `''` can be used as a
special case, but for readability `sys.path` is prepended with
resolved absolute path of temp directory. Also it's essential to
restore back `sys.path` after a test to not break subsequent tests.

Fixes: python#11019
Signed-off-by: Stanislav Levin <[email protected]>
hauntsaninja pushed a commit to python/mypy that referenced this issue Mar 4, 2022
Fixes #11019

`run_stubtest` creates temp directory and prepend `sys.path` with relative path (dot `.`) wrongly assuming that the dot will be resolved to absolute path on *every* import attempt.

But in Python dot(`.`) in sys.path is actually resolved by PathFinder and cached in `sys.path_importer_cache` like:
```
sys.path_importer_cache['.']
FileFinder('/somepath/.')
```
later calls for `find_module` return None and import of `test_module` fails. This resulted in only the first test in stubtest's suite passed in non-pytest-xdist environments.

This issue was hidden with bug or feature in pytest-xdist < 2.3.0:
pytest-dev/pytest-xdist#421

It was fixed in pytest-xdist 2.3.0:
pytest-dev/pytest-xdist#667

- sys.path for pytest-xdist < 2.3.0
  `'.', 'project_path', ''`

- sys.path for pytest-xdist >= 2.3.0 or without xdist
  `'.', 'project_path'`

In Python for denoting cwd the empty path `''` can be used as a special case, but for readability `sys.path` is prepended with resolved absolute path of temp directory. Also it's essential to restore back `sys.path` after a test to not break subsequent tests.

Signed-off-by: Stanislav Levin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants