Running with pytest-xdist changes sys.path

[nedbat]

A simple test to show sys.path (and then fail, to always see the output):

$ cat test_sys_path.py
import pprint, sys

def test_sys_path():
    pprint.pprint(sys.path)
    1/0
$ pip freeze
apipkg==1.5
atomicwrites==1.2.1
attrs==18.2.0
execnet==1.5.0
more-itertools==4.3.0
pluggy==0.8.0
py==1.7.0
pytest==3.10.1
pytest-forked==0.2
pytest-xdist==1.24.1
six==1.11.0
$ pytest -n 0 -q
F                                                                                                                                                        [100%]
=========================================================================== FAILURES ===========================================================================
________________________________________________________________________ test_sys_path _________________________________________________________________________

    def test_sys_path():
        pprint.pprint(sys.path)
>       1/0
E       ZeroDivisionError: division by zero

test_sys_path.py:5: ZeroDivisionError
--------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
['/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/pytest_sys_path',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/bin',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python36.zip',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6/lib-dynload',
 '/usr/local/py/cpython-v3.6.7/lib/python3.6',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6/site-packages']
1 failed in 0.05 seconds
$ pytest -n 1 -q
gw0 [1]
scheduling tests via LoadScheduling
F                                                                                                                                                        [100%]
=========================================================================== FAILURES ===========================================================================
________________________________________________________________________ test_sys_path _________________________________________________________________________
[gw0] darwin -- Python 3.6.7 /usr/local/virtualenvs/tmp-9cf2e53a94c913c/bin/python3.6

    def test_sys_path():
        pprint.pprint(sys.path)
>       1/0
E       ZeroDivisionError: division by zero

test_sys_path.py:5: ZeroDivisionError
--------------------------------------------------------------------- Captured stdout call ---------------------------------------------------------------------
['/private/var/folders/j2/gr3cj3jn63s5q8g3bjvw57hm0000gp/T/pytest_sys_path',
 '',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python36.zip',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6/lib-dynload',
 '/usr/local/py/cpython-v3.6.7/lib/python3.6',
 '/usr/local/virtualenvs/tmp-9cf2e53a94c913c/lib/python3.6/site-packages']
1 failed in 0.54 seconds

The second entry in sys.path is different: without xdist, it's the bin directory of the virtualenv. With xdist, it's an empty string (current directory).

xdist shouldn't change this, it means my tests behave differently under the two scenarios.

[nedbat]

Also, xdist is setting PYTHONPATH to the launch directory and empty string. This was what really hurt my tests.

[alexmazurik]

Does anyone know why pytest-xdist prepend sys.path instead of appending in this line? I suppose changing
import sys ; sys.path.insert(0, %r)
to
import sys ; sys.path.append(%r)
could fix many import-related problems

[nicoddemus]

Thanks @alexmazurik, but that's only when using rsync, I think @nedbat's description of the problem is done right at the start of the remote here:

pytest-xdist/xdist/remote.py

Lines 257 to 260 in f283ddd

	sys.path.insert(0, importpath) # XXX only for remote situations
	os.environ["PYTHONPATH"] = (
	importpath + os.pathsep + os.environ.get("PYTHONPATH", "")
	)

This was added by d81ffaa, which states:

some changes to make remote testing work better - slightly hackish and will remain so until 'tox' is used to create environments and properly set things up.

Not sure what was the problem at the time, but perhaps this can be removed then?

[nicoddemus]

I applied this patch as a quick test:

diff --git a/xdist/remote.py b/xdist/remote.py
index b0092e0..a845d2b 100644
--- a/xdist/remote.py
+++ b/xdist/remote.py
@@ -253,11 +253,11 @@ def remote_initconfig(option_dict, args):
 if __name__ == "__channelexec__":
     channel = channel  # noqa
     workerinput, args, option_dict = channel.receive()
-    importpath = os.getcwd()
-    sys.path.insert(0, importpath)  # XXX only for remote situations
-    os.environ["PYTHONPATH"] = (
-        importpath + os.pathsep + os.environ.get("PYTHONPATH", "")
-    )
+    #importpath = os.getcwd()
+    #sys.path.insert(0, importpath)  # XXX only for remote situations
+    #os.environ["PYTHONPATH"] = (
+    #    importpath + os.pathsep + os.environ.get("PYTHONPATH", "")
+    #)
     os.environ["PYTEST_XDIST_WORKER"] = workerinput["workerid"]
     os.environ["PYTEST_XDIST_WORKER_COUNT"] = str(workerinput["workercount"])
     # os.environ['PYTHONPATH'] = importpath

and all of xdist's tests still pass.

[RonnyPfannschmidt]

I very vaugely recall this was used to ensure correctness in multi interpreter usage id like to do some archeology before dropping it

[nicoddemus]

Oh that makes sense. I think today such multi interpreter usages are rare, given that we now have tox. But it is definitely interesting to know what you can dig. 😁

[nicoddemus]

Released in 1.26.0. 👍

[webknjaz]

FTR I've found out that this is still happening: pypa/setuptools#2459 (comment).