test_result_json failing with tox 3.8.3

[jaraco]

Using the latest master, one test fails for me on my macOS 10.14.3 workstation:

tox-venv master $ tox                                                                                                                                                          
GLOB sdist-make: /Users/jaraco/code/public/tox-venv/setup.py
python create: /Users/jaraco/code/public/tox-venv/.tox/python
python installdeps: tox[testing]>=3.8.1
python inst: /Users/jaraco/code/public/tox-venv/.tox/.tmp/package/1/tox-venv-0.4.0.zip
python installed: apipkg==1.5,atomicwrites==1.3.0,attrs==19.1.0,coverage==4.5.3,execnet==1.5.0,filelock==3.0.10,freezegun==0.3.11,more-itertools==7.0.0,pathlib2==2.3.3,pluggy==0.9.0,psutil==5.6.1,py==1.8.0,pytest==4.3.1,pytest-cov==2.6.1,pytest-forked==1.0.2,pytest-mock==1.10.2,pytest-randomly==1.2.3,pytest-timeout==1.3.3,pytest-xdist==1.27.0,python-dateutil==2.8.0,six==1.12.0,toml==0.10.0,tox==3.8.3,tox-venv==0.4.0,virtualenv==16.4.3
python run-test-pre: PYTHONHASHSEED='2409559960'
python runtests: commands[0] | pytest tests
============================================================================= test session starts =============================================================================
platform darwin -- Python 3.7.3, pytest-4.3.1, py-1.8.0, pluggy-0.9.0
cachedir: .tox/python/.pytest_cache
Using --randomly-seed=1553876083
tox comes from: '/Users/jaraco/code/public/tox-venv/.tox/python/lib/python3.7/site-packages/tox/__init__.py'
rootdir: /Users/jaraco/code/public/tox-venv, inifile:
plugins: xdist-1.27.0, timeout-1.3.3, randomly-1.2.3, mock-1.10.2, forked-1.0.2, cov-2.6.1
collected 112 items                                                                                                                                                           

tests/test_z_cmdline.py .................F.....................................                                                                                         [ 49%]
tests/test_venv.py .........................................................                                                                                            [100%]

================================================================================== FAILURES ===================================================================================
______________________________________________________________________________ test_result_json _______________________________________________________________________________

cmd = <function cmd.<locals>.run at 0x1100801e0>, initproj = <function initproj.<locals>.initproj_ at 0x11004c0d0>
example123 = local('/private/var/folders/c6/v7hnmq453xb6p2dbz1gqc6rr0000gn/T/pytest-of-jaraco/pytest-1/test_result_json0/example123')

    def test_result_json(cmd, initproj, example123):
        cwd = initproj(
            "example123",
            filedefs={
                "tox.ini": """
                [testenv]
                deps = setuptools
                commands_pre = python -c 'print("START")'
                commands = python -c 'print("OK")'
                           - python -c 'raise SystemExit(1)'
                           python -c 'raise SystemExit(2)'
                           python -c 'print("SHOULD NOT HAPPEN")'
                commands_post = python -c 'print("END")'
            """
            },
        )
        json_path = cwd / "res.json"
        result = cmd("--result-json", json_path)
        result.assert_fail()
        data = json.loads(json_path.read_text(encoding="utf-8"))
    
        assert data["reportversion"] == "1"
        assert data["toxversion"] == tox.__version__
    
        for env_data in data["testenvs"].values():
            for command_type in ("setup", "test"):
                if command_type not in env_data:
                    assert False, "missing {}".format(command_type)
                for command in env_data[command_type]:
                    assert isinstance(command["command"], list)
>                   assert command["output"]
E                   AssertionError: assert ''

/Users/jaraco/code/public/tox-venv/tests/test_z_cmdline.py:493: AssertionError
=================================================================== 1 failed, 111 passed in 160.28 seconds ====================================================================
ERROR: InvocationError for command '/Users/jaraco/code/public/tox-venv/.tox/python/bin/pytest tests' (exited with code 1)
___________________________________________________________________________________ summary ___________________________________________________________________________________
ERROR:   python: commands failed

That was with tox 3.7.0. For good measure, I upgraded the tox in my local environment to 3.8.3 and now tests fail to run at all:

tox-venv master $ tox -r                                                                                                                                                       
GLOB sdist-make: /Users/jaraco/code/public/tox-venv/setup.py
python recreate: /Users/jaraco/code/public/tox-venv/.tox/python
___________________________________________________________________________________ summary ___________________________________________________________________________________
  python: commands succeeded
  congratulations :)
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/bin/tox", line 10, in <module>
    sys.exit(cmdline())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/session/__init__.py", line 43, in cmdline
    main(args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/session/__init__.py", line 68, in main
    exit_code = session.runcommand()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/session/__init__.py", line 192, in runcommand
    return self.subcommand_test()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/session/__init__.py", line 220, in subcommand_test
    run_sequential(self.config, self.venv_dict)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/session/commands/run/sequential.py", line 9, in run_sequential
    if venv.setupenv():
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/venv.py", line 584, in setupenv
    status = self.update(action=action)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox/venv.py", line 242, in update
    self.hook.tox_testenv_create(action=action, venv=self)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/__init__.py", line 617, in __call__
    return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/__init__.py", line 222, in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/__init__.py", line 216, in <lambda>
    firstresult=hook.spec_opts.get('firstresult'),
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/callers.py", line 201, in _multicall
    return outcome.get_result()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/callers.py", line 76, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pluggy/callers.py", line 180, in _multicall
    res = hook_impl.function(*args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tox_venv/hooks.py", line 102, in tox_testenv_create
    venv.session.make_emptydir(venv.path)
AttributeError: 'VirtualEnv' object has no attribute 'session'
tox-venv master $                                                          

[jaraco]

That latter error is with tox itself, not with this project.

[jaraco]

Or maybe not. tox-dev/tox#1221 is closed in favor of #29.

[jaraco]

Okay, confirmed after upgrading to tox-venv 0.4, I'm able to run tests again.

[jaraco]

And the originally-reported failure remains, even with the latest tox and tox-venv installed.

[gaborbernat]

tox moved on https://github.com/tox-dev/tox/blob/master/tests/unit/test_z_cmdline.py#L461-L503 (probably one of the reasons why vendoring tox tests is a bad idea) - fixing/modifying the tests now means you need to do so too

GitHub

tox-dev/tox

Command line driven CI frontend and development task automation tool - tox-dev/tox

[jaraco]

hmm. I wonder why the tests fail in master but not on Travis-CI.

[jaraco]

Even replacing that test with the version from tox, the tests continue to fail during the 'setup' command_type, because python -m venv produces no output (where the equivalent virtualenv invocation does).

(Pdb) command 
{'command': ['/Users/jaraco/code/public/tox-venv/.tox/python/bin/python', '-m', 'venv', 'python'], 'output': '', 'retcode': 0}
(Pdb) command_type
'setup'

[jaraco]

Downgrading tox to 3.8.1 in the virtual environment works around the issue, which explains why the tests aren't failing in Travis (because they ran with tox 3.8.1).

[gaborbernat]

Which goes to my point, tox-venv should not vendor toxs test suite but have it's own. A lot more tests would fail if tox-venv would use action.popen (#37 - which it absolutely should) instead of subprocess as it does now.

venv really does not behave like virtualenv 👍 their code base have little in common, their goals also just mostly align, 1 to 1 behaviour mapping cannot be expected.

[rpkilby]

@jaraco Are you sure that you pulled the latest tox-venv master? I released tox-venv v0.4.0 a few days ago, which fixes the venv.session compatibility issue (tox 3.8.0 removed the venv.session attribute).

Also, it's not ideal, but the test suite is just a slightly modified copy of the venv and command line tests from tox. The idea is that I want tox-venv to largely maintain parity with tox, and the simplest way to ensure that is to copy tox's tests. Unfortunately, this is an incredibly brittle solution and basically means that any version of tox-venv is technically pinned against a specific version of tox (in this case to 3.8.1). However in practice, tox-venv is generally compatible - not specific to just one version of tox.

[jaraco]

I just tested against e740a96 and the issue still exists. There's now an additional failure as well.

[rpkilby]

Will be fixed by #44.