matrix testing

[danieleades]

This library looks like exactly what i'm looking for!

I've got a slightly hacky setup trying to test a matrix of both Python versions and versions of specific packages, using Nox and Poetry.

Currently looks like this (i'm still playing with it)

import nox
import tempfile
from pathlib import Path
import os
from distutils.dir_util import copy_tree

python_versions = [
    "3.5",
    "3.6",
    "3.7",
    "3.8",
]

sphinx_versions = [
    "~2.2",
    "~2.3",
    "~2.4",
    "~3.0",
    "~3.2",
]

@nox.session(python=python_versions, reuse_venv=True)
@nox.parametrize("sphinx", sphinx_versions)
def test(session, sphinx):
    # see https://github.com/python-poetry/poetry/issues/2920#issuecomment-693147409
    with tempfile.TemporaryDirectory() as tmp_dir:
        copy_tree(
            Path.cwd(), tmp_dir
        )
        os.chdir(tmp_dir)
        
        session.run('poetry', 'add', f'sphinx@{sphinx}', external=True)
        session.run('poetry', 'install', external=True)
        session.run('nosetests', '-w', 'tests')

@nox.session(python=None)
def lint(session):
    session.run('flake8', 'sphinxcontrib', external=True)

seems to work, but i don't love it. How would you express this idiomatically using your library?

[brechtm]

I was going to open a separate issue for this, but this one is closely related, so I'll add my input here.

I recently ditched setuptools in favor of Poetry, and only now started thinking about version constraints for my dependencies. One thing I cannot find any articles on is testing of these version constraints, however.

Say I start a new project and set my Sphinx dependency version constraint to >=2.2.0. As time passes, new Sphinx releases become available. I have dependabot set up for my project, which automatically informs me of these updates and I can update the lockfile to test against these newer versions. However, as I make changes to my project, it's possible I'm unknowingly breaking compatibility with older Sphinx versions since I'm not testing against them unless I manually parameterize my Nox sessions as above. Ideally, this needs to be done for all dependencies which quickly turns into large maintenance burden.

Updating the version constraint of the dependencies to a recent version makes all this much simpler, but then you will quickly run into dependency version constraint conflicts with other packages (that require an older version), especially now that recent pip versions are actually checking this. This article seems relevant: Watchman: monitoring dependency conflicts for Python library ecosystem (I haven't read this yet).

However, the pyproject.toml contains all the information on which dependency versions my project needs to be tested against. So it would be great if nox-poetry could offer the possibility of parameterizing my sessions according to the version constraints specified in pyproject.toml. Of course, it should not test all possible combinations of these, and probably only the latest patch release of every minor version. Additionally, it would be good to enable this auto-parameterization when a particular command line parameter is passed.

I left out some details out of the analysis above for simplicity (e.g. how to specify version constraints >=2.2.0 vs ^2.2.0, ^3.0.0, skipping tests that are not relevant to an older/newer dependency version, ...). Interested to hear your thoughts!

[danieleades]

@brechtm I think you'd have to very careful applying this automatically.

For example, if you had a project with 10 dependencies, and each of those had 10 releases within the specified range, your matrix build has 10¹⁰ combinations...

Consider also, that in a very large dependency matrix, not all possible combinations are equally interesting to test, because it's rare that a user will have a very old version of one of your dependencies, and a very new version of another, since 'newness' of the various installed packages is going to be correlated. To illustrate, imagine a plot of two dependencies, with their releases over time on the x and y axis respectively. In the general case, you're mainly going to be interested in testing combinations of those dependencies close(ish) to the diagonal.

also, different projects will make different guarantees about the dependency versions they support. Some projects will aim to maintain the widest possible compatibility matrix for each dependency (common for packages intended to be installed system-wide), while others will only care about guaranteeing that there's at least one working combination of dependency versions for each python version in a given range.

Any 'automated' way of running these checks should avoid being too prescriptive.

Having said that, I could imagine some helper functions to make specifying ranges for particular dependencies a little more convenient. Say, a method that took the name of a dependency, and a specified 'granularity' and returned a vector of versions that matched the pyproject.toml specification. That then would be updated automatically after a change in your pyproject.toml file.

dependencyX_versions = get_versions("dependencyX", version.Minor)

# or something...

[danieleades]

@cjolowicz - I really like where @brechtm's head is at with extending this library so provide first-class support for matrix testing, but is there a (possibly less polished) way to do this today?

[brechtm]

For example, if you had a project with 10 dependencies, and each of those had 10 releases within the specified range, your matrix build has 1010 combinations...

Indeed. That's why I said it should not test all possible combinations of these. Even if feasibly, I imagine testing combinations would turn up next to no issues. I think a good approach would be to test the latest patch release for each minor version of a single dependency while using fixed versions (e.g. the latest compatible version) of the other dependencies. But like you say, there should be some configuration possible, since not all packages follow SemVer versioning. Also, for some dependencies you might want to limit the set of versions to test against because the automatic selection would yield too many.

I also consider this something I wouldn't run at each push event (except maybe for a "main" dependency like Sphinx in your case). Perhaps only once every week?

[danieleades]

@brechtm i guess you're wanting a function signature along the lines of

class Version(Enum):
    MAJOR
    MINOR
    PATCH

def get_versions(
    dependency: str # the name of the dependency,
    granularity: Version = Version.MINOR,
    ascending: bool = False  # count backwards from latest version, by default. not much use without the 'limit' arg
    limit: Optional[int] = None,  # maximum number of entries to return
    allow_prerelease: bool = False,  # whether to include pre-release versions
    ) -> Iterable[str]:
    ...

or something?

then you if you wanted every minor version, and the 3 most recent patches, you could do

minor_versions = get_versions("my_cool_dependency", Version.MINOR)
recent_versions = get_versions("my_cool_dependency", Version.PATCH, limit=3)

versions = set(minor_versions) | set(recent_versions)

if i'm being honest this is starting to smell like it should live in a separate little library...

[brechtm]

Yup, that would be a good interface! I might have a go at implementing that. My apologies for hijacking this ticket! 😀

Then we just require a change to nox-poetry to ignore the locked version if a specific version is specified?

[cjolowicz]

hi @danieleades ,

Thanks for raising this use case! We actually had this conversation before 😉 see wntrblm/nox#347

An issue in nox-poetry's support for parametrized sessions was fixed in 0.8.1, released today. You'll need to upgrade to get this to work.

How would you express this idiomatically using your library?

Just install your package, then install the desired Sphinx version on top:

import nox
import nox_poetry

@nox_poetry.session
@nox.parametrize("sphinx", ["2.4.4", "3.4.3"])
def test(session, sphinx):
    session.install(".")
    session.run("pip", "install", f"sphinx=={sphinx}")
    session.run(
        "python",
        "-c",
        "from importlib.metadata import version; sphinx = version('sphinx'); print(f'{sphinx = }')",
    )

(Unfortunately, you cannot simply do session.install(".", f"sphinx=={sphinx}"). It seems pip does not let you override constraints files like this.)

If you want a less verbose output from the session, you can pass silent=True to session.run, which is what session.install does by default. However, the pip output may be valuable as you are installing into an already populated virtualenv.

Would that work for your use case?

@brechtm

I have dependabot set up for my project, which automatically informs me of these updates and I can update the lockfile to test against these newer versions.

In my experience, Dependabot always updates the version constraint in pyproject.toml to require the latest version, even when the existing version constraint includes the new version. AFAIU, Dependabot has support for widening version ranges, but not for Python projects. See https://docs.github.com/en/github/administering-a-repository/configuration-options-for-dependency-updates#versioning-strategy

it would be great if nox-poetry could offer the possibility of parameterizing my sessions according to the version constraints specified in pyproject.toml.

Do you (and @danieleades) see anything additional that nox-poetry could/should do to support this use case?

[danieleades]

hi @danieleades ,

Thanks for raising this use case! We actually had this conversation before see theacodes/nox#347

aha! i've not touched this in some time, but i'm looking to pick it up again.
I'm in the process of trying to update sphinx-needs to Poetry and it's proving quite painful. I can't run their legacy test suite locally, so it's hard to know what's broken because of my nox setup, and what was already broken before i touched it...

[brechtm]

Just install your package, then install the desired Sphinx version on top:

That seems like it would work perfectly 👍

In my experience, Dependabot always updates the version constraint in pyproject.toml to require the latest version, even when the existing version constraint includes the new version. AFAIU, Dependabot has support for widening version ranges, but not for Python projects.

I also saw that in the GitHub docs, but it does seem to be widening the version range in practice.

Do you (and @danieleades) see anything additional that nox-poetry could/should do to support this use case?

No, I think with the information in this thread it should be possible to get this to work.

Of course, it would be great if nox-poetry could include the functionality I discussed above, but that's of course up to you to decide! Or perhaps you would accept a pull request?

I've had a go at implementing the get_versions() function. I'm not sure when I'll have time to transform this into a proper implementation, so I'll just leave it here for future reference:

import json
from collections.abc import Iterable
from pathlib import Path
from typing import Optional
from urllib.request import urlopen, Request

from poetry.core.factory import Factory
from poetry.core.semver import parse_single_constraint as parse_version


VERSION_PARTS = ('major', 'minor', 'patch')


def get_versions(dependency: str, granularity: str = 'minor',
                 # ascending: bool = False, limit: Optional[int] = None,
                 # allow_prerelease: bool = False,
                 ) -> Iterable[str]:
    """Yield all versions of `dependency` considering version constraints

    Args:
        dependency: the name of the dependency
        granularity: yield only the newest patch version of each major/minor
            release
        ascending: count backwards from latest version, by default (not much
            use without the 'limit' arg)
        limit: maximum number of entries to return
        allow_prerelease: whether to include pre-release versions

    Yields:
        All versions of `dependency` that match the version constraints defined
        and in this project's pyproject.toml and the given `granularity`.

    """
    package = Factory().create_poetry(Path(__file__).parent).package
    for requirement in package.requires:
        if requirement.name == dependency:
            break
    else:
        raise ValueError(f"{package.name} has no dependency '{dependency}'")
    constraint = requirement.constraint
    filtered_versions = [version for version in all_versions(dependency)
                         if constraint.allows(version)]
    parts = VERSION_PARTS[:VERSION_PARTS.index(granularity) + 1]
    result = {}
    for version in filtered_versions:
        key = tuple(getattr(version, part) for part in parts)
        result[key] = (max((result[key], version))
                       if key in result else version)
    yield from (str(version) for version in result.values())


def all_versions(dependency):
    request = Request(f'https://pypi.org/pypi/{dependency}/json')
    response = urlopen(request)
    json_string = response.read().decode('utf8')
    json_data = json.loads(json_string)
    yield from (parse_version(version) for version in json_data['releases'])

You need to save this code to a file in the same directory as your pyproject.toml or adjust the path passed to create_poetry().

With a dependency sphinx >= 2.2.1, this returns:

• get_versions('sphinx', 'major'): 2.4.4, 3.4.3
• get_versions('sphinx', 'minor'): 2.2.2, 2.3.1, 2.4.4, 3.0.4, 3.1.2, 3.2.1, 3.3.1, 3.4.3
• get_versions('sphinx', 'patch'): 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.4.2, 2.4.3, 2.4.4, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.4, 3.1.0, 3.1.1, 3.1.2, 3.2.0, 3.2.1, 3.3.0, 3.3.1, 3.4.0, 3.4.1, 3.4.2, 3.4.3

[danieleades]

hi @danieleades ,

Thanks for raising this use case! We actually had this conversation before 😉 see wntrblm/nox#347

An issue in nox-poetry's support for parametrized sessions was fixed in 0.8.1, released today. You'll need to upgrade to get this to work.

How would you express this idiomatically using your library?

Just install your package, then install the desired Sphinx version on top:

import nox
import nox_poetry

@nox_poetry.session
@nox.parametrize("sphinx", ["2.4.4", "3.4.3"])
def test(session, sphinx):
    session.install(".")
    session.run("pip", "install", f"sphinx=={sphinx}")
    session.run(
        "python",
        "-c",
        "from importlib.metadata import version; sphinx = version('sphinx'); print(f'{sphinx = }')",
    )

(Unfortunately, you cannot simply do session.install(".", f"sphinx=={sphinx}"). It seems pip does not let you override constraints files like this.)

...

Do you (and @danieleades) see anything additional that nox-poetry could/should do to support this use case?

This is working an absolute treat by the way. I'll have a think whether any specific API could smooth the road here, but this is definitely functional.

The only concern I have now is the number of test utilities I'm now installing that are outside the isolated environment. Would be nice if you could bootstrap everything with Poetry but I guess that's a little recursive.

[cjolowicz]

Would be nice if you could bootstrap everything with Poetry but I guess that's a little recursive.

It also puts an unnecessary constraint on your dependencies to be compatible with any dependencies of Nox and nox-poetry.

[danieleades]

Would be nice if you could bootstrap everything with Poetry but I guess that's a little recursive.

It also puts an unnecessary constraint on your dependencies to be compatible with any dependencies of Nox and nox-poetry.

You make a good point. Definitely the kind of thing that might get solved by PEP582, should that see wider adoption. That allows for recursive dependencies, so you can have multiple versions of the same package installed.

Feel free to close this ticket by the way. I've got a solution that works for me. Thanks for the pointers!

[cjolowicz]

hey @brechtm

I also saw that in the GitHub docs, but it does seem to be widening the version range in practice.

That's very interesting, thank you! Your Dependabot config does not seem very different from mine. I'd guess that your package is for some reason classified as a library and not an application. Although both of our packages have a console entry point. 🤔 FWIW here's a Dependabot PR in an instance of my project template: https://github.com/cjolowicz/cookiecutter-hypermodern-python-instance/pull/488/files

Of course, it would be great if nox-poetry could include the functionality I discussed above

Thanks for posting your solution. I would prefer to keep the scope of nox-poetry limited to interfacing Poetry with the core functionality of Nox. Iterating over compatible versions does not seem to fall into this area. So unfortunately, I don't see a place for this feature in nox-poetry. Please feel free to argue against this, if I'm missing something here.

[brechtm]

I also saw that in the GitHub docs, but it does seem to be widening the version range in practice.

That's very interesting, thank you! Your Dependabot config does not seem very different from mine. I'd guess that your package is for some reason classified as a library and not an application. Although both of our packages have a console entry point. 🤔 FWIW here's a Dependabot PR in an instance of my project template: https://github.com/cjolowicz/cookiecutter-hypermodern-python-instance/pull/488/files

But there is a difference. The PR you link to is changing a development dependency, while mine was changing a plain dependency. And I think that does make sense, since it is OK for dev dependencies not to widen the constraint (assuming development happens in an isolated venv).

[cjolowicz]

Right, that makes sense!

[danieleades]

Working solution, integrated with GitHub action-

https://github.com/danieleades/sphinxcontrib-needs/actions

[brechtm]

I've finally gotten around to migrating to Nox for my project. It uses the get_versions function discussed above to my run tests against a range of docutils and Sphinx versions (determined from pyproject.toml). See the jobs ending in (docutils) and (Sphinx) at https://github.com/brechtm/rinohtype/actions.