how can I deal with unsupported python versions in my locked dependencies?

[aryarm]

Use-case

I am trying to test my project against different versions of python. Unfortunately, one of the dependencies in my lock file only supports a subset of the python versions that I'd like to test.

For example, in my particular situation, I'd like to test my project against python 3.7 and 3.11. numpy is a dependency of my project.
My lock file contains numpy v1.21.1, which is compatible with python 3.7 but not 3.11. I could upgrade my lock file to a version compatible with 3.11 (like numpy v1.23.2), but unfortunately, that version wouldn't be compatible with 3.7

Feature request

I would like to be able to do something like this:

if python_version == '3.11':
    session.install(".", f"numpy==1.23.2")
else:
    session.install(".")

But I think this will fail because pip does not allow for overriding constraints files like this.
Is there anything that I can do in this situation? Is there a way for me to override constraints from my lock file in nox-poetry?

Related to

• matrix testing #289 (comment)

[patrick91]

@aryarm as a work around I did this:

session._session.install(f"django=={django}")

[aryarm]

@patrick91, great thanks! I'll try it out

can you say a bit more about why this works?

[patrick91]

@aryarm session._session.install(f"django=={django}") uses nox install command (which is just pip install ...) while session.install from nox-poetry uses pip --constraints ... install ... which will use the lock file (if I understood correctly) so it forces you to use specific versions, which is not what I/we want in this case 😊

[aryarm]

ah, I see! thanks

but I suppose the downside of this would be that none of the other dependencies would be locked, right?

[aryarm]

ok, I think I found a better approach to the problem using Nox's parametrization of python. Will this also work with nox-poetry, though? I want all except the parametrized versions to be locked.

[aryarm]

In case anyone else runs into this issue, here is my current workaround, which works only with poetry 2.0+ and poetry-plugin-export 1.9+

1. For the versions of python you would like to support, temporarily specify the dependency with an exact-version specifier and a python version marker in your pyproject.toml file. For example, I wanted numpy==1.21.1 for py 3.7 and numpy==1.23.2 for py 3.11, so I did something like this:

   requires-python = ">=3.7.1"
   dependencies = [
       "numpy==1.21.1 ; python_version <= '3.9'",
       "numpy==1.23.2 ; python_version > '3.9'",
   ]

2. Run poetry lock --regenerate to ensure that both of versions of numpy get written to the lock file. Commit this new lock file to version history.
3. Revert the pyproject.toml file to the desired minimum-version specifier. For example, like this:

   requires-python = ">=3.7.1"
   dependencies = [
       "numpy>=1.21.1",
   ]

4. Run poetry lock and copy the content-hash that is generated at the bottom of the file.
5. Revert the lock file back to its previous state in version history (from step 2). Then, manually edit the content-hash in the lock file so that it matches the content-hash from step 4.

Now, you should be good to go! Unfortunately, running poetry lock after this point will erase the effects of these steps. So you'll have to redo these steps before every time you run poetry lock in the future. In the meantime, I will inquire about this behavior here:
https://github.com/orgs/python-poetry/discussions/10142

[aryarm]

New update

I finally found the perfect solution! This one is a lot less hacky than the last one. Once again, this will only work with poetry 2.0+ and poetry-plugin-export 1.9+

https://github.com/orgs/python-poetry/discussions/10142#discussioncomment-12050625

The answer is to use project.dependencies to list the minimum versions and tool.poetry.dependencies to list the exact versions that you would like to store in your lock file. I'm going to close this issue now because the problem is resolved and, in fact, it was never an issue with nox-poetry