question about "environment marker de-selection"

[vincent-grosbois]

Hello!

I'm trying to understand why a pex built at work cannot execute properly.

If I try to execute this pex in debug mode, I have the following interesting things:

pex: Unresolved requirements:
pex:   - fqdn; extra == "format-nongpl"
pex:   - isoduration; extra == "format-nongpl"
pex:   - jsonpointer>1.13; extra == "format-nongpl"
pex:   - uri-template; extra == "format-nongpl"
pex:   - webcolors>=1.11; extra == "format-nongpl"

and somewhere else in the log:

pex: Skipping activation of `isoduration; extra == "format-nongpl"` due to environment marker de-selection
pex: Skipping activation of `jsonpointer>1.13; extra == "format-nongpl"` due to environment marker de-selection

(etc)

So, my understanding is that these packages are indeed in the pex, but due to not having the correct "environment marker" , pex decides to not activate the module.

I managed to understand where this issue of [format-nongpl] is coming from:

• the lib jsonschema provides a set of optional dependencies ( link ) called "format-nongpl"
• in our project, we have dependencies on jsonschema through several indirection: most of the indirection don't require the optional dependencies, but we also have a dependency on jupyter-event which indeed requires this optional dependency ( link )

Further than this, I'm a bit lost as I can't really easily built the pex myself (it's built in CI system from work, and the script is quite complex to copy/paste).
But I still wanted to understand a few things :

• Is the setup I'm describing supposed to be handled by pex automatically ? ie should the pex build command be able to automatically find out and add the format-nongpl marker "by itself" in the pex file ? (or should this marker be automatically added during the build ?)
• if no, can I manually override it somehow when I'm calling the generated pex file ?

I couldn't really find many infos on what exactly are "environment marker" and how they are handled

thanks!

[vincent-grosbois]

Ah, actually it seems I can just reproduce the issue by doing:
pex jsonschema jupyter-events -o test.pex
I can't execute the resulting pex

Needed macosx_12_0_x86_64-cp-39-cp39 compatible dependencies for:
 1: fqdn; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'fqdn' distributions.
 2: idna; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'idna' distributions.
 3: isoduration; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'isoduration' distributions.
 4: jsonpointer>1.13; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'jsonpointer' distributions.
 5: uri-template; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'uri-template' distributions.
 6: webcolors>=1.11; extra == "format-nongpl"
    Required by:
      jsonschema 4.17.3
    But this pex had no 'webcolors' distributions.

If I create either of the 2 single libs, it works. It's the combination of both that fails

[vincent-grosbois]

....and if I follow what's written in #616 and execute:
pex --resolver-version pip-2020-resolver jsonschema jupyter-events
this works

But I'm still not sure what's going on :)

It seems that if I don't explicitely put --resolver-version pip-2020 we end up by default to use ResolverVersion.PIP_LEGACY , which is now (as of pip 22) depreacted. Am I correct ? if so, shouldn't the defaults be changed now ?

[jsirois]

Changing defaults is a breaking change. I won't do these without a major version change. Pex 3.x will be coming this year and will change several defaults including --resolver-version pip-2020-resolver and --pip-version 22.3.

[jsirois]

I'm going to close this as an answered question. Thanks for figuring out what options you needed.

[vincent-grosbois]

Changing defaults is a breaking change. I won't do these without a major version change. Pex 3.x will be coming this year and will change several defaults including --resolver-version pip-2020-resolver and --pip-version 22.3.

thanks, this makes sense to not want breaking changes
just to be sure, for pip-version, we default to here : https://github.com/pantsbuild/pex/blob/25947de151c639d9cdf893e417f3f2d8a0daa051/pex/pip/version.py#L70 currently ?

[jsirois]

That's correct, the default is a patched version of 20.3.4 that is vendored inside Pex.