Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3 - fix cffi resolver issues #6225

Merged
merged 3 commits into from
Jul 24, 2018

Conversation

Eric-Arellano
Copy link
Contributor

Problem

When running any unit test that extends TestBase with Python 3 on macOS (not tested on Linux), pants throws this exception

Exception message: Package SourcePackage(u'file:///Users/earellano/.cache/pants/python_cache/requirements/CPython-3.7.0/cryptography-2.3.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

I realized this is because I didn't have liffbfi and cffi installed, so I followed their install instructions. Our README already specifies you need these, so this is okay so far.

After installing liffbfi and cffi, I ran into a new exception:

Traceback (most recent call last):
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 157, in save_modules
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 198, in setup_context
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 248, in run_setup
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 278, in run
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 246, in runner
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/sandbox.py", line 47, in _execfile
  File "/var/folders/3q/rl02b2k936bb2x8mqfg48t180000gn/T/easy_install-ijy2a6ow/pycparser-2.18/setup.py", line 54, in <module>

  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/local/Cellar/python/3.7.0/Frameworks/Python.framework/Versions/3.7/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 209, in run
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 245, in zip_safe
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 355, in analyze_egg
  File "/Users/earellano/.cache/pants/python_cache/interpreters/CPython-3.7.0/setuptools-30.0.0-py3.7.egg/setuptools/command/bdist_egg.py", line 392, in scan_module
ValueError: bad marshal data (unknown type code)

Solution

Bumping setuptools to the max possible, 33.1.1, fixes the issue for me, when followed by ./pants clean-all

Note that setuptools goes all the way to 40.0, but PEX requires < 34.

To test locally

Add compatibility='CPython>=3.5', to the :meta entry in pants_test/util/BUILD. Then run

./pants test tests/python/pants_test/util:meta

While the unit test itself will fail, the requirements resolver should pass.

@@ -41,7 +41,7 @@ def execute(self):
safe_concurrent_rename(pants_wd, tmp_trash)
safe_concurrent_rename(tmpdir, pants_wd)

if self.get_options().async:
if self.get_options()['async']:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

async is a reserved word in Py3

new_exception = self.CacheValidationError("Problem validating target {} in {}: {}"
.format(target.id, target.address.spec_path, e))

raise self.CacheValidationError, new_exception, exc_info[2]
raise_from(self.CacheValidationError(new_exception), e)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

raise from is Python 3 syntactic sugar that provides the same functionality as exc_info[2].

@@ -27,7 +27,7 @@ pywatchman==1.4.1
requests[security]>=2.5.0,<2.19
scandir==1.2
setproctitle==1.1.10
setuptools==30.0.0
setuptools==33.1.1
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch. Fixed with latest commit. Thanks!

@jsirois jsirois merged commit d9316dd into pantsbuild:master Jul 24, 2018
@Eric-Arellano Eric-Arellano deleted the py3-fixes_cffi-issues branch July 25, 2018 03:46
CMLivingston pushed a commit to CMLivingston/pants that referenced this pull request Aug 27, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants