Review Python policy for Hydro

[tfoote]

As raised by @jack-oquin in ros-infrastructure/rep#26 the python decision could use a review by ros-users. I'll circulate this to get feedback. I'm opening this ticket here so we can close #411 and have a default option for now.

[jack-oquin]

The updated REP 3 says:

As of Raring Ubuntu will ship with Python 3 by default. While we still support 
Ubuntu versions which ship with Python 2 by default all Python scripts should 
provide dual compatibility with Python 2.7+ and Python 3.3+.

Exactly what does this mean? A literal reading implies that all ROS packages containing any Python code must run unmodified with both a Python 2.7 interpreter and a Python 3.3 interpreter.

I don't believe that literal interpretation is reasonable, or even possible.

While an enthusiastic Python user for several years now, I am no language expert. So, I've done some reading on the net. Here are some pages by knowledgeable people discussing the Python migration problem:

• http://programmers.stackexchange.com/questions/63859/why-do-people-hesitate-using-python-3
• http://wiki.python.org/moin/Python2orPython3
• http://www.python.org/dev/peps/pep-3000/#compatibility-and-transition

In summary, various migration methods are discussed, but no Python experts recommend releasing any non-trivial "bilingual" code that attempts to support both Python versions in a single source tree. Deep semantic incompatibilities such as Unicode string handling seem to dictate against that approach.

The usual recommendation for maintaining a body of code that must work with both versions is to maintain the source in a Python 2.7 branch; use the automated 2to3 script to translate to Python 3 syntax with no manual editing; make all fixes in the original branch; then release separate tarballs for the two versions.

That poses obvious problems for ROS Hydro code that must support both Precise with a Python 2.7 default interpreter, and Raring with Python 3.3 by default in a single source release.

I can think of various ways to address these problems, but first I want to raise the issues and get input from others. Some people have been working on Python 3 ports of various ROS scripts for a while now. I hope they will describe the problems encountered, and suggest how best to proceed.

[tkruse]

This was discussed in yesterdays platform group meeting, here are ideas from the meeting:

One point made was that of the core ROS python stuff, considerable effort has already been made to provide dual compatibility by the members of the MORSE project. I know I have done this also for vcstools and rosinstall. So in the meeting some confidence existed that for Hydro this is doable for a "core" set of packages.

All other "non-core" packages will either be migrated to python3 in due time, and until then only be usable on systems where python2 is the default python interpreter. So it is up to users to not use ringtail or downgrade the ringtail default interpreter, if any package they want to use is not py3k compatible. How users are supposed to know that beforehand was not discussed. Also we mentioned that a problem is that debian packages may get released for ringtail even if they have no py3k compatible python code. To prevent that, the idea is to change the build scripts such that when automated testing fails, the build fails (which s not the case currently, apparently).

Other than that, I also recommend this link as a resource:
https://wiki.ubuntu.com/Python/3
It has useful resources at the bottom, and it recommends dual compatibility and avoidance of 2to3 and six.

[jack-oquin]

Converting all the Python code in the packages I maintain seems like a big job to get done, tested and released in only two months. But, it is the user confusion Thibault mentions that I worry most about.

From an external perspective, it will appear that large segments of "the ROS system" just don't work. Users don't distinguish between core ROS components and various packages they want to use.

Despite that, we may need to attack the Python migration problem differently for different classes of scripts:

1. Infrastructure packages that support multiple ROS distributions: bloom, jenkins, rosdoc, rosdep, rosinstall, etc.
2. Core ROS scripts needed for basic system features: catkin, diagnostics, dynamic reconfigure, genmsg, rospack, rospkg, rostopic, rosnode, rostest, rosunit, tf2, etc.
3. Higher-level ROS packages needed by many robots: drivers, navigation, rqt, smach, etc.
4. Actively-maintained outside repositories: care-o-bot, ROS industrial, Bosch, rocon, ros_serial, turtlebot, OROCOS, etc.
5. Various un-maintained packages that people often try to use.

[jack-oquin]

Thibault's link to the Ubuntu migration page is helpful. Apparently, "bilingual" scripts have become more feasible since the Python 2.7 and 3.3 releases. Many of the links I posted were from the Python 2.6 and 3.0 time frame, when that was more difficult.

Fortunately Hydro specifies both 2.7 and 3.3, so many scripts will be able to take advantage of some newer co-existence features. I expect most of us would prefer a bilingual solution, when it is workable, although some developers needing to support Python 2.6 or older may not have that option.

All recommendations emphasize the importance of good unit test coverage before starting the port to Python 3. We need to provide a clean way for developers to run ROS unit tests with the python2.7 -3 command.

[jack-oquin]

I decided to experiment with a simple unique identifier module. Some observations:

• It is easy to run python -3

  $ roscd unique_id
  $ python -3 tests/test_unique_id.py

• The output will be a mess until the core ROS scripts have been migrated. This message is repeated about 18 times complaining about various statements in rospkg/manifest.py, roslib/manifestlib.py, rospkg/distro.py, rosdep2/sources_list.py, genpy/rostime.py, genmsg/msgs.py, and genpy/message.py:

/usr/lib/pymodules/python2.7/rospkg/manifest.py:196: DeprecationWarning: Overriding __eq__ blocks inheritance of __hash__ in 3.x
  class Platform(object):

• We won't be able to run the unit tests with python3 until after migrated core scripts are available.

$ python3 tests/test_unique_id.py 
Traceback (most recent call last):
  File "tests/test_unique_id.py", line 4, in <module>
    import roslib; roslib.load_manifest(PKG)
  File "/opt/ros/groovy/lib/python2.7/dist-packages/roslib/__init__.py", line 50, in <module>
    from roslib.launcher import load_manifest
  File "/opt/ros/groovy/lib/python2.7/dist-packages/roslib/launcher.py", line 42, in <module>
    import rospkg
ImportError: No module named rospkg

• This suggests that the migration effort will need to be done bottom-up, with higher-level packages waiting until the core scripts are working. Doing all that on a 2-month schedule seems rather unlikely.

[kargm]

Severin Lemaignan, me and other members of the MORSE community together with Ken Conley ported ROS messaging (which involved mostly code in roslib and rospy) to work on Python2 and 3 (we started with 3.1, but meanwhile are using 3.3). We are now using this code since Diamondback and it works fine apart from some minor patches we had to do from time to time (like fixing rospack and catkin for Py2 and 3).

The Ubuntu Ubuntu migration page that Thibault posted seems to cover most of the issues, as mentioned there, I can confirm that string/bytes handling can be a pain.. So one should really listen to the last point "Clarify your data model" in the "Before you start" section of the Ubuntu migration page.

From my experience, this migration is doable but the effort should not be underestimated! 2 month seems rather like a tough schedule...

[severin-lemaignan]

Based on MORSE requirements for ROS support, the last ROS "core" component to require large porting effort to py3 is actionlib (and maybe tf? Michael?).

[kargm]

Yes, tf and actionlib will have to be patched as well (although I guess this should be a minor effort).

We also didn't patch the tools in the ros_comm package (rosgraph, rosconsole, etc.). I didn't look into it in detail, but I guess most of them will have to be patched which could require some more effort...

[jack-oquin]

Based on the MORSE work, I propose the following Hydro release plan:

1. Try to make all the core and infrastructure scripts bilingual (Python 2.7 and 3.3), including:
   • Infrastructure packages that support multiple ROS distributions: bloom, jenkins, rosdoc, rosdep, rosinstall, etc.
   • Core ROS scripts needed for basic system features: actionlib, catkin, diagnostics, dynamic reconfigure, genmsg, rospack, rospkg, rostopic, rosnode, rostest, rosunit, tf2, etc.
2. Because they depend on that work being completed first, it seems unrealistic to expect many higher-level scripts to be ported to Python 3 for Hydro.
3. Set user expectations clearly and appropriately:
   • If people want to run a non-trivial ROS system on Raring they will almost certainly need to make Python 2.7 the default interpreter.
   • Provide a user-visible package attribute that developers can use to mark their packages Python 3 compatible.
   • We can assume that existing packages containing Python probably only work with Python 2.
   • Provide some user tools to: (1) mark packages containing no Python code as compatible; (2) check whether all the packages invoked in a set of launch scripts are compatible.

[jbohren]

Just an update, the SMACH hydro-devel branch is ready for Python3 testing (and maintains 2.7 compatibility): ros/executive_smach@fdf99b1

[jack-oquin]

Until bilingual versions of roslib and rospkg are released, only python -3 testing can occur.

Did you try that with SMACH, Jonathan?

[jbohren]

@jack-oquin When I run the tests with python -3 I get a bunch of warnings about rospkg and roslib but none about the smach sources. The issues with rospkg and roslib are why I said SMACH was ready for Python3 testing and not that it has been tested on Python3, though I could probably write some tests for the SMACH core since it's ROS-independent.

[jack-oquin]

That's my status, too.

If the basic roslib and rospkg updates could be released, there is much more that could be tested. That should top the priority queue for Python 3 migration.

[tfoote]

Conclusion:

Core tools will be bilingual (2.7, 3.3 compatible)

If something is known not to work for either python2 or python3 it should be annotated in the description.

This can be closed when we document this somewhere.

[tfoote]

If we want tooling support we should revisit that.

[jack-oquin]

I volunteer to make an update for REP-0003, explaining that only the core tools will work with Python 3.

For users to actually run ROS on Raring, we will need to recommend configuring the default Python version to 2.7.

Once people start migrating actual robotics packages (I-turtle??), users will definitely need tool help to figure out what they can run in a Python 3 system.

[jack-oquin]

REP-0003 update submitted with Python 3 support explanation.

Please review and provide comments.

[jack-oquin]

@kargm and @severin-lemaignan:

• Where are the patched versions of the ROS core that MORSE uses?

[kargm]

They are checked in into the ROS trunk. Basically standard ros core and rospy allow messaging with python3 since Electric. However, we didn't test for code coverage with py3, so there might still be some issues, but we are using messaging as well as services with Python3 since Diamondback and it seems to work fine... (for Diamondback and the first Electric release, you still had to overlay ros core and rospy, but in one maintenance update of Electric, all of our patches made it into the trunk)

[severin-lemaignan]

As far as I know, everything has been already merged upstream by Ken a while ago

See for instance: ros/genpy#3

Currently we only have on "MORSE specific" version of a ROS core component: tfMessage.py

[jack-oquin]

On Hydro and Groovy rosunit seems to be broken. At least, it uses the old-style print command.

So, should I presume that is a new problem and open an issue for it?

[jack-oquin]

The hydro-devel branch of rosunit does not work with python 3.2 on Precise.

• rosunit: uses python2 print statement, which fails with python3 ros#11
• rosunit: test_junitxml.py defines unicode strings, which are not supported by python3 in Precise  ros#12

Since Raring is not yet supported on Hydro, converted code needs to be tested with 3.2, which is less compatible than 3.3, which supports u"string" syntax.

[jack-oquin]

The deeper I delve into this problem, the harder it looks.

• To port scripts from Python 2 to 3, developers need to switch back and forth between interpreters while testing and making changes. That means we need to support both interpreters on Precise and Quantal, as well as Raring.
• REP-0122 installs Python modules under lib/pythonX.Y/dist-packages/, where X.Y is the Python version currently in use on the build system. That prevents releasing a binary package that supports multiple versions. Current Debian and Ubuntu practice installs the source modules in share/pyshared/, then compiles them into lib/pythonX.Y/dist-packages/ for all installed Python versions.
• The ros-infrastructure scripts (like rospkg), which are distro-independent, seem to do this correctly already. But the scripts that are released with a ROS distro (like ros-hydro-rospy) do not.

[tfoote]

So on closer reading of the Python 3 Ubuntu page it seems that we've gotten ahead of ourselves. Or Ubuntu has rolled back their ambition to roll out Python 3. https://wiki.ubuntu.com/Python/3

It looks like they are now targeting 14.04 to be the first distro without 2.7 installed, but at least in the mean time 2.7 will continue to be installed, and remain the default.

Dirk did some testing on a Raring vm and confirmed the following:

By default Raring installs Python 2.7 and 3.3 and 2.7 is the default (/usr/bin/python):
  /usr/bin/python -> python2.7
  /usr/bin/python2 -> python2.7
  /usr/bin/python2.7
  /usr/bin/python3 -> python3.3
  /usr/bin/python3.3

Based on this Raring from our point of view should be no different than Precise or Quantal, however we should keep up the effort to make our code bilingual. This just takes the short term pressure off for the next release.

[jack-oquin]

That is a huge relief. I was getting really worried about the viability of Hydro on Raring. I bet they were surprised by the level of difficulty, like we were.

The Hydro Python 3 scare was probably a blessing. It encouraged us to think about how to do the transition, and to realize how big an infrastructure change is likely to come with it. Some of those updates should probably be released before Ubuntu 14.04 and J-turtle.

I'll make another REP-0003 update reflecting this.

[tfoote]

REP 003 updated in ros-infrastructure/rep#35

[tfoote]

REP 3 discussion for Python 3 compatibility is continuing on ros-infrastructure/rep#64