Release binaries

[mpowelson]

A few options. @Levi-Armstrong Feel free to edit to add more

• ROS-I PPA
• Snap package

Ideally this would happen in CI as if it is manual we are never going to do it.

[johnwason]

Conda is also an option. It claims to be platform independent, and would be good for Python integration. I don't have any experience using it.

[johnwason]

Also deploying PyPi wheel packages. This will require statically liking in all libraries and dependencies in to the Python C++ module.

[Levi-Armstrong]

Conda is also an option. It claims to be platform independent, and would be good for Python integration. I don't have any experience using it.

Yea, I recently started looking at Conda-Forge but not sure where to start. Have you every release anything on conda?

[traversaro]

Conda is also an option. It claims to be platform independent, and would be good for Python integration. I don't have any experience using it.

Yea, I recently started looking at Conda-Forge but not sure where to start. Have you every release anything on conda?

Regarding conda-forge, I was looking into packaging some of the dependencies of tesseract (in particular opw_kinematics and ifopt), so I would be happy to give an hand in packaging also tesseract.

At a first glance all the external dependencies used in https://github.com/ros-industrial-consortium/tesseract_ext are already available in conda-forge and updated to the latest version, so w.r.t. to that you should not need anything special.

I guess the first step is to ensure that you have a set of released version for tesseract and all its dependencies, and then the following step is adding recipes in https://github.com/conda-forge/staged-recipes .

[Levi-Armstrong]

@traversaro That is great news. We are cleaning up a few things and I plan to have the feature/CommandLanguage brach merged into master this week. We will create a tag prior to merging in this branch for those needing access to the current master. We will then create new tag release for the master after the merge of feature/CommandLanguage. I will reach out very soon once everything is merged and tagged.

[johnwason]

It is probably worth creating vcpkg portfile as well. I have found it is the best way to statically link a program, which is useful when creating a plugin or a final product.

[johnwason]

The conda ROS packages don't support Windows. It will be necessary to check the dependencies for tesseract to make sure they have Windows build scripts.

[traversaro]

The conda ROS packages don't support Windows.

To which package are you referring? The one in conda-forge such as are actually kind of deprecated (not sure how we could make clear this to users @wolfv @Tobias-Fischer) as it turns out that maintaining a large collection of software with no ABI compatibility policy on a distro such as conda-forge that heavily relies on ABI policies for ensure smooth transitions from one version to another (see https://conda-forge.org/blog/posts/2019-12-06-cfep09/).

The only current effort I am aware of for packaging ROS on conda is https://github.com/RoboStack/ros-noetic for ROS1 and https://github.com/RoboStack/ros-foxy for ROS2 (both are still kind of experimental). This repos effectively build a sort of "secondary" distribution of the ROS packages on the top of conda-forge, taking the necessary metadata from https://github.com/ros/rosdistro .

This is a "secondary" distribution in the sense that is not a distribution that builds the complete software stack, but just a layer of packages on a top of another existing distribution, similar to how bioconda (https://bioconda.github.io/) is built on top of conda-forge, or the ROS Buildfarm packages are a secondary distribution on the top of the official repositories of Ubuntu. The main difference w.r.t. official Ubuntu repositories is that with conda-forge is much easier and fast to add ROS-independent packages directly in the main conda-forge repository. In my previous comment I was referring to adding the ROS-independent tesseract core in conda-forge, for which (if I am not mistaken) all the required dependencies support Windows. Then, at the RoboStack level we are still sorting out how to deal with packages that are in conda-forge but are also release in the ROS Buildfarm (see https://github.com/RoboStack/ros-noetic/issues/20), but I think for packages such as tesseract it would be doable to just skip their build.

[johnwason]

I am looking at this package as an example:

https://github.com/conda-forge/ros-roscpp-feedstock

The package builds the library from source rather than using a pre-build ROS binary, so I don't think there is an ABI issue.

https://github.com/conda-forge/ros-roscpp-feedstock/blob/master/recipe/build.sh

This recipe doesn't have a bld.bat file, so it can't be built on Windows.

[wolfv]

Hi @johnwason thanks for looking at conda!
Indeed, our original ROS 1 packages on conda-forge did not build on Windows. However, we have started our own "channel" where we upload ROS packages, and build them on Azure Pipelines (very similar to how conda-forge operates).

You can find the channel on Anaconda.org here: https://anaconda.org/RoboStack/repo

As you can see, we have desktop-full for Windows, Linux (x64 and ARM64) and OS X.

These packages are built from here: http://github.com/robostack/ros-noetic and the build scripts can be found in the build_... branch: https://github.com/RoboStack/ros-noetic/tree/buildbranch_win

The build logs can be found here: https://dev.azure.com/roboforge/ros_pipelines/_build

So this was just a small overview over our current, ROS specific efforts. Due to how ROS & ROS distros work, we found this single-repo approach for build script generation simpler vs. maintaining many independent feedstocks. In that sense, the ros-*-feedstocks on conda-forge can be seen as deprecated.

However, we actively maintain the lower-level libraries on conda-forge: PCL, Ogre, Gazebo, FCL, ignition libraries, colcon, catkin_tools,... and we build purely on top of conda-forge. So I think for tesseract it would be awesome if it goes into conda-forge!
The only issue is that we can (currently) only put the non-ros dependent parts on conda-forge (otherwise we'd need to depend on the robostack channel).

I'd be very excited tesseract gets a PR to staged-recipes and I would be happy to help with the recipe in that PR!

[johnwason]

Thanks @wolfv

Currently the core tesseract, tesseract_planning, and tesseract_python don't depend on ROS. There is a separate tesseract_ros repo that does. @Levi-Armstrong do you plan on using one of the main channels or creating a ROS Industrial specific channel?

I am currently the technical lead on a team developing an open-source teach pendant based on Python, Tesseract, and Robot Raconteur, among other dependencies. My primary focus right now is getting Tesseract integrated into this project and find a way to easily deploy binaries to end users. My plan was originally to deploy on PyPi or docker, but I am thinking that using anaconda may be a better option if we can get dependencies with consistent ABI versions relatively easily. Having ROS available through anaconda would also make things much easier since then we don't need a system install. Currently the development of the teach pendant is pretty scattered, and we are beginning the effort to pull everything into the main organization repositories. I've created an issue there to discuss for that project: pyri-project/pyri-core#2

[wolfv]

That sounds great @johnwason! Using the conda constructor, you could even create a single-file installer for the project.

Does anyone of you want to start a PR to staged recipes? Otherwise I can look into it.

[johnwason]

@wolfv I am looking at the RoboStack GitHub. How does vinca select which packages to build? If tesseract is released to ROS, would it automatically pull the packages into RobotStack? Is it possible to use a catkin tools workspace with conda and have it build separate conda packages for each ROS package using vinca with the GitHub source repos as upstream?

[wolfv]

We select packages in the vinca.yaml files (https://github.com/RoboStack/ros-noetic/blob/master/vinca_linux_64.yaml). We use rosdistro tools to figure out all dependencies and create recipes where necessary.

If there are released tesseract versions into ROS, we can just select them in this vinca file and we'd get a recipe for them, yes.

Yes, basically that's possible and mostly what we're doing here (instead of catkin-tools we just use cmake etc).

However, if tesseract is just cmake based and doesn't need anything ROS specific (at least for some packages), then one can just write a recipe and build script manually. For example, a libignition recipe like this one https://github.com/conda-forge/libignition-common-feedstock/tree/master/recipe could be a good starting point?

[wolfv]

PS this is the source for vinca if you're curious: https://github.com/robostack/vinca

When you have a package.xml you can also try running vinca with a single package.xml (no rosdistro release necessary):

vinca -p package.xml -d ~/Programs/ros-noetic/

this generates a recipe and build scripts in place (very experimental, not great for general use just yet).

Additionally there are a bunch of complications (our version of vinca generates build scripts for boa which is a slightly enhanced version of conda-build, so you need to run boa build . etc...)

This is the recipe I get for tesseract_common

package:
  name: ros
  version: 2021.01.26

source:
  - path: ''
    folder: ros-noetic-tesseract-common/src/work

build:
  number: 0

about:
  home: https://www.ros.org/
  license: BSD-3-Clause
  summary: |
    Robot Operating System

extra:
  recipe-maintainers:
    - ros-forge

outputs:
  - package:
      name: ros-noetic-tesseract-common
      version: 0.1.0
    about:
      home: https://www.ros.org/
      license:
        - Apache 2.0
      summary: Contains common macros, utils and types used throughout
    extra:
      recipe-maintainers:
        - robostack
    build:
      number: 0
      script:
        sel(win): bld_catkin.bat
        sel(unix): build_catkin.sh
    requirements:
      build:
        - "{{ compiler('cxx') }}"
        - "{{ compiler('c') }}"
        - ninja
        - sel(unix): make
        - cmake
      host:
        - cmake
        - eigen
        - console_bridge
        - tinyxml2
        - eigen
        - console_bridge
        - tinyxml2
        - gtest
        - lcov
      run:
        - eigen
        - console_bridge
        - tinyxml2
        - ros-noetic-catkin
        - console_bridge
        - tinyxml2

Well, I hope this was not too confusing. If yes, then sorry!

[johnwason]

I am confused, why are there three different bullet packages? Isn't this going to cause a conflict?

[wolfv]

there is pybullet and bullet-cpp. both are actually built quite differently, and strictly speaking pybullet doesn't depend at all on bullet-cpp. It's also really difficult to properly package bullet. I tried for a while but had to give up because one would have to completely rewrite the build system.

py-bullet is built from a setup.py and statically links & vendors a lot of stuff.

bullet-cpp is built from a CMake and properly installs stuff but is not linked by pybullet (IIRC).

So there should not be any conflicts (except for this case with the wrong bullet "meta-package" that should get both, pybullet + bullet-cpp).

[wolfv]

I just wanted to link some past discussion from the issue tracker, but it's closed on bullet (https://github.com/bulletphysics/bullet3#issues) 🤷

[johnwason]

Thanks @wolfv and @Tobias-Fischer, switching to bullet-cpp package seems to have fixed the issue. The conda package is now working for Python 3.8 and 3.9 on Windows and Linux.

https://anaconda.org/Tesseract-Robotics/tesseract-robotics-superpack

[johnwason]

I tried to install ros-noetic-rospy from robostack after installing tesseract-robotics-superpack and got a large number of conflicts. Are conda-forge and robostack incompatible?

[johnwason]

Here is the conflict output:

tesseract_rospy_conda_conflict.txt

[Tobias-Fischer]

We are currently in the process of migrating our RoboStack packages to tinyxml2=9, which recently happened on conda-forge. Once this migration will have happened, we will be able to install the tesseract-robotics-superpack and ros-noetic packages side-by-side.

[johnwason]

The PyRI Open Source Teach Pendant was the main motivation for releasing Tesseract on Conda. See #629

[johnwason]

I am having another weird ompl dependency problem. See johnwason/tesseract-robotics-superpack-feedstock#9

[johnwason]

I created the tesseract-robotics-packaging organization to hold conda-smithy feedstocks. https://github.com/tesseract-robotics-packaging

I have a new project using Tesseract so I am working back working on conda packaging and Python development.

[Tobias-Fischer]

Hi @johnwason - we'd be more than happy to help out getting these feedstocks into conda-forge! You will benefit from all the automated migrations (staying in sync with the dependencies) and bots (automated version updates) etc. by using this approach. It's as simple as adding the contents in the recipe folder into a PR in the recipes/name-of-your-package folder of https://github.com/conda-forge/staged-recipes - @wolfv has the magic rights to merge in these PRs, which automatically creates a feedstock.

[johnwason]

Thanks @Tobias-Fischer I am also interested in moving the packages here sometime soon: https://github.com/robotraconteur-packaging . Once I get these packages building we can look into merging them in.

[johnwason]

I am seeing a version conflict on linux-64 when trying to use jsoncpp: conda-forge/jsoncpp-feedstock#35

[johnwason]

I created a new discussion thread to discuss conda problems so I don't create issues in the wrong place tracking down errors: tesseract-robotics-packaging/tesseract-robotics-feedstock#1

Please follow this thread if you are interested in helping.

[johnwason]

I have finished updating the conda superpack to the more recent tesseract release. I also tried breaking up the tesseract packages into individual conda packages, but I am getting various failures during tests that I don't have time to figure out. More discussion on conda packaging problems here: tesseract-robotics-packaging/tesseract-robotics-feedstock#1

Currently the conda package is using 32-bit precision floats for bullet collision detection. This isn't ideal, but the conda bullet package does not have 64 bit precision support. I am going to work on that sometime in the future.

To install the superpack, run:

conda install -c conda-forge -c tesseract-robotics tesseract-robotics-superpack

[Levi-Armstrong]

The primary reason for the 64bit bullet was to be able to detect 2mm contact distances accurately.

[johnwason]

I've released another update for tesseract-robotics-superpack. This update include Bullet double-precision floats, and several stability fixes. See johnwason/tesseract-robotics-superpack-feedstock#5, tesseract-robotics/tesseract_ext#44, #746, and #747

https://anaconda.org/Tesseract-Robotics/tesseract-robotics-superpack

[johnwason]

Tesseract is now available on Python PyPI (pip), see #754