[FEATURE REQUEST] pip install OpenImageIO #3249
Comments
c0ffymachyne
changed the title
lgritz
added
help wanted
|
This would be a great thing. But it's well outside my expertise, so what I really need is for somebody who has more Python ecosystem knowledge, preferably somebody who has done this exact thing on another project, to step forward and take the lead on this task. I think that this has recently been done on both OpenColorIO and OpenTimelineIO, so maybe there is somebody who both works on those projects and OIIO and would like to give this a try and can do it efficiently since they already know how it works? |
|
upvote on this one! |
|
Please! It's sooo complicated to get oiio working with Python in Windows, especially for casual Python users. Just spent an entire day figuring it out (including little idiosyncrasies like making sure it builds with ocio) via Microsoft vcpkg - and even them I'm stuck with Python 3.10 and have no idea how to build it for 3.9. Anyone?? |
|
I'm currently working on that. I have fully static builds working on Linux and Windows. I haven't started macOS yet. I'm aiming for having a library that comes with most plugins. The only excluded plugins are OpenVDB, R3D, Nuke, DCMTK, OpenCV and the |
|
Oh, and I'll need testers eventually. If you (or someone you know) are interested, please let me know and also tell which platform and architecture you would like to test on. Thanks! |
|
I can test on mac, windows and centos 7. |
|
willing to test on ubuntu 20.04 x86_64 |
|
This would be a really big help, especially for pure python developers (or tech-artists). |
|
@JeanChristopheMorinPerso pinging - any updates on this? can we help you test? thanks! |
|
I'll update when I have more news. The work I'm doing is not sponsored by any company, which means I do that in my own free time. |
|
Windows users are looking forward to this function, the person who makes it will be respected by everyone |
|
I can probably give an update. I had all platforms working last year. But I'm blocked on licensing issues. OIIO has a lot of dependencies (direct and transitive) and there is a lot of licenses involved. |
|
I'm happy to help test on macOS Ventura if needed. A pip installer would be great, or even improved docs on installing for python. @JeanChristopheMorinPerso, how much work needs to be done? I'd be happy to help with writing a |
|
Hi @ziggycross, I don't have any update. I'm mainly waiting on the TSC to be fully formed and in place and for things to settle before starting to work on this again. The reason being we'll have to figure out which plugin is safe to ship with the wheels in terms of licenses. In other words, I'm not blocked on the software side, I'm blocked on the legal stuff. Creating the python package isn't too complex and I'd say is even quite straightforward for someone with compilation and packaging experience. What's complex is really figuring out the right set of plugins, taking into consideration their licenses and the compatibility of these licenses with OIIO's license and how these dependencies can be redistributed without violating their licenses and without changing OIIO's license. And the thing is this is out of my area of expertise and I'm also not in a position of making these decisions anyway. |
|
Thanks for the thoughtful response @JeanChristopheMorinPerso. Do you think perhaps it would be possible to include the files needed to run an offline pip install? That way anyone who has built the necessary files or installed OIIO from homebrew, etc, could install it to their environment with a single command. If you think this could be a good idea, I'd be happy to give it a shot. |
|
Very familiar with Python packaging and can hit this one on Dev Days if folks are still interested |
|
Hopefully @JeanChristopheMorinPerso can fill you in on where he is with this, and there is some meaningful way to divide up the work that's left to do? |
|
@JeanChristopheMorinPerso Do you have a WIP branch somewhere? I can ask @tieguy for help with the licensing. |
|
Hi @aclark4life! I'm quite happy to hear that someone with python packaging experience is wants to help! I have plenty of packaging experience too, but I'm lacking time. So all help is very welcome! I have a branch (actually two, but the branch I'm pointing you to is the main one that has all the changes). See https://github.com/JeanChristopheMorinPerso/oiio/compare/master..python_wheels_windows. The branch is named windows something, but it also contains all the stuff for macOS and Linux. The last time I pushed to this branch, everything was compiling, linking and everything was working on all platforms. The last commit was probably a year ago, so I'm sure there will be tons of conflicts and plenty of things to change. But it has the general idea in it. The general idea is to statically link everything we legally can, support all platforms, support python 3.7+ and we need wheels. The wheels shouldn't be too heavy (I think what I had done was to statically link everything in the libiio (or whatever the library name is), dynamically link it to the python binding and the CLI apps). You'll see that I use conan to build everything statically. Conan was chosen because it was the most straight forward way to do that without hundreds of lines of cross-platform bash to get all dependencies that we can in static, no shared. The elephant in the room is ffmpeg and other GPL dependencies. But since the OIIO is now part of the ASWF, we have access to LF lawyers to answer any question we have licensing wise. It would actually be good to go through LF because we have other projects that will also have to deal with this type of stuff (OpenRV, etc). But I don't think for now that we need to care much about GPL stuff. Just getting everything working will be a good first step, and we can disable some dependencies if required, and dynamically link others if required, etc. As you can see from what I just wrote, this issue is more a compilation exercise than python packaging in itself. Though there is still a big packaging part in the sense that we need to ship something that is rock solid (ie OIIO users should have to deal with DLL hell when pip install OIIO). Compiling is the big issue, and getting everything working and something that won't generate a huge amount of support once released (which is quite important and why I want to statically link stuff). Feel free to take a look at what I did, ask questions and give us your opinion! |
|
@JeanChristopheMorinPerso Thanks! OK I agree this is not a one day task, but if I work on it between now and dev days, then we can probably get something ready to merge by then 🚀 Here's a PR from @JeanChristopheMorinPerso branch to a rebased master so folks can follow along: I made a "brute force" attempt to rebase the branch against master, but gave up about half way through when I realized the result wouldn't be coherent 😄 So, falling back to a PR and I'll continue to review all the moving parts in hopes if starting to make some tangible progress. First q:
Additional comments:
|
|
@aclark4life libaom is a dependency of libheif if I'm not mistaken. |
|
@JeanChristopheMorinPerso Right and neither build script (aom, heif) is currently in master so my question is: were these added for the sole purpose of creating wheels? I assume "yes" and if that's confirmed then we may want to break out tasks for this issue into at the very least two tracks, maybe three:
In other words, I'm making the assumption that a PR to master for a dep like this could get merged without issue whereas the entire wheel creation process has a much broader scope. |
|
OIIO has a direct dependency on Anyway, I wouldn't focus on this right now. I would focus on getting something working first, and then we can worry on which plugins we will bundle or not, and which dependency we dynamically or statically link. |
|
Right! As far as that goes, a rebase or merge of your code with current master could still happen if I put some effort into it, but I'm not entirely convinced that a new branch including lessons learned isn't a better way to go. Opinions welcome. Here's what your branch build looks like for me: Not sure exactly what's going on there so right now I'm in a test repo experimenting with C++, CMake, Python. |
|
The As for starting from scratch, that's an option and probably what I would do. Lots of things changed since I made this experiment, so it'll be easier to start from scratch. I think my modifications to CMakeLists.txt, conanfile.txt, the pyproject.toml + setup.py can probably be taken as is. The patch on OCIO is probably not needed anymore, the But feel free to start from scratch. That's what I would do (though, like I just said, I would cherry-pick stuff). One thing I'd like to do though is to not use setuptools. I'd prefer to use https://github.com/scikit-build/scikit-build-core instead. Note that this is not scikit-build. It's a new build backend related to scikit-build but is not scikit-build-core. This backend simplifies quite a bit of things since it nateively understands cmake and we don't need a setup.py file anymore. We would only need a pyproject.toml. |
|
OK here's a new branch with a new pyproject.toml that appears to build something with scikit-build-core and no setuptools! As for cherry-picking, here's a list of commits, if possible, please let me know which ones to keep and I'll deal with all the conflicts.
|
|
Hi @aclark4life, it's a little bit difficult to point to which commit can be cherry picked. The commits where made organically and not methodologically. I tried to be methodical, but thins evolved through the commits. So here is some hints: I think my modifications to CMakeLists.txt (and the other cmake files), conanfile.txt, the pyproject.toml + setup.py can probably be taken as is (or setup.py dropped if we can use |
|
@aclark4life @JeanChristopheMorinPerso Hello, just curious if there has been any progress on this? I also tried looking into a building the most recent release (2.5.5.0) using |
|
@taranlu-houzz Loosely targeting end-of-year to finish something that can be merged. |
|
Hello @aclark4life, just curious if there has been any update on this? |
|
@taranlu-houzz Sorry, didn't get to it by end of last year! Now waiting for @lgritz to review the latest… it's not done, but I think he is targeting fall to coincide with OIIO release. Any suggestions for specific tasks to move PR forward are welcome! I don't mind doing the work, but I get distracted by other projects, so thank you for re-raising. |
|
huhu, small studio dev here coming from this -> #4011 (comment) would also be happy to test this on windows if there's a branch around. |
Can anyone help answer the above? What are the right set of plugins? What are their licenses and those licenses' compatibility with OIIO including redistribution? E.g.
If we could get that information combined with knowing most of the build is handled by cibuildwheel maybe we can build some momentum here. |
|
@aclark4life Hey, sorry for not responding to your earlier post. I don't know enough about how the licenses would interact, but I would imagine this would be the kind of question that we might be able to get some insight on from the Blender devs. They are pretty active on the Blender dev chat, and while this isn't directly related to Blender, I don't think that it is too far off topic that it would be closed immediately (although I could be wrong). If I remember, I can try to make a post there: https://devtalk.blender.org/ |
|
@aclark4life That's a good question. I think MIT, BSD, and Apache-2.0 are all no problem at all. OIIO is already primarily Apache-2.0 with a few bits and pieces that are those other licenses, and I think account for almost all the dependencies are within this set (or are compatible). I think that only a couple of the format "plugins" depend in any way on dependencies with incompatible licenses (GPL variants are the main thing to watch for here). In theory, this shouldn't be any different than a distribution like RedHat or Homebrew distributing OIIO binaries in some form. I'll scan the list of plugins and their dependencies and remind myself which might have any GPL-like components. Off the top of my head, my recollection is that the only problematic ones are
LGPL is fine to link dynamically, so I believe the only consideration there is that if those plugins are enabled, we must link them as dll/so dynamic libraries, not statically link them. But for ffmpeg, which I believe is the only dependency we have that has any GPL license, there is a build-time (of ffmpeg itself, I mean) choice of whether to build with or without the GPL components. I think it's important to be sure we're using an ffmpeg library that was built without the GPL part. (See https://ffmpeg.org/legal.html) If any of these get tricky, we can just avoid the whole issue by disabling particular formats for the sake of the python wheel, and people who need those formats can be advised to build from source. |
|
Any progress on this topic? |
|
Planning to work on it again at Dev Days … also see: https://gist.github.com/zachlewis/20a3040e75b568526572f4007b3aa05e |
### Summary This PR is the spiritual successor to #4011. It implements a `scikit-build-core`-based python build-backend, making it possible to use `pip install .` to build from source; and it adds a Github workflow for building with `cibuildwheel` and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.13, across major operating systems and architectures. When you `pip install OpenImageIO`, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary. ### PEP-Compliant Packaging: `pyproject.toml` The `pyproject.toml` file is organized in three parts: 1. **Package metadata**: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth. 2. **scikit-build-core options**: governs how `pip install ...` interacts with `cmake`. 3. **cibuildwheel options**: additional steps and considerations for building, packaging, and testing relocatable wheel build artifacts in isolated environments. ### Additions to `__ init __.py` Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH). This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs. It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in `pyproject.toml`, each item under `[project.scripts]` is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name. ### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel` [cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions. Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size, compared to "Release" ### "Wheels" Github workflow I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide. Note: A "trusted publisher" must be set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ ### Other Changes I made some minor adjustments to `pythonutils.cmake` and `fancy_add_executable.cmake` that only affect scikit-build-core-based installs: - -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc. ## Tests `cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to `import OpenImageIO`; that the actual binary executable `oiiotool` is properly packaged and exists in the expected location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all runtime dependencies are found. ## Inspiration, Credit, Prior Art - @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See [#3249](#3249), and [#4011](#4011), as well as JCM's [python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels) and [python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows) branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging. - OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here. - @joaovbs96's help and patience with testing stuff on Windows. --------- Signed-off-by: Zach Lewis <[email protected]> Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Basile Fraboni <[email protected]> Signed-off-by: Vlad (Kuzmin) Erium <[email protected]> Signed-off-by: Joseph Goldstone <[email protected]> Signed-off-by: Darby Johnston <[email protected]> Signed-off-by: Jeremy Retailleau <[email protected]> Signed-off-by: zachlewis <[email protected]> Co-authored-by: Larry Gritz <[email protected]> Co-authored-by: Anton Dukhovnikov <[email protected]> Co-authored-by: Basile Fraboni <[email protected]> Co-authored-by: Vlad (Kuzmin) Erium <[email protected]> Co-authored-by: Joseph Goldstone <[email protected]> Co-authored-by: Darby Johnston <[email protected]> Co-authored-by: Jeremy Retailleau <[email protected]> Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
…eFoundation#4428) ### Summary This PR is the spiritual successor to AcademySoftwareFoundation#4011. It implements a `scikit-build-core`-based python build-backend, making it possible to use `pip install .` to build from source; and it adds a Github workflow for building with `cibuildwheel` and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.13, across major operating systems and architectures. When you `pip install OpenImageIO`, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary. ### PEP-Compliant Packaging: `pyproject.toml` The `pyproject.toml` file is organized in three parts: 1. **Package metadata**: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth. 2. **scikit-build-core options**: governs how `pip install ...` interacts with `cmake`. 3. **cibuildwheel options**: additional steps and considerations for building, packaging, and testing relocatable wheel build artifacts in isolated environments. ### Additions to `__ init __.py` Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH). This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs. It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in `pyproject.toml`, each item under `[project.scripts]` is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name. ### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel` [cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions. Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size, compared to "Release" ### "Wheels" Github workflow I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide. Note: A "trusted publisher" must be set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/ ### Other Changes I made some minor adjustments to `pythonutils.cmake` and `fancy_add_executable.cmake` that only affect scikit-build-core-based installs: - -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc. ## Tests `cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to `import OpenImageIO`; that the actual binary executable `oiiotool` is properly packaged and exists in the expected location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all runtime dependencies are found. ## Inspiration, Credit, Prior Art - @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See [AcademySoftwareFoundation#3249](AcademySoftwareFoundation#3249), and [AcademySoftwareFoundation#4011](AcademySoftwareFoundation#4011), as well as JCM's [python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels) and [python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows) branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging. - OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here. - @joaovbs96's help and patience with testing stuff on Windows. --------- Signed-off-by: Zach Lewis <[email protected]> Signed-off-by: Larry Gritz <[email protected]> Signed-off-by: Anton Dukhovnikov <[email protected]> Signed-off-by: Basile Fraboni <[email protected]> Signed-off-by: Vlad (Kuzmin) Erium <[email protected]> Signed-off-by: Joseph Goldstone <[email protected]> Signed-off-by: Darby Johnston <[email protected]> Signed-off-by: Jeremy Retailleau <[email protected]> Signed-off-by: zachlewis <[email protected]> Co-authored-by: Larry Gritz <[email protected]> Co-authored-by: Anton Dukhovnikov <[email protected]> Co-authored-by: Basile Fraboni <[email protected]> Co-authored-by: Vlad (Kuzmin) Erium <[email protected]> Co-authored-by: Joseph Goldstone <[email protected]> Co-authored-by: Darby Johnston <[email protected]> Co-authored-by: Jeremy Retailleau <[email protected]> Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
Hi,
Please automate and unify python builds and make it available via
pip install OpenImageIOExemplary project providing build for all possible python versions under releases: https://github.com/isl-org/Open3D/releases/tag/v0.14.1
The text was updated successfully, but these errors were encountered: