Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Python wheels workflow and build backend #4428

Merged
merged 200 commits into from
Jan 21, 2025

Conversation

zachlewis
Copy link
Contributor

@zachlewis zachlewis commented Sep 16, 2024

Description

Summary

This PR is the spiritual successor to #4011. It implements a scikit-build-core-based python build-backend, making it possible to use pip install . to build from source; and it adds a Github workflow for building with cibuildwheel and publishing to pypi.org binary distributions (bdists) of the Python module / extensions / CLI tools for cpython 3.8-3.13, across major operating systems and architectures.

When you pip install OpenImageIO, pip attempts to retrieve an OpenImageIO bdist from pypi.org for the host's platform / architecture / Python interpreter. If it can't find something appropriate, pip will attempt to build locally from the OpenImageIO source distribution (sdist), downloading and temporarily installing cmake and ninja if necessary.

PEP-Compliant Packaging: pyproject.toml

The pyproject.toml file is organized in three parts:

  1. Package metadata: standard attributes identifying and describing the Python package, its run-time and build-time requirements, entry-points to executable scripts, and so forth.
  2. scikit-build-core options: governs how pip install ... interacts with cmake.
  3. cibuildwheel options: additional steps and considerations for building, packaging, and testing relocatable wheel build artifacts in isolated environments.

Additions to __ init __.py

Previously, we were using a custom OpenImageIO/__ init__.py file to help Python-3.8+ on Windows load the shared libraries linked by the Python module (i.e., the .dll files that live alongside oiiotool.exe under $PATH).

This PR adds an additional method for loading the DLL path, necessitated by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the .../site-packages/OpenImageIO/bin directory. This provides a means for exposing Python script "shims" for each CLI tool, installed to platform-specific locations under $PATH, while keeping the actual binaries in a static location relative to the dynamic libraries. Upshot is, in pyproject.toml,
each item under [project.scripts] is turned into a Python script upon installation that behaves indistinguishably to the end user to the CLI binary executable of the same name.

Relocatable Binary Distributions with cibuildwheel + repairwheel

cibuildwheel is a widely-used tool for drastically streamlining the process of building, repairing, and testing Python wheels across platforms, architectures, interpreters, and interpreter versions.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to "MinSizeRel" to optimize for size (instead of speed) -- this seems to shave ~1.5MB off each .whl's size, compared to "Release"

"Wheels" Github workflow

I straight-up copied .github/workflows/wheel.yml from OpenColorIO and made a few OIIO-specific modifications. When pushing a commit tagged v3*, the workflow will invoke a platform-agnostic "build sdist" (source distribution) task, followed by a series of tasks for building OIIO wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts; followed finally by a task for publishing the build artifacts to pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few changes to OpenColorIO's wheel.yml as I could get away with; but in the future, we can also build wheels for the PyPy interpreter, and possibly pyodide.

Note: A "trusted publisher" must be set up on pypi.org. See https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

Other Changes

I made some minor adjustments to pythonutils.cmake and fancy_add_executable.cmake that only affect scikit-build-core-based installs:

  • -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property to point to the relative path to the dynamic libraries, for the Python module and CLI tools, respectively. This helps ensure that pip-based builds and installs from source (as opposed to installs from repaired, pre-built wheels) yield relocatable, importable packages, without needing to mess with $LD_LIBRARY_PATH etc.

Tests

cibuildwheel tests if oiiotool --buildinfo runs. If that command elicits code zero, it means the "oiiotool" Python script installed by the wheel is able to import OpenImageIO; that the actual binary executable oiiotool is properly packaged and exists in the expected location (e.g., at .../site-packages/OpenImageIO/bin); and that all runtime dependencies are found.

Inspiration, Credit, Prior Art

  • @aclark4life's and @JeanChristopheMorinPerso's efforts + direction + discussion + advice. See #3249, and #4011, as well as JCM's python_wheels and python_wheels_windows branches. This PR is an attempt to leverage OIIO-2.6+ self-building-dependency features with # 4011's minimalist and modern approach to packaging.
  • OpenColorIO -- I tried to copy as much as I could from @remia et al's fantastic work with all things wheels-related. The __init __.py modifications, the way we're wrapping the CLI tools, and the github Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about reinventing the wheel here.
  • @joaovbs96's help and patience with testing stuff on Windows.

zachlewis and others added 30 commits September 16, 2024 10:55
The build system has been updated to specifically detect the Python3 Development.Module meta component, as opposed to the entire Development component. This allows for better compatibility with python distributions that do not provide the Development.Embed component, which is only required for projects that ship embedded Python interpreters. The changes have been made in CMakeLists.txt and pythonutils.cmake files.

Signed-off-by: Zach Lewis <[email protected]>
Scikit-build-core is used for collecting CMake and Ninja as needed, and for invoking the build. When invoked via cibuildwheels, `repairwheel` is used after each build to re-bundle and relink the shared library dependencies into properly redistributable whl archives. The command-line tools are exposed under the [project.scripts] section.
This commit incorporates or is otherwise inspired by similar efforts by @aclark4life and @JeanChristopheMorinPerso, as well as @remia's work on the OpenColorIO wheels.

Signed-off-by: Zach Lewis <[email protected]>
Use Apache-2.0 license identifier instead of BSD-3-Clause.
Add "OpenImageIO Contributors" / "[email protected]" as author.
I could not bring myself to remove Larry as an author and maintainer.

Signed-off-by: Zach Lewis <[email protected]>
…tic libdeflate

It's now possible to disable building of shared `libdeflate` libs.

Also, we're checking for and aliasing `libdeflate` in `externalpackages.cmake`, just before checking for TIFF, as opposed to only doing so within `build_TIFF.cmake`. This change is necessary for certain build systems and pipelines that utilize cached dependency builds.

Specifically, when building wheels for multiple versions of cpython, `cibuildwheel` would complete the first build, and then throw an exception on the *second* build re: not being able to find `Deflate::Deflate`. Moving the aliasing above the check for TIFF ensures that the expected aliasing always takes place, whether or not TIFF needs to be built; whereas before, we were only creating the alias when initially building TIFF.

Signed-off-by: Zach Lewis <[email protected]>
Build and link missing libjpeg-turbo shared + static libs

Signed-off-by: Zach Lewis <[email protected]>
Instead of creating a separate OpenImageIO.OpenImageIO.command_line module for the CLI shims, move the CLI shim logic up to a "_command_line()" method in OpenImageIO.__init__.py.
(Maybe this method should still be called "main()" though?)

This also means the module is technically importable from OpenImageIO.OpenImageIO, but that's an improvement over OpenImageIO.OpenImageIO.OpenImageIO...!

Signed-off-by: Zach Lewis <[email protected]>
…oftwareFoundation#4358)

As suggested by Moritz Moeller

Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
…twareFoundation#4359)

* Get rid of some obsolete cmake code.

* Movement (but no change) to some parts of CMakeLists.txt, primarily to
make it closer to the corresponding file in OSL to make it easy for me
to diff them and port innovations back and forth between them.

* Some typo/etc fixes

* Remove unused OIIO_UNUSED_OK macro that's been deprecated since 2.0,
and OIIO_CONSTEXPR and OIIO_CONSTEXPR14, neither of which have been
needed for years.

Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
This seems to break builds under certain circumstances. Better to handle the problem with cached rebuilds another way, either in a FindLibdeflate.cmake, or by always locally-building libdeflate and TIFF.

Signed-off-by: Zach Lewis <[email protected]>
…directory, under module root

Added conditional logic to set relative RPATHs when building with scikit-build. This change ensures that the Python module and compiled cli tools correctly find all built dynamic libraries relative to a shared root, and keeps distributions self-contained and relocatable (i.e., without requiring a `repairwheel` step).

Signed-off-by: Zach Lewis <[email protected]>
* The build-directory is no longer hard-coded to a local "build_wheels" path.
The 'repairwheel' tool needs to know where it can find any dynamic libs compiled by the build system; and, frustratingly, doesn't seem to consider libraries already bundled in the whl. We can either point repairwheel to directory within an unzipped whl; or, we can point repairwheel back to where the compiled dependencies live inside the build-directory, under .../dist/deps/lib. There isn't a straightforward way of passing information from skbuild to repairwheel directly; but here, we're using cibw to set an environment variable dictating to where scikit-build-core builds, which we can also reference in the repair-wheel step.

* Always (re)build the TIFF dependency when building local wheels.
This is a workaround for an issue where "Deflate::Deflate" either can't be found, or can't be redeclared, under certain circumstances. A more robust solution might be to instead write a FindLibdeflate.cmake module that adds an alias for Deflate::Deflate as needed.

* Add "wheelhouse" directory created by cibuildwheel to .gitignore.

Signed-off-by: Zach Lewis <[email protected]>
We don't want to locally build TIFF when it already exists; but if we build static TIFF libs locally once, we have to rebuild every time (i.e., for subsequent builds), or else "Deflate::Deflate" is forgotten. This commit forces TIFF to be rebuilt every time, but only for cibuildwheel unix builds. Under normal circumstances, only missing dependencies will be locally built.

Signed-off-by: Zach Lewis <[email protected]>
Update pyproject.toml configuration

The project's TOML file has been updated to reflect changes in the build system, dependencies, and licensing. The scikit-build-core version requirement has been bumped up, and new tools have been added for wheel repair and invocation. The license text has also been simplified to only include Apache-2.0.

Signed-off-by: Zach Lewis <[email protected]>
The wheel repair command in the pyproject.toml file has been refactored to use an invoke task. he before-build step now also installs invoke. A new tasks.py file has been added with a 'wheel_repair' task that slims down and repairs the wheel file.

Step 1: Remove `lib`, `include`, `share` directories from wheel
Step 2: Let `repairwheel` fix the wheel with freshly-built libraries found in {build_dir}/lib and {build_dir}/deps/dist/lib.

Signed-off-by: Zach Lewis <[email protected]>
Also, tiny bit of tidying.

Signed-off-by: Zach Lewis <[email protected]>
It's an OCIO dependency.

Signed-off-by: Zach Lewis <[email protected]>
Apparently MSVC is having trouble linking Python3::Python otherwise...

Signed-off-by: Zach Lewis <[email protected]>
- Lowered the required Python version from 3.9 to 3.7
- Added numpy as a new dependency
- Refined DLL loading for Windows in Python 3.8+
- Adjusted build verbosity settings
- Moved CMAKE_INSTALL_LIBDIR setting into platform-specific overrides for Linux and macOS only
- On Windows, do not make adjustments to the INSTALL_RPATH.
- Reorganized variables in cibuildwheel configuration for better readability
- Refactored variable names in __init__._call_program function to follow PEP8 guidelines

Signed-off-by: Zach Lewis <[email protected]>
Stolen nearly line-for-line from OpenColorIO's wheel workflow.

Signed-off-by: Zach Lewis <[email protected]>
…cademySoftwareFoundation#4365)

Fixes a simple copy/paste error in a copy constructor where the y
coordinate gets initialised twice instead of y and z.

Signed-off-by: Anton Dukhovnikov <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
…emySoftwareFoundation#4348)

Additional stack manipulation commands:
* `--popbottom` discards the bottom-of-stack image
* `--stackreverse` reverses the order of the whole stack
* `--stackclear` fully empties the stack
* `--stackextract <index>` moves the indexed item from the stack (index
0 means the top) to the top.

Make `--for` work correctly in both directions:
* Correct behavior if `--for` has a negative step value.
* If the end value is less than the begin value and no step is supplied,
assume -1 (analogous to how we usually assueme step=1 under ordinary
circumstances).
* Error if step is 0 (presume it will make an infinite loop).

Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
…TORY (AcademySoftwareFoundation#4368)

It's not just in oiiotool. This seems clearer and adheres to the env
variable naming convention we chose.

Reminder: This controls whether command line history gets written to
output image metadata by default by oiiotool and maketx. We historically
did it, but recently stopped because of security concerns.

Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
JPEG output configuration hint "jpeg:iptc" (default: 1), if set to 0,
will suppress IPTC block output to the file.

In the process, we changed the return type of utility function
encode_iptc_iim() to return true if anything was successfully encoded,
false otherwise.

Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Zach Lewis <[email protected]>
src/iconvert/CMakeLists.txt Outdated Show resolved Hide resolved
Copy link
Collaborator

@lgritz lgritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pointed out just a few things that are mostly stylistic: some places where you could have done something simpler, or added a settable control instead of changing a default behavior.

The only major concern I have is that the Windows builds are broken in the CI with this patch.

I'm not really enough of a Python guru or have ever made wheels myself, so don't have much basis to comment on the logic of the wheel building and whether it works.

It's not clear to me why I'd disabled WebP support in the first place...!

Signed-off-by: Zach Lewis <[email protected]>
OCIO exposes an `OCIO_USE_SOVERSION` cmake option to prevent the creation of "namelink" copies (e.g., libOpenColorIO.so, libOpenColorIO.so.2, libOpenColorIO.so.2.4.1, etc).

This would be a filesize concern for the wheels bdists if we were building and linking dynamic OCIO libs; but we're not, so it isn't.

Signed-off-by: Zach Lewis <[email protected]>
Instead of making adjustments to individual tool CMakeLists, take advantage of fancy_add_executable's "ENABLE_xxx" idiom, and disable all tools other than `maketx` and `oiiotool` all at once in the top-level CMakeLists.txt

Signed-off-by: Zach Lewis <[email protected]>
@zachlewis
Copy link
Contributor Author

I do believe we're in a good place now.

The only major concern I have is that the Windows builds are broken in the CI with this patch.

I've implemented the adjustments from #4590, which addresses the CI / Windows-2022 VS2022 failure.

The CI / Windows-2019 VS2019 failure isn't induced by the changes here; the failures seem to be occurring on other PRs as well.

@JeanChristopheMorinPerso
Copy link
Member

I did an analysis of the generated wheels and they look good to me. I looked at:

  • Included files in all wheels
  • Name of each wheel
  • DSOs on Linux (checked the RPATHs and linked libraries).
  • I did not check DSOs on macOS and Windows.

I'm making the assumption that they should be fine if the linux ones are fine since tests are run on the produced wheels on all platforms.

Outstanding work @zachlewis!

Copy link
Collaborator

@lgritz lgritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits pointed out in comments, but nothing high stakes that couldn't be fixed later. I trust you to decide what to do with the suggestions. Let me know when you've changed anything you are inclined to and are ready for a merge.

.github/workflows/wheel.yml Outdated Show resolved Hide resolved
.github/workflows/wheel.yml Outdated Show resolved Hide resolved
.github/workflows/wheel.yml Outdated Show resolved Hide resolved
.github/workflows/wheel.yml Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
zachlewis and others added 3 commits January 20, 2025 12:44
Addresses minor suggestions made by Larry

Signed-off-by: zachlewis <[email protected]>
It's not super intuitive what the "github.event_name != 'schedule' || github.repository == 'AcademySoftwareFoundation/OpenImageIO'" conditionals are doing when considered on their own; so clarify in a comment at the top of the workflow that the wheels workflow will run on forks when submitting a PR that modifies Python-related files; OR it will run according to a nightly schedule ONLY on the official upstream ASWF repo.

Signed-off-by: zachlewis <[email protected]>
@zachlewis
Copy link
Contributor Author

Thanks everyone, I really appreciate the feedback!

@lgritz, unless you (or anyone else) have any remaining questions / comments / doubts / fears / concerns, I think we're good to merge 🤞!

.github/workflows/wheel.yml Outdated Show resolved Hide resolved
src/python doesn't have an OpenImageIO subdirectory...!

Signed-off-by: zachlewis <[email protected]>
Copy link
Collaborator

@lgritz lgritz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Thanks for all the hard work on this, it's a very exciting addition to merge!

Comment on lines 392 to 405
upload_pypi:
needs: [sdist, linux, linux-arm, macos, macos-arm, windows]
runs-on: ubuntu-latest
permissions:
id-token: write
if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v3')
steps:
- uses: actions/setup-python@f677139bbe7f9c59b41e40162b753c062f5d49a3 # v5.2.0

- uses: actions/download-artifact@fa0a91b85d4f404e444e00e005971372dc801d16 # v4.1.8
with:
pattern: cibw-*
path: dist
merge-multiple: true
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, one last question here:
Is this correct to ensure that the upload to pypi is only going to happen on tagged releases from the main repo?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to apologize!

You mean, is there anything preventing folks from submitting PRs to their own forks' main branches and pushing a v3 tag locally to kick off an "unsanctioned" upload_pypi step?

Welp... on the pypi.org side, only AcademySoftwareFoundation/OpenImageIO is set up as a "trusted publisher" for the OpenImageIO project; so PyPi wouldn't even consider attempted uploads from other forks. As I understand it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, PyPi also strongly advocates for setting up a dedicated Github Actions environment for the publishing workflow, and I don't really know what that means...!

Copy link

@aclark4life aclark4life Jan 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think lines 392 to 405 ensures that, but if you look at the entire file, LGTM. I think the uploads actually happen on upload-artifact.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I took for granted that others can't actually publish the wheels on PyPI, but what I meant was, are we sure that (a) user forks won't attempt to do so and look like failed jobs, and (b) we are only pushing to PyPI for release tags, and not for every merged PR? Also, now that I mention that, we probably want to restrict the PyPI publishes to v3.0.* tags, and not accidentally appear to publish new releases if we happen to make a developer preview tag in main (which would be called something like v3.1.2.0-dev).

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That being said, PyPi also strongly advocates for setting up a dedicated Github Actions environment for the publishing workflow, and I don't really know what that means...!

Maybe this, something to do with running jobs from context created in other jobs I think: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/#defining-a-workflow-job-environment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I took for granted that others can't actually publish the wheels on PyPI, but what I meant was, are we sure that (a) user forks won't attempt to do so and look like failed jobs, and (b) we are only pushing to PyPI for release tags, and not for every merged PR? Also, now that I mention that, we probably want to restrict the PyPI publishes to v3.0.* tags, and not accidentally appear to publish new releases if we happen to make a developer preview tag in main (which would be called something like v3.1.2.0-dev).

If it's not there already, you may want to add a step for Test PyPI and merge with only that one enabled until you get used to the workflow. Then enable PyPI after sufficient testing.

.github/workflows/wheel.yml Outdated Show resolved Hide resolved
Co-authored-by: Larry Gritz <[email protected]>
Signed-off-by: zachlewis <[email protected]>
@lgritz
Copy link
Collaborator

lgritz commented Jan 21, 2025

OK, I'm out of questions or suggested changes. Let's do this!

@lgritz lgritz merged commit 280e1c7 into AcademySoftwareFoundation:main Jan 21, 2025
66 checks passed
@JeanChristopheMorinPerso
Copy link
Member

Awesome work @zachlewis and everyone else that were involved at one point or another!

@MrLixm
Copy link

MrLixm commented Jan 21, 2025

Thanks so much to everyone involved, I cannot overstate how game-changer this is for some of us !
It makes OpenImageIO so much more accessible and easier to recommend to less tech-savvy people.

If oiiotool is still shipped with the wheels (since I tested them), I'm pretty sure that using uv, its possible to make it one command away from being available to anyone from any context.

lgritz added a commit to lgritz/OpenImageIO that referenced this pull request Jan 21, 2025
…eFoundation#4428)

### Summary
This PR is the spiritual successor to AcademySoftwareFoundation#4011. It implements a
`scikit-build-core`-based python build-backend, making it possible to
use `pip install .` to build from source; and it adds a Github workflow
for building with `cibuildwheel` and publishing to pypi.org binary
distributions (bdists) of the Python module / extensions / CLI tools for
cpython 3.8-3.13, across major operating systems and architectures.

When you `pip install OpenImageIO`, pip attempts to retrieve an
OpenImageIO bdist from pypi.org for the host's platform / architecture /
Python interpreter. If it can't find something appropriate, pip will
attempt to build locally from the OpenImageIO source distribution
(sdist), downloading and temporarily installing cmake and ninja if
necessary.

### PEP-Compliant Packaging: `pyproject.toml`

The `pyproject.toml` file is organized in three parts:
1. **Package metadata**: standard attributes identifying and describing
the Python package, its run-time and build-time requirements,
entry-points to executable scripts, and so forth.
2. **scikit-build-core options**: governs how `pip install ...`
interacts with `cmake`.
3. **cibuildwheel options**: additional steps and considerations for
building, packaging, and testing relocatable wheel build artifacts in
isolated environments.

### Additions to `__ init __.py` 

Previously, we were using a custom OpenImageIO/__ init__.py file to help
Python-3.8+ on Windows load the shared libraries linked by the Python
module (i.e., the .dll files that live alongside oiiotool.exe under
$PATH).

This PR adds an additional method for loading the DLL path, necessitated
by differences between pip-based and traditional CMake-based installs.

It also adds a mechanism for invoking binary executables found in the
.../site-packages/OpenImageIO/bin directory. This provides a means for
exposing Python script "shims" for each CLI tool, installed to
platform-specific locations under $PATH, while keeping the actual
binaries in a static location relative to the dynamic libraries. Upshot
is, in `pyproject.toml`,
each item under `[project.scripts]` is turned into a Python script upon
installation that behaves indistinguishably to the end user to the CLI
binary executable of the same name.

### Relocatable Binary Distributions with `cibuildwheel` + `repairwheel`

[cibuildwheel](https://github.com/pypa/cibuildwheel) is a widely-used
tool for drastically streamlining the process of building, repairing,
and testing Python wheels across platforms, architectures, interpreters,
and interpreter versions.

Additionally, the cibuildwheel-based builds set CMAKE_BUILD_TYPE to
"MinSizeRel" to optimize for size (instead of speed) -- this seems to
shave ~1.5MB off each .whl's size, compared to "Release"

### "Wheels" Github workflow

I straight-up copied `.github/workflows/wheel.yml` from OpenColorIO and
made a few OIIO-specific modifications. When pushing a commit tagged
v3*, the workflow will invoke a platform-agnostic "build sdist" (source
distribution) task, followed by a series of tasks for building OIIO
wheels for cpython-3.8-3.13 on Windows, Linux (x86_64 + aarch64, new
libstdc++), and MacOS (x86_64 + arm64) and persisting build artifacts;
followed finally by a task for publishing the build artifacts to
pypi.org

Note: For the sake of simplicity and troubleshooting, I've made as few
changes to OpenColorIO's wheel.yml as I could get away with; but in the
future, we can also build wheels for the PyPy interpreter, and possibly
pyodide.

Note: A "trusted publisher" must be set up on pypi.org. See
https://docs.pypi.org/trusted-publishers/creating-a-project-through-oidc/

### Other Changes

I made some minor adjustments to `pythonutils.cmake` and
`fancy_add_executable.cmake` that only affect scikit-build-core-based
installs:
- -- namely, on Linux and macOS, I'm setting the INSTALL_RPATH property
to point to the relative path to the dynamic libraries, for the Python
module and CLI tools, respectively. This helps ensure that pip-based
builds and installs from source (as opposed to installs from repaired,
pre-built wheels) yield relocatable, importable packages, without
needing to mess with $LD_LIBRARY_PATH etc.


## Tests

`cibuildwheel` tests if `oiiotool --buildinfo` runs. If that command
elicits code zero, it means the "oiiotool" Python script installed by
the wheel is able to `import OpenImageIO`; that the actual binary
executable `oiiotool` is properly packaged and exists in the expected
location (e.g., at `.../site-packages/OpenImageIO/bin`); and that all
runtime dependencies are found.

## Inspiration, Credit, Prior Art
- @aclark4life's and @JeanChristopheMorinPerso's efforts + direction +
discussion + advice. See
[AcademySoftwareFoundation#3249](AcademySoftwareFoundation#3249),
and
[AcademySoftwareFoundation#4011](AcademySoftwareFoundation#4011),
as well as JCM's
[python_wheels](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels)
and
[python_wheels_windows](https://github.com/JeanChristopheMorinPerso/oiio/tree/python_wheels_windows)
branches. This PR is an attempt to leverage OIIO-2.6+
self-building-dependency features with # 4011's minimalist and modern
approach to packaging.
- OpenColorIO -- I tried to copy as much as I could from @remia et al's
fantastic work with all things wheels-related. The __init __.py
modifications, the way we're wrapping the CLI tools, and the github
Wheels workflow are lifted almost-verbatim from OCIO. Insert pun about
reinventing the wheel here.
  - @joaovbs96's help and patience with testing stuff on Windows.

---------

Signed-off-by: Zach Lewis <[email protected]>
Signed-off-by: Larry Gritz <[email protected]>
Signed-off-by: Anton Dukhovnikov <[email protected]>
Signed-off-by: Basile Fraboni <[email protected]>
Signed-off-by: Vlad (Kuzmin) Erium <[email protected]>
Signed-off-by: Joseph Goldstone <[email protected]>
Signed-off-by: Darby Johnston <[email protected]>
Signed-off-by: Jeremy Retailleau <[email protected]>
Signed-off-by: zachlewis <[email protected]>
Co-authored-by: Larry Gritz <[email protected]>
Co-authored-by: Anton Dukhovnikov <[email protected]>
Co-authored-by: Basile Fraboni <[email protected]>
Co-authored-by: Vlad (Kuzmin) Erium <[email protected]>
Co-authored-by: Joseph Goldstone <[email protected]>
Co-authored-by: Darby Johnston <[email protected]>
Co-authored-by: Jeremy Retailleau <[email protected]>
Co-authored-by: Jean-Christophe Morin <38703886+JeanChristopheMorinPerso@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build / testing / port / CI Affecting the build system, tests, platform support, porting, or continuous integration. python Python APIs roadmap This is a priority item on the roadmap for the next major release.
Projects
None yet
Development

Successfully merging this pull request may close these issues.