Skip to content

Commit

Permalink
ci: refactor using a single steps workflow (AcademySoftwareFoundation…
Browse files Browse the repository at this point in the history
…#4545)

The logic for the GHA steps was repeated across multiple workflows and
platforms. Consolidate it all into a common "ci-steps" workflow, and
have all the other workflows reference that when possible. This should
hopefully make maintenance of the CI scripts easier in the future, with
less redundancy of the step logic.

The only major place where we deviate is that we have a separeately
specified set of steps for the "old" VFX Platform 2021 and 2022 based
tests (because we need to specify different container arguments to it).

---------

Signed-off-by: Larry Gritz <[email protected]>
  • Loading branch information
lgritz committed Dec 23, 2024
1 parent 32c668e commit 632717a
Show file tree
Hide file tree
Showing 6 changed files with 474 additions and 541 deletions.
126 changes: 38 additions & 88 deletions .github/workflows/analysis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,14 @@ on:
- main
- '*analysis*'
- '*sonar*'
paths-ignore:
- '**.md'
- '**.rst'
- 'docs/**'
paths:
- '**'
- '!**.md'
- '!**.rst'
- '!**/analysis.yml'
- '!**/docs.yml'
- '!**.properties'
- '!docs/**'
# Allow manual kicking off of the workflow from github.com
workflow_dispatch:
# Uncomment the following line if we want to run analysis on all PRs:
Expand All @@ -40,20 +44,46 @@ jobs:
# Exclude runs on forks, since only the main org has the SonarCloud
# account credentials.
if: github.repository == 'AcademySoftwareFoundation/OpenImageIO'
uses: ./.github/workflows/build-steps.yml
with:
nametag: ${{ matrix.nametag || 'unnamed!' }}
runner: ${{ matrix.runner || 'ubuntu-latest' }}
container: ${{ matrix.container }}
cc_compiler: ${{ matrix.cc_compiler }}
cxx_compiler: ${{ matrix.cxx_compiler }}
cxx_std: ${{ matrix.cxx_std || '17' }}
build_type: ${{ matrix.build_type || 'Release' }}
depcmds: ${{ matrix.depcmds }}
extra_artifacts: ${{ matrix.extra_artifacts }}
fmt_ver: ${{ matrix.fmt_ver }}
opencolorio_ver: ${{ matrix.opencolorio_ver }}
openexr_ver: ${{ matrix.openexr_ver }}
pybind11_ver: ${{ matrix.pybind11_ver }}
python_ver: ${{ matrix.python_ver }}
setenvs: ${{ matrix.setenvs }}
simd: ${{ matrix.simd }}
skip_build: ${{ matrix.skip_build }}
skip_tests: ${{ matrix.skip_tests }}
abi_check: ${{ matrix.abi_check }}
build_docs: ${{ matrix.build_docs }}
generator: ${{ matrix.generator }}
ctest_args: ${{ matrix.ctest_args }}
ctest_test_timeout: ${{ matrix.ctest_test_timeout }}
coverage: ${{ matrix.coverage || '0' }}
sonar: ${{ matrix.sonar || '0' }}
strategy:
fail-fast: false
matrix:
include:
- desc: sonar gcc9/C++14 py39 exr3.1 ocio2.2
nametag: static-analysis-sonar
os: ubuntu-latest
container: aswf/ci-osl:2023-clang15
vfxyear: 2023
container: aswf/ci-osl:2024-clang17
cxx_std: 17
python_ver: "3.10"
python_ver: "3.11"
simd: "avx2,f16c"
fmt_ver: 10.1.1
pybind11_ver: v2.10.0
pybind11_ver: v2.12.0
coverage: 1
# skip_tests: 1
sonar: 1
Expand All @@ -64,83 +94,3 @@ jobs:
CMAKE_UNITY_BUILD=OFF
CODECOV=1
CTEST_TEST_TIMEOUT=1200

runs-on: ${{ matrix.os }}
container:
image: ${{ matrix.container }}
env:
CXX: ${{matrix.cxx_compiler}}
CC: ${{matrix.cc_compiler}}
CMAKE_CXX_STANDARD: ${{matrix.cxx_std}}
USE_SIMD: ${{matrix.simd}}
FMT_VERSION: ${{matrix.fmt_ver}}
OPENEXR_VERSION: ${{matrix.openexr_ver}}
PYBIND11_VERSION: ${{matrix.pybind11_ver}}
PYTHON_VERSION: ${{matrix.python_ver}}
# DEBUG_CI: 1
steps:
# We would like to use harden-runner, but it flags too many false
# positives, every time we download a dependency. We should use it only
# on CI runs where we are producing artifacts that users might rely on.
# - name: Harden Runner
# uses: step-security/harden-runner@248ae51c2e8cc9622ecf50685c8bf7150c6e8813 # v1.4.3
# with:
# egress-policy: audit # TODO: change to 'egress-policy: block' after couple of runs
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
fetch-depth: '0'
- name: Prepare ccache timestamp
id: ccache_cache_keys
run: echo "date=`date -u +'%Y-%m-%dT%H:%M:%SZ'`" >> $GITHUB_OUTPUT
- name: ccache
id: ccache
uses: actions/cache@13aacd865c20de90d75de3b17ebe84f7a17d57d2 # v4.0.0
with:
path: /tmp/ccache
key: ${{github.job}}-${{matrix.nametag}}-${{steps.ccache_cache_keys.outputs.date}}
restore-keys: ${{github.job}}-
- name: Build setup
run: |
${{matrix.setenvs}}
src/build-scripts/ci-startup.bash
- name: Dependencies
run: |
${{matrix.depcmds}}
src/build-scripts/gh-installdeps.bash
- name: Install sonar-scanner and build-wrapper
if: matrix.sonar == '1'
uses: sonarsource/sonarcloud-github-c-cpp@e4882e1621ad2fb48dddfa48287411bed34789b1 # v2.0.2
- name: Build
run: src/build-scripts/ci-build.bash
- name: Testsuite
if: matrix.skip_tests != '1'
run: src/build-scripts/ci-test.bash
- name: Code coverage
if: matrix.coverage == '1'
run: src/build-scripts/ci-coverage.bash
- name: Sonar-scanner
if: matrix.sonar == 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: |
which sonar-scanner
ls -l /__w/OpenImageIO/OpenImageIO/bw_output
echo "BUILD_OUTPUT_DIR is " "${{ env.BUILD_WRAPPER_OUT_DIR }}"
find . -name "*.gcov" -print
# sonar-scanner --define sonar.cfamily.build-wrapper-output="${{ env.BUILD_WRAPPER_OUT_DIR }}"
time sonar-scanner --define sonar.host.url="${{ env.SONAR_SERVER_URL }}" --define sonar.cfamily.build-wrapper-output="$BUILD_WRAPPER_OUT_DIR" --define sonar.cfamily.gcov.reportsPath="_coverage" --define sonar.cfamily.threads="$PARALLEL"
# Consult https://docs.sonarcloud.io/advanced-setup/ci-based-analysis/sonarscanner-cli/ for more information and options

# - uses: actions/upload-artifact@26f96dfa697d77e81fd5907df203aa23a56210a8 # v4.3.0
# if: failure()
# with:
# name: oiio-${{github.job}}-${{matrix.nametag}}
# path: |
# build/cmake-save
# build/testsuite/*/*.*
# !build/testsuite/oiio-images
# !build/testsuite/openexr-images
# !build/testsuite/fits-images
# !build/testsuite/j2kp4files_v1_5

205 changes: 205 additions & 0 deletions .github/workflows/build-steps.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,205 @@
# Copyright Contributors to the OpenImageIO project.
# SPDX-License-Identifier: Apache-2.0
# https://github.com/AcademySoftwareFoundation/OpenImageIO

##########################################################################
# Common steps for all CI workflows
##########################################################################

name: CI Steps

on:
workflow_call:
# This inputs receive values via the "with:" section in ci_workflow.yml
inputs:
build:
type: string
runner:
type: string
container:
type: string
cc_compiler:
type: string
cxx_compiler:
type: string
cxx_std:
type: string
build_type:
type: string
depcmds:
type: string
extra_artifacts:
type: string
fmt_ver:
type: string
opencolorio_ver:
type: string
openexr_ver:
type: string
pybind11_ver:
type: string
python_ver:
type: string
setenvs:
type: string
simd:
type: string
skip_build:
type: string
skip_tests:
type: string
abi_check:
type: string
build_docs:
type: string
generator:
type: string
ctest_args:
type: string
ctest_test_timeout:
type: string
coverage:
type: string
sonar:
type: string
nametag:
type: string

permissions: read-all


jobs:

steps:
name: "${{inputs.cxx_compiler}} c++${{inputs.cxx_std}} py${{inputs.python_ver}}"
runs-on: ${{ inputs.runner }}
container:
image: ${{ inputs.container }}

env:
CXX: ${{inputs.cxx_compiler}}
CC: ${{inputs.cc_compiler}}
CMAKE_BUILD_TYPE: ${{inputs.build_type}}
CMAKE_CXX_STANDARD: ${{inputs.cxx_std}}
CMAKE_GENERATOR: ${{inputs.generator}}
CTEST_ARGS: ${{inputs.ctest_args}}
CTEST_TEST_TIMEOUT: ${{inputs.ctest_test_timeout}}
USE_SIMD: ${{inputs.simd}}
FMT_VERSION: ${{inputs.fmt_ver}}
OPENCOLORIO_VERSION: ${{inputs.opencolorio_ver}}
OPENEXR_VERSION: ${{inputs.openexr_ver}}
PYBIND11_VERSION: ${{inputs.pybind11_ver}}
PYTHON_VERSION: ${{inputs.python_ver}}
ABI_CHECK: ${{inputs.abi_check}}
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION: node16
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true

steps:
# - uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
# # Note: can't upgrade to actions/checkout 4.0 because it needs newer
# # glibc than these containers have.
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
fetch-depth: '0'
- name: Prepare ccache timestamp
id: ccache_cache_keys
shell: bash
run: echo "date=`date -u +'%Y-%m-%dT%H:%M:%SZ'`" >> $GITHUB_OUTPUT
- name: ccache
id: ccache
uses: actions/cache@e12d46a63a90f2fae62d114769bbf2a179198b5c # v3.3.3
# Note: can't upgrade to actions/cache 4.0 because it needs newer
# glibc than these containers have.
with:
path: ./ccache
key: ${{github.job}}-${{inputs.nametag}}-${{steps.ccache_cache_keys.outputs.date}}
restore-keys: ${{github.job}}-
save-always: true
- name: Setup Nuget.exe (Windows only)
if: runner.os == 'Windows'
uses: nuget/setup-nuget@a21f25cd3998bf370fde17e3f1b4c12c175172f9 # v2.0.0
- name: Build setup
shell: bash
run: |
${{inputs.setenvs}}
src/build-scripts/ci-startup.bash
- name: Dependencies
shell: bash
run: |
${{inputs.depcmds}}
if [[ "$RUNNER_OS" == "Linux" ]]; then
src/build-scripts/gh-installdeps.bash
elif [[ "$RUNNER_OS" == "macOS" ]]; then
src/build-scripts/install_homebrew_deps.bash
elif [[ "$RUNNER_OS" == "Windows" ]]; then
src/build-scripts/gh-win-installdeps.bash
fi
- name: Install sonar-scanner and build-wrapper
if: inputs.sonar == '1'
uses: sonarsource/sonarcloud-github-c-cpp@e4882e1621ad2fb48dddfa48287411bed34789b1 # v2.0.2
- name: Build
if: inputs.skip_build != '1'
shell: bash
run: src/build-scripts/ci-build.bash
- name: Testsuite
if: inputs.skip_tests != '1'
shell: bash
run: src/build-scripts/ci-test.bash
- name: Code coverage
if: inputs.coverage == '1'
run: src/build-scripts/ci-coverage.bash
- name: Sonar-scanner
if: inputs.sonar == 1
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
run: |
which sonar-scanner
ls -l /__w/OpenImageIO/OpenImageIO/bw_output
echo "BUILD_OUTPUT_DIR is " "${{ env.BUILD_WRAPPER_OUT_DIR }}"
find . -name "*.gcov" -print
# sonar-scanner --define sonar.cfamily.build-wrapper-output="${{ env.BUILD_WRAPPER_OUT_DIR }}"
time sonar-scanner --define sonar.host.url="${{ env.SONAR_SERVER_URL }}" --define sonar.cfamily.build-wrapper-output="$BUILD_WRAPPER_OUT_DIR" --define sonar.cfamily.gcov.reportsPath="_coverage" --define sonar.cfamily.threads="$PARALLEL"
# Consult https://docs.sonarcloud.io/advanced-setup/ci-based-analysis/sonarscanner-cli/ for more information and options
- name: Check out ABI standard
if: inputs.abi_check != ''
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
ref: ${{inputs.abi_check}}
path: abi_standard
- name: Build ABI standard
if: inputs.abi_check != ''
shell: bash
run: |
mkdir -p abi_standard/build
pushd abi_standard
src/build-scripts/ci-build.bash
popd
- name: Check ABI
if: inputs.abi_check != ''
shell: bash
run: |
src/build-scripts/ci-abicheck.bash ./build abi_standard/build libOpenImageIO libOpenImageIO_Util
- name: Build Docs
if: inputs.build_docs == '1'
shell: bash
run: |
cd src/doc
time make doxygen
time make sphinx
- uses: actions/upload-artifact@a8a3f3ad30e3422c9c7b888a15615d19a852ae32 # v3.1.3
# Note: versions of upload-artifact >= 4.0 can't run on the ASWF
# containers because they need a newer glibc than they have.
if: ${{ failure() || inputs.build_docs == '1'}}
with:
name: oiio-${{github.job}}-${{inputs.nametag}}
path: |
build/cmake-save
build/compat_reports
build/sphinx
build/testsuite/*/*.*
!build/testsuite/oiio-images
!build/testsuite/openexr-images
!build/testsuite/fits-images
!build/testsuite/j2kp4files_v1_5
${{ inputs.extra_artifacts }}
Loading

0 comments on commit 632717a

Please sign in to comment.