Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

On-Board check-catalyst workflow to use GitHub Large Runners #675

Merged
merged 23 commits into from
Apr 25, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c219290
Try large runners
rashidnhm Apr 23, 2024
319a71b
Move re-usable action to .github/workflows/
rashidnhm Apr 23, 2024
6d726f9
Ensure job is never skipped
rashidnhm Apr 23, 2024
c51cef4
Bump actions version. Swap `True` values to `true`
rashidnhm Apr 23, 2024
4b2563a
Change logic on what type of runner is used for workflow
rashidnhm Apr 23, 2024
ed2f5e4
Fix bug
rashidnhm Apr 23, 2024
ac0e0f1
Add startsWith constraint
rashidnhm Apr 24, 2024
14deb2f
Try actions/cache boolean
rashidnhm Apr 24, 2024
72d4051
Try downgrade actions/cache
rashidnhm Apr 24, 2024
6070595
Remove testing steps
rashidnhm Apr 24, 2024
77299b5
Merge branch 'main' into sc-61791-ci-use-large-runners
rashidnhm Apr 24, 2024
80b5180
On-board build-wheel-linux-x86_64 to use large runners
rashidnhm Apr 24, 2024
d168650
Merge branch 'sc-61791-ci-use-large-runners' of github.com:PennyLaneA…
rashidnhm Apr 24, 2024
702552a
Trigger CI
rashidnhm Apr 24, 2024
4074c33
Fix incorrect strategy matrix
rashidnhm Apr 24, 2024
412439d
Bump action versions across the board. Fix typo cmake call
rashidnhm Apr 24, 2024
5c2b7fb
Merge branch 'main' of github.com:PennyLaneAI/catalyst into sc-61791-…
rashidnhm Apr 24, 2024
7f36b5f
Merge branch 'main' of github.com:PennyLaneAI/catalyst into sc-61791-…
rashidnhm Apr 25, 2024
69a3193
Bump arm64 workflow to use upload-artifact@v4
rashidnhm Apr 25, 2024
6cd34e9
Downgrade manylinux build to checkout@v3
rashidnhm Apr 25, 2024
6b0a3fa
Merge branch 'main' of github.com:PennyLaneAI/catalyst into sc-61791-…
rashidnhm Apr 25, 2024
591ded0
Change other @v4 to v3 for actions/checkout
rashidnhm Apr 25, 2024
670798b
Downgrade cache@v4 to v3 for manylinux
rashidnhm Apr 25, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
68 changes: 38 additions & 30 deletions .github/workflows/build-wheel-linux-x86_64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -16,17 +16,25 @@ concurrency:
cancel-in-progress: true

jobs:
determine_runner:
name: Determine runner type to use
uses: ./.github/workflows/determine-workflow-runner.yml
with:
default_runner: ubuntu-latest

check_if_wheel_build_required:
uses: ./.github/workflows/check-for-wheel-build.yml

constants:
needs: [check_if_wheel_build_required]
needs: [check_if_wheel_build_required, determine_runner]
if: needs.check_if_wheel_build_required.outputs.build-wheels == 'true'
name: "Set build matrix"
uses: ./.github/workflows/constants.yaml
with:
runs_on: ${{ needs.determine_runner.outputs.runner_group }}

build-dependencies:
needs: [constants, check_if_wheel_build_required]
needs: [constants, check_if_wheel_build_required, determine_runner]

strategy:
fail-fast: false
Expand All @@ -37,14 +45,14 @@ jobs:
container_img: ["quay.io/pypa/manylinux2014_x86_64"]

name: Build Dependencies (Python ${{ matrix.python_version }})
runs-on: ubuntu-latest
runs-on: ${{ needs.determine_runner.outputs.runner_group }}
container: ${{ matrix.container_img }}

if: needs.check_if_wheel_build_required.outputs.build-wheels == 'true'

steps:
- name: Checkout Catalyst repo
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Set Ownership in Container
run: |
Expand All @@ -56,47 +64,47 @@ jobs:
# GH doesn't support using caches across un-/containerized actions (those with `container_img`).
- name: Cache LLVM Source
id: cache-llvm-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/llvm-project
key: ${{ runner.os }}-llvm-${{ needs.constants.outputs.llvm_version }}-generic-source
enableCrossOsArchive: True

- name: Cache MHLO Source
id: cache-mhlo-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/mlir-hlo
key: ${{ runner.os }}-mhlo-${{ needs.constants.outputs.mhlo_version }}-generic-source
enableCrossOsArchive: True

- name: Cache Enzyme Source
id: cache-enzyme-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/Enzyme
key: ${{ runner.os }}-enzyme-${{ needs.constants.outputs.enzyme_version }}-generic-source
enableCrossOsArchive: True

- name: Clone LLVM Submodule
if: steps.cache-llvm-source.outputs.cache-hit != 'true'
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: llvm/llvm-project
ref: ${{ needs.constants.outputs.llvm_version }}
path: mlir/llvm-project

- name: Clone MHLO Submodule
if: steps.cache-mhlo-source.outputs.cache-hit != 'true'
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: tensorflow/mlir-hlo
ref: ${{ needs.constants.outputs.mhlo_version }}
path: mlir/mlir-hlo

- name: Clone Enzyme Submodule
if: steps.cache-enzyme-source.outputs.cache-hit != 'true'
uses: actions/checkout@v3
uses: actions/checkout@v4
with:
repository: EnzymeAD/Enzyme
ref: ${{ needs.constants.outputs.enzyme_version }}
Expand All @@ -105,21 +113,21 @@ jobs:
# Cache external project builds
- name: Cache LLVM Build
id: cache-llvm-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: llvm-build
key: ${{ runner.os }}-llvm-${{ needs.constants.outputs.llvm_version }}-${{matrix.python_version}}-generic-build

- name: Cache MHLO Build
id: cache-mhlo-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mhlo-build
key: ${{ runner.os }}-mhlo-${{ needs.constants.outputs.mhlo_version }}-generic-build

- name: Cache Enzyme Build
id: cache-enzyme-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: enzyme-build
key: ${{ runner.os }}-enzyme-${{ needs.constants.outputs.llvm_version }}-${{ needs.constants.outputs.enzyme_version }}-generic-build
Expand Down Expand Up @@ -206,13 +214,13 @@ jobs:
cmake --build enzyme-build --target EnzymeStatic-18

catalyst-linux-wheels-x86-64:
needs: [constants, build-dependencies]
needs: [constants, build-dependencies, determine_runner]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest]
os: ${{ fromJSON(format('[{0}]', needs.determine_runner.outputs.runner_group)) }}
arch: [x86_64]
python_version: ${{ fromJson(needs.constants.outputs.python_versions) }}
python_version: ${{ fromJSON(needs.constants.outputs.python_versions) }}
container_img: ["quay.io/pypa/manylinux2014_x86_64"]

name: Build Wheels (Python ${{ matrix.python_version }})
Expand All @@ -221,7 +229,7 @@ jobs:

steps:
- name: Checkout Catalyst repo
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Install dependencies (CentOS)
run: |
Expand All @@ -235,7 +243,7 @@ jobs:

- name: Get Cached LLVM Source
id: cache-llvm-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/llvm-project
key: ${{ runner.os }}-llvm-${{ needs.constants.outputs.llvm_version }}-generic-source
Expand All @@ -244,15 +252,15 @@ jobs:

- name: Get Cached LLVM Build
id: cache-llvm-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: llvm-build
key: ${{ runner.os }}-llvm-${{ needs.constants.outputs.llvm_version }}-${{matrix.python_version}}-generic-build
fail-on-cache-miss: True

- name: Get Cached MHLO Source
id: cache-mhlo-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/mlir-hlo
key: ${{ runner.os }}-mhlo-${{ needs.constants.outputs.mhlo_version }}-generic-source
Expand All @@ -261,23 +269,23 @@ jobs:

- name: Get Cached MHLO Build
id: cache-mhlo-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mhlo-build
key: ${{ runner.os }}-mhlo-${{ needs.constants.outputs.mhlo_version }}-generic-build
fail-on-cache-miss: True

- name: Get Cached Enzyme Source
id: cache-enzyme-source
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: mlir/Enzyme
key: ${{ runner.os }}-enzyme-${{ needs.constants.outputs.enzyme_version }}-generic-source
fail-on-cache-miss: True

- name: Get Cached Enzyme Build
id: cache-enzyme-build
uses: actions/cache@v3
uses: actions/cache@v4
with:
path: enzyme-build
key: ${{ runner.os }}-enzyme-${{ needs.constants.outputs.llvm_version }}-${{ needs.constants.outputs.enzyme_version }}-generic-build
Expand Down Expand Up @@ -349,36 +357,36 @@ jobs:
auditwheel repair dist/*.whl -w ./wheel --no-update-tags --exclude libopenblasp-r0-23e5df77.3.21.dev.so

- name: Upload Wheel Artifact
uses: actions/upload-artifact@v3
uses: actions/upload-artifact@v4
with:
name: catalyst-manylinux2014_x86_64-wheel-py-${{ matrix.python_version }}.zip
path: wheel/
retention-days: 14

test-wheels:
needs: [constants, catalyst-linux-wheels-x86-64]
needs: [constants, catalyst-linux-wheels-x86-64, determine_runner]
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, ubuntu-20.04]
python_version: ${{ fromJson(needs.constants.outputs.python_versions) }}
os: ${{ fromJSON(format('[{0}]', needs.determine_runner.outputs.runner_group)) }}
python_version: ${{ fromJSON(needs.constants.outputs.python_versions) }}

# To check all wheels for supported python3 versions
name: Test Wheels (Python ${{ matrix.python_version }}) on ${{ matrix.os }}
runs-on: ${{ matrix.os }}

steps:
- name: Checkout Catalyst repo
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Download Wheel Artifact
uses: actions/download-artifact@v3
uses: actions/download-artifact@v4
with:
name: catalyst-manylinux2014_x86_64-wheel-py-${{ matrix.python_version }}.zip
path: dist

- name: Set up Python ${{ matrix.python_version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python_version }}

Expand Down
Loading
Loading