Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP]CI: cache dependencies build #266

Closed
wants to merge 19 commits into from
Closed
61 changes: 56 additions & 5 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,23 @@ concurrency:
cancel-in-progress: true

jobs:
build_ubuntu:
name: Build [ubuntu]
build_dependencies:
name: Build deps [ubuntu]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Get dependencies hash
id: get-hash
run: echo "deps_hash=`cat blazingmq/docker/build_deps.sh | shasum`" >> $GITHUB_OUTPUT
- name: Cache lookup
uses: actions/cache/restore@v4
id: cache-lookup
with:
path: deps
key: deps-${{ steps.get-hash.outputs.deps_hash }}
lookup-only: true
- name: Set up dependencies
if: steps.cache-lookup.outputs.cache-hit != 'true'
run: |
sudo apt-get update
sudo apt-get install -qy build-essential \
Expand All @@ -34,11 +45,51 @@ jobs:
libbenchmark-dev \
libgmock-dev \
libz-dev
- name: Create dependency fetcher working directory
run: mkdir -p deps
- name: Fetch & Build non packaged dependencies
if: steps.cache-lookup.outputs.cache-hit != 'true'
run: |
mkdir -p deps
cd deps
../docker/build_deps.sh
- name: Cache save
if: steps.cache-lookup.outputs.cache-hit != 'true'
uses: actions/cache/save@v4
with:
path: deps
key: deps-${{ steps.get-hash.outputs.deps_hash }}

build_ubuntu:
name: Build [ubuntu]
runs-on: ubuntu-latest
needs: build_dependencies
steps:
- uses: actions/checkout@v4
- name: Get dependencies hash
id: get-hash
run: echo "deps_hash=`cat blazingmq/docker/build_deps.sh | shasum`" >> $GITHUB_OUTPUT
- uses: actions/cache@v4
with:
path: deps
key: deps-${{ steps.get-hash.outputs.deps_hash }}
- name: Install cached non packaged dependencies
working-directory: deps
run: ../docker/build_deps.sh
run: ../docker/build_deps.sh --install-only
- name: Set up dependencies
run: |
sudo apt-get update
sudo apt-get install -qy build-essential \
gdb \
curl \
python3.10 \
python3-pip \
cmake \
ninja-build \
pkg-config \
bison \
libfl-dev \
libbenchmark-dev \
libgmock-dev \
libz-dev
- name: Build BlazingMQ
env:
PKG_CONFIG_PATH: /usr/lib/x86_64-linux-gnu/pkgconfig:/opt/bb/lib64/pkgconfig
Expand Down
22 changes: 17 additions & 5 deletions docker/build_deps.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,11 @@

set -euxo pipefail

install_only=false
if [[ -n "${1-}" && "${1-}" = "--install-only" ]]; then
install_only=true
fi

fetch_git() {
local org=$1
local repo=$2
Expand All @@ -24,6 +29,9 @@ fetch_git() {
}

fetch_deps() {
if [ "$install_only" = true ]; then
return 0
fi
fetch_git bloomberg bde-tools 4.8.0.0
fetch_git bloomberg bde 4.8.0.0
fetch_git bloomberg ntf-core latest
Expand All @@ -37,17 +45,21 @@ configure() {

build_bde() {
pushd srcs/bde
bbs_build configure
bbs_build build -j8
if [ "$install_only" = false ]; then
bbs_build configure
bbs_build build -j8
fi
bbs_build --install=/opt/bb --prefix=/ install
popd
}

build_ntf() {
pushd srcs/ntf-core
sed -i s/CMakeLists.txt//g ./configure
./configure --prefix /opt/bb --without-usage-examples --without-applications --ufid opt_64_cpp17
make -j8
if [ "$install_only" = false ]; then
sed -i s/CMakeLists.txt//g ./configure
./configure --prefix /opt/bb --without-usage-examples --without-applications --ufid opt_64_cpp17
make -j8
fi
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't 100% like how obscure all these moments to get the script install only.
There is also an issue with ntf library being rebuild from scratch each time we call this script, even if build files already exist. And this rebuild takes 5 minutes.

make install
popd
}
Expand Down
Loading