Skip to content
This repository was archived by the owner on Jan 30, 2023. It is now read-only.

Commit 4c023d5

Browse files
saraedumsaraedum
committed
Enable CI backed by GitLab CI and CircleCI
1 parent 93020a4 commit 4c023d5

24 files changed

+709
-4
lines changed

‎.ci/README.md

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Continuous Integration (CI)
2+
3+
We support several implementations of CI. All these implementations rely on
4+
[docker](https://docker.com) in some way. This directory contains bits which
5+
are shared between these CI implementations. The relevant docker files can be
6+
found in `/docker/`.
7+
8+
* [CircleCI](https://circleci.com) is configured in `/.circleci/`.
9+
* [GitLab CI](https://gitlab.com) is configured in `/.gitlab-ci.yml`.

‎.ci/build-docker.sh

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to build several flavours of docker images
4+
# which contain Sage.
5+
6+
# ****************************************************************************
7+
# Copyright (C) 2018 Julian Rüth <[email protected]>
8+
#
9+
# This program is free software: you can redistribute it and/or modify
10+
# it under the terms of the GNU General Public License as published by
11+
# the Free Software Foundation, either version 2 of the License, or
12+
# (at your option) any later version.
13+
# http://www.gnu.org/licenses/
14+
# ****************************************************************************
15+
set -ex
16+
17+
[[ -z "$DOCKER_TAG" ]] && DOCKER_TAG=none
18+
[[ "$DOCKER_TAG" = "master" ]] && DOCKER_TAG=latest
19+
20+
. .ci/setup-make-parallelity.sh
21+
22+
# We speed up the build process by copying built artifacts from ARTIFACT_BASE
23+
# during docker build. See /docker/Dockerfile for more details.
24+
ARTIFACT_BASE=${ARTIFACT_BASE:-sagemath/sagemath-dev:latest}
25+
26+
# Seed our cache with $ARTIFACT_BASE if it exists
27+
docker pull $ARTIFACT_BASE || true
28+
29+
function docker_build {
30+
time docker build -f docker/Dockerfile --build-arg "MAKE=${MAKE}" --build-arg ARTIFACT_BASE=$ARTIFACT_BASE $@
31+
}
32+
33+
# We use a multi-stage build /docker/Dockerfile. For the caching to be
34+
# effective, we populate the cache by building the make-all target. (Just
35+
# building the last target is not enough as intermediate targets would be
36+
# discarded from the cache and therefore the caching would fail for our actual
37+
# builds below.)
38+
docker_build --pull --tag make-all --target make-all .
39+
40+
# Build the release image without build artifacts.
41+
DOCKER_IMAGE_CLI=${DOCKER_USER:-sagemath}/sagemath:$DOCKER_TAG
42+
docker_build --target sagemath --tag "$DOCKER_IMAGE_CLI" .
43+
# Build the developer image with the build artifacts intact.
44+
# Note: It's important to build the dev image last because it might be tagged as ARTIFACT_BASE.
45+
DOCKER_IMAGE_DEV=${DOCKER_USER:-sagemath}/sagemath-dev:$DOCKER_TAG
46+
docker_build --target sagemath-dev --tag "$DOCKER_IMAGE_DEV" .

‎.ci/protect-secrets.sh

+40
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
#!/bin/bash
2+
3+
# This script protects all environment variables that start with "SECRET_".
4+
# It puts them in a temporary file. The name of the variable contains the path
5+
# of that file. This filename can then safely be used in `cat` even if `set
6+
# -x` has been turned on. Also you can run "export" to understand the
7+
# environment without danger.
8+
# Be careful, however, not to use this like the following:
9+
# docker login $DOCKER_USER $(cat $SECRET_DOCKER_PASS)
10+
# as this would expose the password if `set -x` has been turned on.
11+
12+
# ****************************************************************************
13+
# Copyright (C) 2018 Julian Rüth <[email protected]>
14+
#
15+
# This program is free software: you can redistribute it and/or modify
16+
# it under the terms of the GNU General Public License as published by
17+
# the Free Software Foundation, either version 2 of the License, or
18+
# (at your option) any later version.
19+
# http://www.gnu.org/licenses/
20+
# ****************************************************************************
21+
22+
set -eo pipefail
23+
set +x
24+
25+
function encrypt {
26+
RET=`mktemp`
27+
eval " echo \$$1" > "$RET"
28+
echo $RET
29+
}
30+
31+
for name in `awk 'END { for (name in ENVIRON) { print name; } }' < /dev/null`; do
32+
case "$name" in
33+
SECRET_*)
34+
export $name="$(encrypt $name)"
35+
echo "Protected $name"
36+
;;
37+
esac
38+
done
39+
40+
unset encrypt

‎.ci/pull-gitlab.sh

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to pull the Sage docker images that were
4+
# built during the "build" phase to pull all the connected docker daemon
5+
# (likely a docker-in-docker.)
6+
# This script expects a single parameter, the base name of the docker image
7+
# such as sagemath or sagemath-dev.
8+
# The variable $DOCKER_IMAGE is set to the full name of the pulled image;
9+
# source this script to use it.
10+
11+
# ****************************************************************************
12+
# Copyright (C) 2018 Julian Rüth <[email protected]>
13+
#
14+
# This program is free software: you can redistribute it and/or modify
15+
# it under the terms of the GNU General Public License as published by
16+
# the Free Software Foundation, either version 2 of the License, or
17+
# (at your option) any later version.
18+
# http://www.gnu.org/licenses/
19+
# ****************************************************************************
20+
21+
set -ex
22+
23+
[[ -z "$DOCKER_TAG" ]] && (echo "Can not pull untagged build."; exit 0)
24+
[[ "$DOCKER_TAG" = "master" ]] && DOCKER_TAG=latest
25+
26+
# Pull the built images from the gitlab registry and give them the original
27+
# names they had after built.
28+
# Note that "set -x" prints the $CI_BUILD_TOKEN here but GitLab removes it
29+
# automatically from the log output.
30+
docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
31+
docker pull $CI_REGISTRY_IMAGE/$1:$DOCKER_TAG
32+
DOCKER_IMAGE="${DOCKER_USER:-sagemath}/$1:$DOCKER_TAG"
33+
docker tag $CI_REGISTRY_IMAGE/$1:$DOCKER_TAG $DOCKER_IMAGE

‎.ci/push-dockerhub.sh

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to push our docker images to
4+
# $DOCKER_USER/sagemath* on the Docker Hub.
5+
# This script expects a single parameter, the base name of the docker image
6+
# such as sagemath or sagemath-dev.
7+
8+
# ****************************************************************************
9+
# Copyright (C) 2018 Julian Rüth <[email protected]>
10+
#
11+
# This program is free software: you can redistribute it and/or modify
12+
# it under the terms of the GNU General Public License as published by
13+
# the Free Software Foundation, either version 2 of the License, or
14+
# (at your option) any later version.
15+
# http://www.gnu.org/licenses/
16+
# ****************************************************************************
17+
18+
set -ex
19+
20+
[[ -z "$DOCKER_TAG" ]] && (echo "Can not push untagged build."; exit 0)
21+
[[ "$DOCKER_TAG" = "master" ]] && DOCKER_TAG=latest
22+
23+
# Push the built images to the docker hub (and fail silently if
24+
# DOCKER_USER/SECRET_DOCKER_PASS have not been configured.)
25+
if [[ -z "$DOCKER_USER" || -z "$SECRET_DOCKER_PASS" ]]; then
26+
echo "DOCKER_USER/SECRET_DOCKER_PASS variables have not been configured in your Continuous Integration setup. Not pushing built images to Docker Hub."
27+
else
28+
cat "$SECRET_DOCKER_PASS" | docker login -u $DOCKER_USER --password-stdin
29+
docker push ${DOCKER_USER:-sagemath}/$1:$DOCKER_TAG
30+
fi

‎.ci/push-gitlab.sh

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
set -ex
3+
4+
# This script gets called from CI to push our docker images to registry
5+
# configured in GitLab. (Mostly, so we can pull them again to push them to the
6+
# Docker Hub.)
7+
# This script expects a single parameter, the base name of the docker image
8+
# such as sagemath or sagemath-dev.
9+
10+
# ****************************************************************************
11+
# Copyright (C) 2018 Julian Rüth <[email protected]>
12+
#
13+
# This program is free software: you can redistribute it and/or modify
14+
# it under the terms of the GNU General Public License as published by
15+
# the Free Software Foundation, either version 2 of the License, or
16+
# (at your option) any later version.
17+
# http://www.gnu.org/licenses/
18+
# ****************************************************************************
19+
20+
[[ -z "$DOCKER_TAG" ]] && (echo "Can not push untagged build."; exit 0)
21+
[[ "$DOCKER_TAG" = "master" ]] && DOCKER_TAG=latest
22+
23+
# Note that "set -x" prints the $CI_BUILD_TOKEN here but GitLab removes it
24+
# automatically from the log output.
25+
docker login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
26+
docker tag ${DOCKER_USER:-sagemath}/$1:$DOCKER_TAG $CI_REGISTRY_IMAGE/$1:$DOCKER_TAG
27+
docker push $CI_REGISTRY_IMAGE/$1:$DOCKER_TAG

‎.ci/setup-make-parallelity.sh

+26
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
#!/bin/bash
2+
3+
# ****************************************************************************
4+
# Copyright (C) 2018 Julian Rüth <[email protected]>
5+
#
6+
# This program is free software: you can redistribute it and/or modify
7+
# it under the terms of the GNU General Public License as published by
8+
# the Free Software Foundation, either version 2 of the License, or
9+
# (at your option) any later version.
10+
# http://www.gnu.org/licenses/
11+
# ****************************************************************************
12+
set -ex
13+
14+
# Determine the number of threads that can run simultaneously on this system
15+
# (we might not have nproc available.)
16+
# Note that this value is incorrect for some CI providers (notably CircleCI:
17+
# https://circleci.com/docs/2.0/configuration-reference/#resource_class) which
18+
# provision fewer vCPUs than shown in /proc/cpuinfo. Also, setting this value
19+
# too high can lead to RAM being insufficient, so it's best to set this
20+
# variable manually in your CI configuration.
21+
[[ -z "$NTHREADS" ]] && NTHREADS=`grep -E '^processor' /proc/cpuinfo | wc -l` || true
22+
# Set -j and -l for make (though -l is probably stripped by Sage)
23+
[[ -z "$MAKEOPTS" ]] && MAKEOPTS="-j $NTHREADS -l $((NTHREADS-1)).8" || true
24+
# Not all parts of Sage seem to honor MAKEOPTS, so the current way of telling
25+
# the system which concurrency to use, seems to be setting $MAKE.
26+
[[ -z "$MAKE" ]] && MAKE="make $MAKEOPTS" || true

‎.ci/test-cli.sh

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to run minimal tests on the sagemath image.
4+
5+
# Usage: ./test-cli.sh sage-cli-image
6+
7+
# ****************************************************************************
8+
# Copyright (C) 2018 Julian Rüth <[email protected]>
9+
#
10+
# This program is free software: you can redistribute it and/or modify
11+
# it under the terms of the GNU General Public License as published by
12+
# the Free Software Foundation, either version 2 of the License, or
13+
# (at your option) any later version.
14+
# http://www.gnu.org/licenses/
15+
# ****************************************************************************
16+
set -exo pipefail
17+
18+
echo "Checking that Sage starts and can calculate 1+1…"
19+
# Calculate 1+1 (remove startup messages and leading & trailing whitespace)
20+
TWO=`docker run "$1" sage -c "'print(1+1)'" | tail -1 | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'`
21+
[[ "x$TWO" = "x2" ]]

‎.ci/test-dev.sh

+35
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to run minimal tests on the sagemath-dev image.
4+
# This script expects a single argument, the full name of the docker image to
5+
# test.
6+
7+
# ****************************************************************************
8+
# Copyright (C) 2018 Julian Rüth <[email protected]>
9+
#
10+
# This program is free software: you can redistribute it and/or modify
11+
# it under the terms of the GNU General Public License as published by
12+
# the Free Software Foundation, either version 2 of the License, or
13+
# (at your option) any later version.
14+
# http://www.gnu.org/licenses/
15+
# ****************************************************************************
16+
set -exo pipefail
17+
18+
IMAGE="$1"
19+
20+
. .ci/setup-make-parallelity.sh
21+
22+
# Usage: timed_run limit args
23+
# Runs $IMAGE with args and check that it terminates with a zero exit code in at most limit seconds.
24+
function timed_run {
25+
START=`date +%s`
26+
docker run -e MAKE="$MAKE" "$IMAGE" "$2"
27+
END=`date +%s`
28+
TOTAL=$((END-START))
29+
echo "Checking that \"$2\" was fast…"
30+
[[ $TOTAL -lt $1 ]]
31+
}
32+
33+
timed_run 60 true # runs make build
34+
# TODO: Can't we get this faster than that?
35+
timed_run 300 make # runs make build and then make

‎.ci/test-jupyter.sh

+23
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
#!/bin/bash
2+
3+
# This script gets called from CI to run minimal tests on the sagemath-jupyter
4+
# image.
5+
6+
# Usage: ./test-jupyter.sh sage-jupyter-image [host]
7+
8+
# ****************************************************************************
9+
# Copyright (C) 2018 Julian Rüth <[email protected]>
10+
#
11+
# This program is free software: you can redistribute it and/or modify
12+
# it under the terms of the GNU General Public License as published by
13+
# the Free Software Foundation, either version 2 of the License, or
14+
# (at your option) any later version.
15+
# http://www.gnu.org/licenses/
16+
# ****************************************************************************
17+
set -ex
18+
19+
docker run --name sage-jupyter -p 8888:8888 -d "$1" "sage -n jupyter --no-browser --ip='*' --port=8888"
20+
echo "Checking that the Jupyter notebook is running…"
21+
sleep 10 # giving the server some time to start
22+
docker logs sage-jupyter
23+
wget --retry-connrefused --tries=10 --wait=3 "http://${2:-localhost}:8888"

‎.circleci/config.yml

+41
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# This file configures automatic builds of Sage on [CircleCI](https://circleci.com).
2+
# To make the build time not too excessive, we seed the build cache with
3+
# sagemath/sagemath-dev:latest. When basic SPKGs change, this does not help much
4+
# and the full build might exceed CircleCI's limits for open source projcets (five
5+
# hours on 2 vCPUs as of early 2018.)
6+
# You might want to try to build locally or with GitLab CI, see
7+
# `.gitlab-ci.yml` for more options.
8+
# Note that we do not use workflows because it was not clear whether the docker
9+
# images sizes would not exceed the size limits of CircleCI workspaces. Also,
10+
# workflows would not make things faster. We do not get more machines for free
11+
# and the frequent loading of docker images probably exceeds the cost of the
12+
# actual tests.
13+
14+
version: 2
15+
jobs:
16+
# As https://circleci.com/docs/2.0/docker-layer-caching/ is a paid feature,
17+
# we do build & test & release in one step.
18+
build:
19+
machine: true
20+
steps:
21+
- checkout
22+
- run:
23+
# The docker commands sometimes take a while to produce output
24+
no_output_timeout: 30m
25+
command: |
26+
# CircleCI has no mechanism to hide secret variables.
27+
# Therefore we roll our own to protect $SECRET_* variables.
28+
. .ci/protect-secrets.sh
29+
30+
export DOCKER_TAG=${CIRCLE_TAG:-$CIRCLE_BRANCH}
31+
# Build docker images
32+
# TODO: Change this line to sagemath/sagemath-dev:latest
33+
export ARTIFACT_BASE=saraedum/sagemath-dev:gitlabci
34+
. .ci/build-docker.sh
35+
# Test that the images work
36+
. .ci/test-dev.sh $DOCKER_IMAGE_DEV
37+
. .ci/test-cli.sh $DOCKER_IMAGE_CLI
38+
. .ci/test-jupyter.sh $DOCKER_IMAGE_CLI localhost
39+
# Push docker images to dockerhub if a dockerhub user has been configured
40+
. .ci/push-dockerhub.sh sagemath-dev
41+
. .ci/push-dockerhub.sh sagemath

‎.dockerignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.gitignore

‎.gitignore

+6
Original file line numberDiff line numberDiff line change
@@ -81,3 +81,9 @@ $RECYCLE.BIN/
8181
###########
8282
.ipynb_checkpoints
8383
Untitled*.ipynb
84+
85+
#############################
86+
# GitLab CI generated files #
87+
#############################
88+
gitlab-build-docker.log
89+

0 commit comments

Comments
 (0)