Skip to content

Commit

Permalink
Trac #29143: tox.ini update: Fix breakage caused by src/ext move, add…
Browse files Browse the repository at this point in the history 
… 32-bit linux environments and other architectures supported by docker

#29053 added `tox.ini` to support testing the sage distribution in
isolated environments via docker.

In 9.1.beta6, the generated docker files do not work due to #21785.

This ticket fixes it and also adds linux environments for various other
architectures supported by docker (see https://github.com/docker-library
/official-images#architectures-other-than-amd64).

Examples:
{{{
$ tox -e docker-debian-buster-standard-i386
$ tox -e docker-ubuntu-bionic-standard-arm64
$ tox -e docker-ubuntu-bionic-minimal-armhf
$ tox -e docker-fedora-31-standard-arm64v8         # Docker for Mac only
$ tox -e docker-ubuntu-latest-standard-s390x       # Docker for Mac only
}}}

(The last two work with macOS Docker Desktop because it has
`binfmt_misc` with `qemu` enabled.)

See also: https://trac.sagemath.org/wiki/ExoticPorts

---------

Resources:
- How to Build and Run ARM Docker Containers on x86 Hosts · MatchboxBlog
- https://matchboxdorry.gitbooks.io/matchboxblog/content/blogs/build_and
_run_arm_images.html
- Cross Building and Running Multi-Arch Docker Images – Micheal Waltz –
DevOps Engineer - https://www.ecliptik.com/Cross-Building-and-Running-
Multi-Arch-Docker-Images/
- multiarch/qemu-user-static - Docker Hub -
https://hub.docker.com/r/multiarch/qemu-user-static/
- multiarch/ubuntu-core - Docker Hub -
https://hub.docker.com/r/multiarch/ubuntu-core/

URL: https://trac.sagemath.org/29143
Reported by: mkoeppe
Ticket author(s): Matthias Koeppe
Reviewer(s): Volker Braun
  • Loading branch information
Release Manager committed Mar 7, 2020
2 parents abcb40a + 7f122bd commit b85b793
Show file tree
Hide file tree
Showing 3 changed files with 116 additions and 36 deletions.
5 changes: 5 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,8 @@ gitlab-build-docker.log
/src/bin/sage-env-config
/build/bin/sage-build-env-config

#######################
# tox generated files #
#######################

/.tox
1 change: 0 additions & 1 deletion build/bin/write-dockerfile.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,6 @@ ADD build ./build
ADD src/bin/sage-version.sh src/bin/sage-version.sh
$RUN ./bootstrap
#:configuring:
ADD src/ext src/ext
ADD src/bin src/bin
ADD src/Makefile.in src/Makefile.in
ARG EXTRA_CONFIGURE_ARGS=""
Expand Down
146 changes: 111 additions & 35 deletions tox.ini
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,26 @@ envlist =
centos-{7,8},
arch-latest,
conda-forge,conda-anaconda3
}-{minimal,standard,maximal}
}
-{# https://github.com/docker-library/official-images#architectures-other-than-amd64
# architectures officially supported by Docker, Inc. for running Docker
amd64,arm32v6,arm32v7,arm64v8,
# windows-amd64 # https://hub.docker.com/u/winamd64/
# Other architectures built by official images:
# (but not officially supported by Docker, Inc.)
arm32v5,ppc64le,s390x,i386}
### "local" targets should be run from a source tree that is freshly checked out
### (for example, by 'git worktree add ...') or has been cleaned by 'make bdist-clean' --
### because they build within the source tree (because we have no VPATH support).
### Only one "local" target can be run at a time.
### However, "local" targets install in a separate prefix (SAGE_LOCAL=.tox/TOXENV/local)
### rather than "local/" and also place log files into .tox/TOXENV/log, where TOXENV
### is the name of the environment.
-{minimal,standard,maximal},
### The "local-direct" toxenv passes the whole environment on to the sage build.
### Whatever is in PATH etc. will be used.
local-direct

skipsdist = true

[testenv]
Expand All @@ -38,73 +57,130 @@ setenv =
debian: EXTRA_CONFIGURE_ARGS=--with-system-m4ri=no --with-system-m4rie=no --with-system-givaro=no
fedora: EXTRA_CONFIGURE_ARGS=--with-system-m4ri=no --with-system-m4rie=no
#
# default tag is "latest"
#
docker: BASE_TAG=latest
#
# https://hub.docker.com/_/ubuntu?tab=description
# as of 2020-01, latest=bionic=18.04, eoan=rolling=19.10, focal=devel=20.04
#
ubuntu: SYSTEM=debian
ubuntu-trusty: BASE_IMAGE=ubuntu:trusty
ubuntu: BASE_IMAGE=ubuntu
ubuntu-trusty: BASE_TAG=trusty
ubuntu-trusty: IGNORE_MISSING_SYSTEM_PACKAGES=yes
ubuntu-xenial: BASE_IMAGE=ubuntu:xenial
ubuntu-xenial: BASE_TAG=xenial
ubuntu-xenial: IGNORE_MISSING_SYSTEM_PACKAGES=yes
ubuntu-bionic: BASE_IMAGE=ubuntu:bionic
ubuntu-latest: BASE_IMAGE=ubuntu:latest
ubuntu-eoan: BASE_IMAGE=ubuntu:eoan
ubuntu-rolling: BASE_IMAGE=ubuntu:rolling
ubuntu-focal: BASE_IMAGE=ubuntu:focal
ubuntu-devel: BASE_IMAGE=ubuntu:devel
ubuntu-bionic: BASE_TAG=bionic
ubuntu-latest: BASE_TAG=latest
ubuntu-eoan: BASE_TAG=eoan
ubuntu-rolling: BASE_TAG=rolling
ubuntu-focal: BASE_TAG=focal
ubuntu-devel: BASE_TAG=devel
#
# https://hub.docker.com/_/debian
#
debian: SYSTEM=debian
debian-jessie: BASE_IMAGE=debian:jessie
debian-jessie: IGNORE_MISSING_SYSTEM_PACKAGES=yes
debian-stretch: BASE_IMAGE=debian:stretch
debian-buster: BASE_IMAGE=debian:buster
debian-bullseye: BASE_IMAGE=debian:bullseye
debian-sid: BASE_IMAGE=debian:sid
debian: SYSTEM=debian
debian: BASE_IMAGE=debian
debian-jessie: BASE_TAG=jessie
debian-jessie: IGNORE_MISSING_SYSTEM_PACKAGES=yes
debian-stretch: BASE_TAG=stretch
debian-buster: BASE_TAG=buster
debian-bullseye: BASE_TAG=bullseye
debian-sid: BASE_TAG=sid
#
# https://hub.docker.com/_/fedora
#
fedora: SYSTEM=fedora
fedora-26: BASE_IMAGE=fedora:26
fedora-26: IGNORE_MISSING_SYSTEM_PACKAGES=yes
fedora-27: BASE_IMAGE=fedora:27
fedora-28: BASE_IMAGE=fedora:28
fedora-29: BASE_IMAGE=fedora:29
fedora-30: BASE_IMAGE=fedora:30
fedora-31: BASE_IMAGE=fedora:31 # latest
fedora-32: BASE_IMAGE=fedora:32 # rawhide
fedora: SYSTEM=fedora
fedora: BASE_IMAGE=fedora
fedora-26: BASE_TAG=26
fedora-26: IGNORE_MISSING_SYSTEM_PACKAGES=yes
fedora-27: BASE_TAG=27
fedora-28: BASE_TAG=28
fedora-29: BASE_TAG=29
fedora-30: BASE_TAG=30
fedora-31: BASE_TAG=31 # latest
fedora-32: BASE_TAG=32 # rawhide
#
# https://hub.docker.com/_/centos
#
centos: SYSTEM=fedora
centos: IGNORE_MISSING_SYSTEM_PACKAGES=yes
#centos-6: BASE_IMAGE=centos:centos6 # only has autoconf 2.63 -- too old for bootstrap
centos-7: BASE_IMAGE=centos:centos7
centos-8: BASE_IMAGE=centos:centos8
centos: SYSTEM=fedora
centos: BASE_IMAGE=centos
centos: IGNORE_MISSING_SYSTEM_PACKAGES=yes
#centos-6: BASE_TAG=centos6 # only has autoconf 2.63 -- too old for bootstrap
centos-7: BASE_TAG=centos7
centos-8: BASE_TAG=centos8
#
# https://hub.docker.com/_/archlinux/
#
archlinux: SYSTEM=arch
archlinux-latest: BASE_IMAGE=archlinux:latest
archlinux: SYSTEM=arch
archlinux: BASE_IMAGE=archlinux
#
# https://hub.docker.com/r/continuumio
#
conda: SYSTEM=conda
conda-forge: BASE_IMAGE=continuumio/miniconda3:latest
conda-forge: BASE_IMAGE=continuumio/miniconda3
conda-forge: CONDARC=condarc.yml
conda-anaconda3: BASE_IMAGE=continuumio/anaconda3:latest
conda-anaconda3: BASE_IMAGE=continuumio/anaconda3
conda-anaconda3: CONDARC=/dev/null
conda-anaconda3: IGNORE_MISSING_SYSTEM_PACKAGES=yes
#
# Other architectures:
#
# Many docker images for another architecture are named the same, in the arch prefix.
# All work for Docker on Mac; but only i386 works for Linux Docker.
#
arm32v5: ARCH_IMAGE_PREFIX=arm32v5/
arm32v6: ARCH_IMAGE_PREFIX=arm32v6/
arm32v7: ARCH_IMAGE_PREFIX=arm32v7/
arm64v8: ARCH_IMAGE_PREFIX=arm64v8/
i386: ARCH_IMAGE_PREFIX=i386/
ppc64le: ARCH_IMAGE_PREFIX=ppc64le/
s390x: ARCH_IMAGE_PREFIX=s390x/
#
# For Linux Docker, we need to work with known multiarch images.
# https://www.ecliptik.com/Cross-Building-and-Running-Multi-Arch-Docker-Images/
# (or we would need to change our tests from "docker build" to "docker run")
#
# https://hub.docker.com/r/multiarch/ubuntu-core:
# multiarch/ubuntu-core:arm64-bionic, multiarch/ubuntu-core:armhf-bionic
ubuntu-arm64: BASE_IMAGE=ubuntu-core
ubuntu-arm64: ARCH_IMAGE_PREFIX=multiarch/
ubuntu-arm64: ARCH_TAG_PREFIX=arm64-
ubuntu-armhf: BASE_IMAGE=ubuntu-core
ubuntu-armhf: ARCH_IMAGE_PREFIX=multiarch/
ubuntu-armhf: ARCH_TAG_PREFIX=armhf-
#
# https://hub.docker.com/u/raspbian
#
raspbian: SYSTEM=debian
raspbian: ARCH_IMAGE_PREFIX=
raspbian-jessie: BASE_IMAGE=raspbian/jessie
raspbian-stretch: BASE_IMAGE=raspbian/stretch
#
# Resulting full image:tag name
#
docker: FULL_BASE_IMAGE_AND_TAG={env:ARCH_IMAGE_PREFIX:}{env:BASE_IMAGE}:{env:ARCH_TAG_PREFIX:}{env:BASE_TAG}
###
### "local" envs
###
local: HOME={envdir}
local-!direct: PATH=/usr/bin:/bin:/usr/sbin:/sbin

# environment will be skipped if regular expression does not match against the sys.platform string
platform =
local-macos: darwin

whitelist_externals =
bash
docker
commands =
docker: bash -c 'build/bin/write-dockerfile.sh {env:SYSTEM} "{env:TYPE_PATTERN:}" {env:WITH_SYSTEM_SPKG} {env:IGNORE_MISSING_SYSTEM_PACKAGES} > {envdir}/Dockerfile'
# From https://hub.docker.com/r/multiarch/ubuntu-core/
# configure binfmt-support on the Docker host (works locally or remotely, i.e: using boot2docker)
docker-{arm64,armhf}: docker run --rm --privileged multiarch/qemu-user-static:register --reset
docker: docker build . -f {envdir}/Dockerfile \
docker: --build-arg EXTRA_CONFIGURE_ARGS="{env:EXTRA_CONFIGURE_ARGS:}" \
docker: --build-arg BASE_IMAGE={env:BASE_IMAGE} \
docker: --build-arg BASE_IMAGE={env:FULL_BASE_IMAGE_AND_TAG} \
docker-conda: --build-arg USE_CONDARC="{env:CONDARC}" \
docker: {env:EXTRA_DOCKER_BUILD_ARGS:}

Expand Down

0 comments on commit b85b793

Please sign in to comment.