Build Fails on ARM Docker Image within AWS CodeBuild

[bryang-spindance]

Brief

When attempting to install awscrt within a Docker container in an amazonlinux2-aarch64-standard:2.0 Codebuild project, I am receiving ARM based errors. This happens both when I try to install via pre-built pip packages pip install awscrt and when I try to build awscrt from source.

Environment

I am using an AWS Codebuild project with the aws/codebuild/amazonlinux2-aarch64-standard:2.0 environment image. When this Codebuild project runs, it builds a Docker container which will attempt to install the awscrt package. The docker container is using the python:3.9-slim-buster image as a base.

Dockerfile

FROM --platform=linux/arm/v7 public.ecr.aws/docker/library/python:3.9-slim-buster

RUN apt-get update && apt-get -y install git cmake libffi-dev libssl-dev build-essential
RUN git clone https://github.com/awslabs/aws-crt-python.git && cd ./aws-crt-python && git submodule update --init && python3 -m pip install . && cd ..

buildspec.yaml

version 0.2

phases:
  install:
    runtime-versions:
      python: 3.9
  build:
    commands:
      - docker build --platform=linux/arm/v7 -t my-docker-tag .

Problem

Please view the Codebuild awscrt Install Log for the full log of the awscrt installation process. The resulting errors are the same whether I attempt to install awscrt from pip or if I attempt to build awscrt from source.

The errors are all similar to the following block. I wanted to add some of these in here so people can search for this Github issue since the full logs are attached as a gist.

/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:49: Error: bad instruction `ldp x5,x6,[x1]'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:50: Error: bad instruction `ldp x4,x3,[x2]'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:51: Error: ARM register expected -- `adds x5,x5,x4'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:52: Error: ARM register expected -- `adcs x6,x6,x3'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:53: Error: bad instruction `ldp x7,x8,[x1,#16]'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:54: Error: bad instruction `ldp x4,x3,[x2,#16]'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:55: Error: ARM register expected -- `adcs x7,x7,x4'
/aws-crt-python/crt/aws-lc/third_party/s2n-bignum/arm/p384/bignum_add_p384.S:56: Error: ARM register expected -- `adcs x8,x8,x3'

Additional Info

I have also tested installing awscrt from within the Codebuild image itself (outside of the Docker image) and I am able to install it using pip without error so this error is only occurring within my Docker image.

A workaround for this issue is to switch my Codebuild project to use the aws/codebuild/standard:5.0 environment image. Then within my buildspec.yaml file, I add the following command to my install phase:

docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

By doing this, I am able to install awscrt within my Docker image by simply using pip.

This workaround is not preferable, however, as I'd like to remove the multiarch/qemu-user-static dependency from this project by utilizing the aarch64 Codebuild environment image.

[moimart1]

Hi !

Same issue here, do you found a better solution @bryang-spindance ?

[oschwartz10612]

I am also having the same issue using buildx trying to build to armv7 on a 64bit gravitation instance using an ubuntu 22.04 container. What can we do to fix this?

Do you have any ideas @graebm?

[oschwartz10612]

Okay I wasted hours on this! The solution (at lease when running on an ec2 or other linux based distro) is to add setarch linux32 -B to the beginning of the ExecStart line in /lib/systemd/system/docker.service and /lib/systemd/system/containerd.service.

[TingDaoK]

It's probably fixed by aws/aws-lc#1015 already.

[jmklix]

Can you confirm if you are still running into this build failure @bryang-spindance

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.