Sedna control plane supports amd64/arm64 #196

JimmyYang20 · 2021-09-15T08:49:09Z

fix issues: #170

Signed-off-by: JimmyYang20 [email protected]

llhuii · 2021-09-15T09:01:36Z

hack/lib/mutil-arch.sh

+    https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-amd64
+chmod a+x ~/.docker/cli-plugins/docker-buildx
+
+cat > ~/.docker/config.json <<EOF


this would overwrite the user original docker config

llhuii · 2021-09-15T09:03:53Z

Makefile

+	for target in "gm" "lc"; do \
+	docker buildx build --push \
+	--build-arg GO_LDFLAGS=${GO_LDFLAGS} \
+	--platform linux/amd64,linux/arm64 \


make the platforms configurable.

llhuii · 2021-09-16T04:06:47Z

hack/lib/mutil-arch.sh

+
+# Update buildx plugin to version 6.0+.
+mkdir -pv ~/.docker/cli-plugins/
+wget -O ~/.docker/cli-plugins/docker-buildx \


please check it first, buildx maybe already built in docker, or installed by this way.

llhuii · 2021-09-16T04:07:59Z

hack/lib/mutil-arch.sh

+# Update buildx plugin to version 6.0+.
+mkdir -pv ~/.docker/cli-plugins/
+wget -O ~/.docker/cli-plugins/docker-buildx \
+    https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-amd64


please get arch from current build machine first instead of hard amd64

Suggested change

https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-amd64

https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-$arch

llhuii · 2021-09-16T04:09:33Z

hack/lib/golang.sh

@@ -207,6 +207,9 @@ sedna::golang::build_binaries() {
  goldflags="${GOLDFLAGS=-s -w -buildid=} $(sedna::version::ldflags)"
  gogcflags="${GOGCFLAGS:-}"

+  CGO_ENABLED=0
+  GOARCH=arm64


Only build arm64, how about other archs, e.g. x86?

llhuii · 2021-09-22T01:47:20Z

hack/lib/mutil-arch.sh

+docker_buildx=~/.docker/cli-plugins/docker-buildx
+if [ ! -x $docker_buildx ];then
+  mkdir -pv $(dirname $docker_buildx)
+  arch=$(dpkg --print-architecture)


dpkg only available in debain like system, use uname -m

delete install buildx

llhuii · 2021-09-22T01:48:05Z

hack/lib/mutil-arch.sh

+  arch=$(dpkg --print-architecture)
+  wget -O $docker_buildx \
+      https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-$arch
+  chmod a+x ~/.docker/cli-plugins/docker-buildx


Suggested change

chmod a+x ~/.docker/cli-plugins/docker-buildx

chmod a+x $docker_buildx

delete install buildx

llhuii · 2021-09-22T01:51:36Z

scripts/storage-initializer/push_image.sh

@@ -13,8 +13,14 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+set -x


Suggested change

set -x

llhuii · 2021-09-22T01:53:13Z

hack/lib/mutil-arch.sh

+  mkdir -pv $(dirname $docker_buildx)
+  arch=$(dpkg --print-architecture)
+  wget -O $docker_buildx \
+      https://github.com/docker/buildx/releases/download/v0.6.3/buildx-v0.6.3.linux-$arch


make buildx version as variable

delete install buildx

llhuii · 2021-09-22T02:33:11Z

Makefile

@@ -18,6 +18,10 @@ OUT_DIR ?= _output
 OUT_BINPATH := $(OUT_DIR)/bin

 IMAGE_REPO ?= kubeedge
+
+PLATFORMS ?= "linux/amd64,linux/arm64"


Can you add a description about full list of platforms?

llhuii · 2021-09-22T02:34:47Z

scripts/storage-initializer/push_image.sh

@@ -17,4 +17,9 @@
 cd "$(dirname "${BASH_SOURCE[0]}")"

 source build_image.sh
-docker push $IMAGE
+
+if [ ! -n "$1" ]; then


Use environment variable instead of argument list to toggle cross build capability.

Such as

Suggested change

if [ ! -n "$1" ]; then

if [ -z "${CROSS_PLATFORMS:-}" ]; then

llhuii · 2021-09-22T03:37:22Z

hack/lib/mutil-arch.sh

@@ -0,0 +1,47 @@
+#!/usr/bin/env bash


delete this script

No need since we can leave the buildx download/config to developer.

we need to have buildx multiple platforms features config, leave the buildx download to developer.

llhuii · 2021-09-22T04:16:55Z

Add a cross build target in Makefile, not only image push action

JimmyYang20 · 2021-09-22T07:12:01Z

@llhuii
the current limitation of buildx is combination of --load and multiple values to --platform.
This means that buildx uses --load only for single platform。

llhuii · 2021-09-23T01:41:03Z

hack/lib/buildx.sh

+
+# Check whether buildx exists.
+if ! docker buildx >/dev/null 2>&1; then
+  echo "ERROR: docker buildx not available. Docker 19.03 or higher is required with experimental features enabled"


log error to stderr

Suggested change

echo "ERROR: docker buildx not available. Docker 19.03 or higher is required with experimental features enabled"

echo "ERROR: docker buildx not available. Docker 19.03 or higher is required with experimental features enabled" >&2

llhuii · 2021-09-23T01:44:23Z

hack/lib/buildx.sh

+docker run --privileged --rm tonistiigi/binfmt --install all
+
+# Create a new builder which gives access to the new multi-architecture features.
+builder_instance="build-node-sedna"


prefix sedna?

Suggested change

builder_instance="build-node-sedna"

builder_instance="sedna-buildx"

llhuii · 2021-09-23T02:09:03Z

hack/lib/buildx.sh

+
+# Create a new builder which gives access to the new multi-architecture features.
+builder_instance="build-node-sedna"
+temp=$(docker buildx ls | awk NF=1)


How about using docker buildx inspect $builder_instance?

https://github.com/docker/buildx/blob/master/docs/reference/buildx_inspect.md

ok, use if ! docker buildx inspect $build_instance

remember to discard stdout/stdin

llhuii · 2021-09-23T02:26:00Z

@llhuii
the current limitation of buildx is combination of --load and multiple values to --platform.
This means that buildx uses --load only for single platform。

Use the -o option for local build, https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#output

JimmyYang20 · 2021-09-23T06:41:48Z

@llhuii
the current limitation of buildx is combination of --load and multiple values to --platform.
This means that buildx uses --load only for single platform。

Use the -o option for local build, https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#output

can not do it by my test, https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#image

JimmyYang20 · 2021-09-26T09:18:03Z

@llhuii
the current limitation of buildx is combination of --load and multiple values to --platform.
This means that buildx uses --load only for single platform。

Use the -o option for local build, https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#output

can not do it by my test, https://github.com/docker/buildx/blob/master/docs/reference/buildx_build.md#image

I mean cross built local file instead of pushing image to registry directly.

You can refer the kubernetes repo:
# make DBG_MAKEFILE=1 release-images KUBE_BUILD_CONFORMANCE=n KUBE_BUILD_PLATFORMS=linux/amd64
                                         
# tree _output/release-stage/server/       
_output/release-stage/server/                         
├── linux-amd64                                
│   └── kubernetes                         
│       └── server                                    
│           └── bin                            
│               ├── kube-apiserver         
│               ├── kube-apiserver.docker_tag         
│               ├── kube-apiserver.tar         
│               ├── kube-controller-manager
│               ├── kube-controller-manager.docker_tag
│               ├── kube-controller-manager.tar
│               ├── kube-proxy
│               ├── kube-proxy.docker_tag
│               ├── kube-proxy.tar
│               ├── kube-scheduler
│               ├── kube-scheduler.docker_tag
│               └── kube-scheduler.tar
├── linux-arm
│   └── kubernetes
... omit here ...

_output/
└── images
    ├── gm-v0.3.0-linux-amd64.tar
    ├── gm-v0.3.0-linux-arm64.tar
    ├── lc-v0.3.0-linux-amd64.tar
    └── lc-v0.3.0-linux-arm64.tar

llhuii · 2021-09-27T02:34:54Z

good job
/lgtm

llhuii · 2021-09-27T03:24:12Z

The checker docker cross build images for gm/lc/kb caused 38min, too slow.

llhuii · 2021-09-27T03:42:07Z

/hold

JimmyYang20 · 2021-09-27T12:36:57Z

The checker docker cross build images for gm/lc/kb caused 38min, too slow.

fix it ,
The checker docker cross build images for gm/lc/kb caused 8min 11s

reference: https://docs.docker.com/buildx/working-with-buildx/#build-multi-platform-images
FROM --platform=$BUILDPLATFORM golang:alpine AS build

llhuii · 2021-09-28T01:19:29Z

hack/lib/buildx.sh

+  for component in ${COMPONENTS[@]}; do
+    echo "building ${PLATFORMS} image for ${component}"
+
+    sed "/AS builder/s/FROM/FROM --platform=\$BUILDPLATFORM/g" build/${component}/Dockerfile > ${temp_dockerfile}


The cross-build-dockerfile generating code happend at two places, use function generate-cross-build-dockerfile

control plane images support amd64/arm64 add a workerflow action: docker_cross_build Signed-off-by: JimmyYang20 <[email protected]>

llhuii · 2021-09-28T03:04:48Z

/lgtm
/hold cancel

llhuii · 2021-09-28T03:31:33Z

/approve

kubeedge-bot · 2021-09-28T03:31:40Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: llhuii

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [llhuii]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

kubeedge-bot requested review from jaypume and TymonXie September 15, 2021 08:49

kubeedge-bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 15, 2021

JimmyYang20 force-pushed the optimize-platform branch from 8a1cad5 to 31e53c6 Compare September 15, 2021 08:52

JimmyYang20 changed the title ~~Sedna supports arm64 arch~~ Sedna supports mult-arch Sep 15, 2021

JimmyYang20 force-pushed the optimize-platform branch 6 times, most recently from 22826f1 to 2666fd9 Compare September 16, 2021 03:29

llhuii reviewed Sep 16, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch 3 times, most recently from 7780eaf to 4017c62 Compare September 18, 2021 09:53

JimmyYang20 changed the title ~~Sedna supports mult-arch~~ Sedna component images support amd64/arm64 arch Sep 18, 2021

llhuii reviewed Sep 22, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch 2 times, most recently from 1211c18 to 365641d Compare September 22, 2021 02:29

llhuii reviewed Sep 22, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch from 365641d to e6e10c1 Compare September 22, 2021 03:04

llhuii reviewed Sep 22, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch from e6e10c1 to b9fee0b Compare September 22, 2021 07:04

llhuii reviewed Sep 23, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch from b9fee0b to bbbe967 Compare September 23, 2021 06:56

JimmyYang20 force-pushed the optimize-platform branch 3 times, most recently from b3dcbba to 776e71b Compare September 26, 2021 11:47

JimmyYang20 changed the title ~~Sedna component images support amd64/arm64 arch~~ Sedna control plane supports amd64/arm64 Sep 26, 2021

JimmyYang20 force-pushed the optimize-platform branch 3 times, most recently from 98a76af to 1a65964 Compare September 27, 2021 02:28

kubeedge-bot assigned llhuii Sep 27, 2021

kubeedge-bot added the lgtm Indicates that a PR is ready to be merged. label Sep 27, 2021

kubeedge-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 27, 2021

JimmyYang20 force-pushed the optimize-platform branch from 1a65964 to a737a08 Compare September 27, 2021 12:33

kubeedge-bot removed the lgtm Indicates that a PR is ready to be merged. label Sep 27, 2021

llhuii reviewed Sep 28, 2021

View reviewed changes

JimmyYang20 force-pushed the optimize-platform branch from a737a08 to 9165ea1 Compare September 28, 2021 02:06

Sedna control plane supports amd64/arm64

d8766bd

control plane images support amd64/arm64 add a workerflow action: docker_cross_build Signed-off-by: JimmyYang20 <[email protected]>

JimmyYang20 force-pushed the optimize-platform branch from 9165ea1 to d8766bd Compare September 28, 2021 02:20

kubeedge-bot added lgtm Indicates that a PR is ready to be merged. and removed do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. labels Sep 28, 2021

kubeedge-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 28, 2021

kubeedge-bot merged commit 1d7bd48 into kubeedge:main Sep 28, 2021

This was referenced Oct 15, 2021

Add arm support #13

Closed

Fix building wrong GM/LC arm64 docker images #215

Merged

	chmod a+x ~/.docker/cli-plugins/docker-buildx
	chmod a+x $docker_buildx

	if [ ! -n "$1" ]; then
	if [ -z "${CROSS_PLATFORMS:-}" ]; then

	echo "ERROR: docker buildx not available. Docker 19.03 or higher is required with experimental features enabled"
	echo "ERROR: docker buildx not available. Docker 19.03 or higher is required with experimental features enabled" >&2

	builder_instance="build-node-sedna"
	builder_instance="sedna-buildx"

Sedna control plane supports amd64/arm64 #196

Sedna control plane supports amd64/arm64 #196

Conversation

JimmyYang20 commented Sep 15, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llhuii commented Sep 22, 2021

JimmyYang20 commented Sep 22, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llhuii commented Sep 23, 2021 • edited Loading

JimmyYang20 commented Sep 23, 2021

JimmyYang20 commented Sep 26, 2021 • edited Loading

llhuii commented Sep 27, 2021

llhuii commented Sep 27, 2021

llhuii commented Sep 27, 2021

JimmyYang20 commented Sep 27, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

llhuii commented Sep 28, 2021

llhuii commented Sep 28, 2021

kubeedge-bot commented Sep 28, 2021

JimmyYang20 commented Sep 15, 2021 •

edited

Loading

llhuii commented Sep 23, 2021 •

edited

Loading

JimmyYang20 commented Sep 26, 2021 •

edited

Loading

JimmyYang20 commented Sep 27, 2021 •

edited

Loading