Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform #130

torao · 2023-01-19T11:05:04Z

Description

This PR is a continuation of PR #126 and is related to issue #116. With this PR, executable binaries for the linux/arm64 platform will be built for each release (darwin/amd64 and darwin/arm64 are now disabled as they should be run on the self-hosted runner) and added to the release note. A major change is that cross-compiling through Docker containers has been abandoned in favour of building with the appropriate native compiler on the platform OS. This is because our LBM involves several native compilation steps (cgo) steps and cannot be built using Golang cross-compilation alone. For this reason, the docker image and make reproducible for cross-compilation have been removed and they have been replaced with a simple make build in release-build.yml.

The current lbm v0.46.0 binaries depend on the libwasmvm v1.0.0-0.10.0 library. As of this PR, release assets are no longer individual binaries, but TGZ archive files that bundle the appropriate libraries. Each libwasmvm library can also be obtained separately.

LBM archives can be extracted and run for each platform by specifying LD_LIBRARY_PATH (in macOS use DYLD_LIBRARY_PATH instead).

root@6a840a37a64e:~# uname -a
Linux 6a840a37a64e 5.15.49-linuxkit #1 SMP PREEMPT Tue Sep 13 07:51:32 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

root@6a840a37a64e:~# tar zxvf lbm-v0.0.92-linux-amd64.tgz 
lbm-v0.0.92-linux-amd64/
lbm-v0.0.92-linux-amd64/lbm-v0.0.92-linux-amd64
lbm-v0.0.92-linux-amd64/libwasmvm.x86_64.so
root@6a840a37a64e:~# cd lbm-v0.0.92-linux-amd64
root@6a840a37a64e:~/lbm-v0.0.92-linux-amd64# LD_LIBRARY_PATH=. ./lbm-v0.0.92-linux-amd64 version
v0.0.92

Building from source can also follow the same steps. In this case you'll need go, make, git, etc. on your environment.

root@6a840a37a64e:~# tar zxvf lbm-v0.0.92.tgz 
lbm-v0.0.92/
lbm-v0.0.92/.gitattributes
...
root@6a840a37a64e:~# cd lbm-v0.0.92
root@6a840a37a64e:~/lbm-v0.0.92# make clean build VERSION=vX.Y.Z
...
root@6a840a37a64e:~/lbm-v0.0.92# build/lbm version
vX.Y.Z

Motivation and context

See Issue #116

How has this been tested?

Screenshots (if appropriate):

Checklist:

I followed the contributing guidelines.
I have updated the documentation accordingly.
I have added tests to cover my changes.

tkxkd0159 · 2023-01-20T09:21:52Z

I think it's very bad UX that the user have to clone the repo and build dynamic library for binary execution. The intention of serving binary on release note is ready-to-run. So we need to discuss about simplifying that flow.

The approach of cross-compiling for each platform on Linux was difficult for cases like LBM that required native compilation of C/C++. For this reason, I use compilers suitable for each OS to perform release builds. Along with this, the docker image for cross-compiling run from `make reproducible` and the shell script used for it are no longer required.

torao · 2023-01-24T10:22:49Z

@tkxkd0159
I found the pre-builds libwasmvm.x86_64.so (for linux/amd64), libwasmvm.aarch64.so (for linux/arm64) and libwasmvm.dylib (for darwin/??) in the line/wasmvm repository (the current version corresponding to lbm is 1.0.0-0.10.0).
https://github.com/line/wasmvm/tree/v1.0.0-0.10.0/api

And we may need to decide in how we're going to stay distributed.

Use the instructions to checkout the line/wasmvm repository and do a source build.
Guide uses to download pre-built files from the line/wasmvm repository (we need to specify what version they are).
Distribute as tgz or zip, including *.so files, rather than as a single binary.
Change the build settings to make it a single binary (if possible).

tkxkd0159 · 2023-01-25T04:46:26Z

And we may need to decide in how we're going to stay distributed.

Use the instructions to checkout the line/wasmvm repository and do a source build.

Guide uses to download pre-built files from the line/wasmvm repository (we need to specify what version they are).

Distribute as tgz or zip, including *.so files, rather than as a single binary.

Change the build settings to make it a single binary (if possible).

@torao
Imo, third, fourth option is best for UX. Fourth option is only possible in linux because wasmvm doesn't support static lib for MacOS now. So I suggest to make a package bundle for MacOS if it can.

torao · 2023-01-25T06:58:17Z

Yes, I think 3 or 4 is better too. 3 is the one with less difficulty, just more steps. How do you want to distribute them? @zemyblue

zemyblue · 2023-01-26T02:16:43Z

Yes, I think 3 or 4 is better too. 3 is the one with less difficulty, just more steps. How do you want to distribute them? @zemyblue

I think 4 is better, but if impossible 3 option.
And I think it would be nice to be able to make it into an installable package like golang pkg(https://go.dev/dl/).

.github/workflows/release-build.yml

zemyblue

And please add changelog

torao · 2023-01-31T03:26:46Z

Changed to use our self-hosted runner and exclude darwin/* release builds.

The self-hosted runner cannot be run from personal repositories, so I push this branch to line/lbm repository to ensure release builds were successfully created. (The branch and release will be removed later.)

zemyblue · 2023-01-31T03:54:56Z

RELEASE_CHANGELOG.md

+## [v0.8.0]
+
+### Features
+* [\#126](https://github.com/line/lbm/pull/126), [\#130](https://github.com/line/lbm/pull/130) Automatically generate a release note, linux/amd64 and linux/arm64 binaries on release


Please move to CHANGELOG.md.
This RELEASE_CHANGELOG.md is wrote when release.

fixed, please check it

tkxkd0159 · 2023-01-31T04:36:44Z

.github/workflows/release-build.yml

+      - name: "Upload artifact: compressed repository source"
+        uses: actions/upload-release-asset@v1
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
+        with:
+          upload_url: ${{ steps.create_release.outputs.upload_url }}
+          asset_path: lbm-${{ env.VERSION }}.tgz
+          asset_name: lbm-${{ env.VERSION }}.tgz
+          asset_content_type: application/gzip


why do you need to upload compressed repo source? Because every github release already serve them by default.

This release build process is based on Gaia's release.yml and their release. Not a strong reason but creating the source archive separate from the auto-generated one has the following advantages.

The checksum of the source archive can be described.

It has been verified that the source archive can be expanded and built.

It can be ensured that it doesn't contain unnecessary metadata such as .git.

The reasons why these are difficult with the source archive provided by default are:

There is no action to retrieve the archive provided by default. It is therefore not possible to create a release build from the default archive (i.e., it cannot be verified as buildable).

Even if it were possible to retrieve the default archive, there is no action to rewrite the release description. Therefore, before running create_release, all the artifacts must be built and all their checksums are calculated. This is a conflicting dependency as the default source archive is generated at create_release time.

It might be possible to use the Web API directly, e.g. with wget instead of action, but this may not be a step that should be achieved using such a low-level configuration.

RELEASE_NOTE.md

This reverts commit 9c5929d.

zemyblue · 2023-02-01T00:39:09Z

CHANGELOG.md

@@ -37,7 +37,7 @@ Ref: https://keepachangelog.com/en/1.0.0/
 ## [Unreleased]

 ### Features
-* (build) [\#126](https://github.com/line/lbm/pull/126) Automatically generates release note and binaries
+* (build) [\#126](https://github.com/line/lbm/pull/126), [\#130](https://github.com/line/lbm/pull/130) Automatically generate a release note, linux/amd64 and linux/arm64 binaries on release


Please separate changes by each PR even if similar change later. not need to combine it.

Co-authored-by: Sujong Lee <[email protected]>

* chore: apply detached x/wasmd Signed-off-by: zemyblue <[email protected]> * chore: add changelog Signed-off-by: zemyblue <[email protected]> Signed-off-by: zemyblue <[email protected]>

tnasu · 2023-02-01T10:55:39Z

.github/workflows/release-build.yml

+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        arch: [ amd64, arm64 ]


unnecessary

add build target linux/amd64

7287a1f

torao added the enhancement New feature or request label Jan 19, 2023

torao requested review from zemyblue, 0Tech, dudong2 and tkxkd0159 January 19, 2023 11:05

torao self-assigned this Jan 19, 2023

torao marked this pull request as ready for review January 20, 2023 02:06

torao changed the title ~~Add a release build for the linux/amd64 platform~~ Add a release build for the linux/arm64 platform Jan 20, 2023

torao mentioned this pull request Jan 24, 2023

Add release builds for darwin/amd64 and darwin/arm64 #132

Closed

13 tasks

add a dependent library to the artifact instead of a single binary

37ee962

torao changed the title ~~Add a release build for the linux/arm64 platform~~ Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform Jan 27, 2023

torao marked this pull request as draft January 27, 2023 12:13

torao marked this pull request as ready for review January 30, 2023 02:04

zemyblue reviewed Jan 30, 2023

View reviewed changes

.github/workflows/release-build.yml Outdated Show resolved Hide resolved

.github/workflows/release-build.yml Outdated Show resolved Hide resolved

zemyblue reviewed Jan 30, 2023

View reviewed changes

torao added 2 commits January 31, 2023 11:31

change to run on self-hosted runner

c7c4cd9

add changelog

9c5929d

torao requested a review from zemyblue January 31, 2023 03:29

zemyblue reviewed Jan 31, 2023

View reviewed changes

tkxkd0159 reviewed Jan 31, 2023

View reviewed changes

dudong2 reviewed Jan 31, 2023

View reviewed changes

RELEASE_NOTE.md Outdated Show resolved Hide resolved

revert RELEASE_CHANGELOG.md and add it to CHANGELOG.md

584da23

This reverts commit 9c5929d.

torao requested review from tkxkd0159, dudong2 and zemyblue January 31, 2023 11:52

zemyblue approved these changes Feb 1, 2023

View reviewed changes

dudong2 approved these changes Feb 1, 2023

View reviewed changes

tkxkd0159 approved these changes Feb 1, 2023

View reviewed changes

torao and others added 4 commits February 1, 2023 19:32

update changelog

fc48c4a

typo

63fa84c

Co-authored-by: Sujong Lee <[email protected]>

chore: apply detached x/wasmd (#129)

ddcf4fd

* chore: apply detached x/wasmd Signed-off-by: zemyblue <[email protected]> * chore: add changelog Signed-off-by: zemyblue <[email protected]> Signed-off-by: zemyblue <[email protected]>

Merge branch 'main' into feature/prepare_linux_arm64_binary

0fb5bb5

torao merged commit efd5757 into Finschia:main Feb 1, 2023

tnasu reviewed Feb 1, 2023

View reviewed changes

.github/workflows/release-build.yml

runs-on: ubuntu-latest

strategy:

matrix:

arch: [ amd64, arm64 ]

Copy link

Member

tnasu Feb 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unnecessary

torao mentioned this pull request Feb 3, 2023

build and upload various OS release binary in release note automatically #116

Closed

zemyblue mentioned this pull request Apr 25, 2023

Release finschia v1.0.0 #182

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform #130

Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform #130

torao commented Jan 19, 2023 •

edited

Loading

tkxkd0159 commented Jan 20, 2023

torao commented Jan 24, 2023

tkxkd0159 commented Jan 25, 2023 •

edited

Loading

torao commented Jan 25, 2023

zemyblue commented Jan 26, 2023

zemyblue left a comment

torao commented Jan 31, 2023 •

edited

Loading

zemyblue Jan 31, 2023

torao Jan 31, 2023

tkxkd0159 Jan 31, 2023

torao Jan 31, 2023 •

edited

Loading

zemyblue Feb 1, 2023

torao Feb 1, 2023

tnasu Feb 1, 2023

Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform #130

Add a release build for the linux/arm64, darwin/amd64, and darwin/arm64 platform #130

Conversation

torao commented Jan 19, 2023 • edited Loading

Description

Motivation and context

How has this been tested?

Screenshots (if appropriate):

Checklist:

tkxkd0159 commented Jan 20, 2023

torao commented Jan 24, 2023

tkxkd0159 commented Jan 25, 2023 • edited Loading

torao commented Jan 25, 2023

zemyblue commented Jan 26, 2023

zemyblue left a comment

Choose a reason for hiding this comment

torao commented Jan 31, 2023 • edited Loading

zemyblue Jan 31, 2023

Choose a reason for hiding this comment

torao Jan 31, 2023

Choose a reason for hiding this comment

tkxkd0159 Jan 31, 2023

Choose a reason for hiding this comment

torao Jan 31, 2023 • edited Loading

Choose a reason for hiding this comment

zemyblue Feb 1, 2023

Choose a reason for hiding this comment

torao Feb 1, 2023

Choose a reason for hiding this comment

tnasu Feb 1, 2023

Choose a reason for hiding this comment

torao commented Jan 19, 2023 •

edited

Loading

tkxkd0159 commented Jan 25, 2023 •

edited

Loading

torao commented Jan 31, 2023 •

edited

Loading

torao Jan 31, 2023 •

edited

Loading