Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add build image function to GitHub Actions #1008

Merged
merged 1 commit into from
Jul 9, 2020
Merged

Conversation

ecdye
Copy link
Member

@ecdye ecdye commented Jun 29, 2020

The 'Build Image' action was primarily written by Holger Friedrich and
then edited for better functionality in the openHABian core repo.

This also contains other changes to the CI setup, such as moving the
ShellCheck checks out of Travis and instead enforcing them in GitHub
Actions.

Along with theses changes, some improvements to the build script and
install output were also made. These changes to the build script include
using cURL over wget for downloading needed files and also cloning the
openHABian repo in the build process rather than the install process.
This change was made because the repo must be present for offline
install to work, and as such it makes sense to provide it with the image
to reduce points of failure on install.

Fixes #991

Co-authored-by: Holger Friedrich [email protected]
Co-authored-by: Markus Storm [email protected]
Signed-off-by: Ethan Dye [email protected]

@ecdye ecdye added the CI-Pipeline Relating to CI label Jun 29, 2020
@ecdye ecdye mentioned this pull request Jun 29, 2020
13 tasks
@ecdye ecdye requested review from mstormi and holgerfriedrich June 29, 2020 01:08
@holgerfriedrich
Copy link
Member

holgerfriedrich commented Jun 29, 2020

@ecdye Wait, there are a few things to consider.

  1. artifacts will need a lot of space.
  2. to be clarified if we (in this case the organization) will be charged if we exceed the limits (@mstormi ever heard about this?)
  3. artifacts are deleted automatically after 30 days (or by dropping them manually)
  4. we will cause a lot of useless traffic on the download site (as each build) downloads a full image (caching plugin might be possible, but probably not for this size)

...this is why I did it with a manual trigger.

@mstormi
Copy link
Contributor

mstormi commented Jun 29, 2020

to be clarified if we (in this case the organization) will be charged if we exceed the limits (@mstormi ever heard about this?)

For jfrog I think that applies but for Github/Travis I don't know. @ecdye look it up please.

@ecdye
Copy link
Member Author

ecdye commented Jun 29, 2020

@holgerfriedrich To address your concerns, For open source repos, there is no limit on storage. Storage is only charged on private repos so we are fine in that regard. I can look into caching but I do not believe that downloads from source are a huge issue with our scope. If that is a real concern then I can look into making PRs a manual start .

@ecdye ecdye marked this pull request as draft June 29, 2020 16:59
@ecdye ecdye force-pushed the build branch 10 times, most recently from 0b8c7b8 to c464155 Compare June 29, 2020 20:33
@ecdye
Copy link
Member Author

ecdye commented Jun 29, 2020

@holgerfriedrich Caching is possible and is now implemented and working, that should take care of all of your concerns? Any other issues before merging you have?

And for more info on GitHub policy read https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts and https://help.github.com/en/actions/configuring-and-managing-workflows/caching-dependencies-to-speed-up-workflows

@ecdye ecdye marked this pull request as ready for review June 29, 2020 20:46
@holgerfriedrich
Copy link
Member

@ecdye good to know the GitHub policies on storage.
I cannot review today, spent my time on getting apt prefetch :-) but failed with dpkg install in mounted target image :-(

Not sure yet which is the best approach to trigger. I do not like the idea that all of our commits and all PRs create images. This will waste space and computing power. If we trigger only on master and if build.bash is modified, we will not have an chance to test on PRs before merge....

@ecdye
Copy link
Member Author

ecdye commented Jun 30, 2020

I don't feel that it is really a waste of space as GitHub prunes it and takes care of. I am under the belief that because it is a service they offer and we will routinely make use of it, it is not an issue. It also is a part of our CI as it tests that what changes we make allow the image to build and then us to test on real HW.

That is my opinion. What are your thoughts @mstormi?

@mstormi
Copy link
Contributor

mstormi commented Jun 30, 2020

but failed with dpkg install in mounted target image :-(

Nice try! But if you don't get that to work, it's fine to just place the .deb's somewhere and install during the target's installation run.

@mstormi
Copy link
Contributor

mstormi commented Jun 30, 2020

@holgerfriedrich > but failed with dpkg install in mounted target image :-(

apt-get install --download-only PKGNAME # mind that's assuming you run on the same architecture as the target does
cp /var/cache/apt/archives/PKGNAME.deb $TARGETMOUNT/var/cache/apt/archives/ # mind copying all dependencies, too

Then just have the system itself do the usual apt-get install on first time installation. It should use the pre-cached pkg.

@ecdye ecdye force-pushed the build branch 5 times, most recently from 9a615f6 to d78eda4 Compare July 6, 2020 15:13
@ecdye ecdye requested a review from holgerfriedrich July 6, 2020 15:37
@ecdye
Copy link
Member Author

ecdye commented Jul 6, 2020

@mstormi @holgerfriedrich
After the latest push, the image appears to be building and installing correctly and produces a beautiful install output:



2020-05-27_08:24:55_BST [openHABian] Starting the openHABian initial setup.
2020-05-27_08:24:56_BST [openHABian] Storing configuration... OK
2020-05-27_08:24:56_BST [openHABian] Starting webserver with installation log... OK
2020-07-06_20:42:31_BST [openHABian] Changing default username and password... OK
2020-07-06_20:42:38_BST [openHABian] Setting up Ethernet connection... OK
2020-07-06_20:42:38_BST [openHABian] Ensuring network connectivity... OK
2020-07-06_20:42:39_BST [openHABian] Waiting for dpkg/apt to get ready... OK
2020-07-06_20:42:50_BST [openHABian] Updating repositories and upgrading installed packages... OK
2020-07-06_20:44:49_BST [openHABian] Installing git package... OK
2020-07-06_20:45:01_BST [openHABian] Cloning myself from https://github.com/ecdye/openhabian, build branch... OK
2020-07-06_20:45:02_BST [openHABian] Executing openhabian-setup.sh unattended...
2020-07-06_20:45:02_BST [openHABian] Checking for root privileges... OK
2020-07-06_20:45:06_BST [openHABian] Loading configuration file '/etc/openhabian.conf'... OK
2020-07-06_20:45:06_BST [openHABian] Adjusting swap size to 1848 MB ... OK
2020-07-06_20:45:06_BST [openHABian] Setting timezone based on IP geolocation... OK (America/Denver)
2020-07-06_13:45:51_MDT [openHABian] Setting locale based on openhabian.conf... OK
2020-07-06_13:46:26_MDT [openHABian] Setting hostname of the base system... OK
2020-07-06_13:46:26_MDT [openHABian] Setting the GPU memory split down to 16MB for headless system... OK
2020-07-06_13:46:26_MDT [openHABian] Enabling Audio output... OK
2020-07-06_13:46:26_MDT [openHABian] Installing basic can't-be-wrong packages (screen, vim, ...)... OK
2020-07-06_13:47:04_MDT [openHABian] Installing additional needed packages... OK
2020-07-06_13:47:20_MDT [openHABian] Installing additional bluetooth packages... OK
2020-07-06_13:47:27_MDT [openHABian] Adding slightly tuned bash config files to system... OK
2020-07-06_13:47:27_MDT [openHABian] Adding slightly tuned vim config file to system... OK
2020-07-06_13:47:27_MDT [openHABian] Installing FireMotD required packages... OK
2020-07-06_13:47:35_MDT [openHABian] Downloading FireMotD... OK
2020-07-06_13:47:35_MDT [openHABian] Setting up FireMotD... OK
2020-07-06_13:48:20_MDT [openHABian] Generating FireMotD theme... OK
2020-07-06_13:48:24_MDT [openHABian] Setting up FireMotD apt updates count service... OK
2020-07-06_13:48:30_MDT [openHABian] Installing Java Zulu prerequisites (libc, libstdc++, zlib1g)... OK
2020-07-06_13:48:30_MDT [openHABian] Downloading Java Zulu 8 32-Bit OpenJDK... OK
2020-07-06_13:48:52_MDT [openHABian] Installing Java Zulu 8 32-Bit OpenJDK... OK
2020-07-06_13:48:54_MDT [openHABian] Installing or upgrading to latest openHAB release (stable)... OK
2020-07-06_13:49:24_MDT [openHABian] Optimizing Java to run on low memory single board computers... OK
2020-07-06_13:51:26_MDT [openHABian] Adding an openHAB dashboard tile for 'openhabiandocs'... OK
2020-07-06_13:51:26_MDT [openHABian] Adding openHAB syntax to vim editor... OK
2020-07-06_13:51:27_MDT [openHABian] Adding openHAB syntax to nano editor... OK
2020-07-06_13:51:27_MDT [openHABian] Adding openHAB scheme to multitail... OK
2020-07-06_13:51:27_MDT [openHABian] Preparing openHAB folder mounts under /srv/... OK
2020-07-06_13:51:30_MDT [openHABian] Applying file permissions recommendations... OK
2020-07-06_13:51:32_MDT [openHABian] Applying miscellaneous system settings... OK
2020-07-06_13:51:32_MDT [openHABian] Installing Samba... OK
2020-07-06_13:52:21_MDT [openHABian] Setting up Samba network shares... 
2020-07-06_13:52:21_MDT [openHABian] Setting up Samba service... OK
2020-07-06_13:52:23_MDT [openHABian] Installing Frontail prerequsites (NodeJS)... 
2020-07-06_13:52:23_MDT [openHABian] Adding required keys to apt... OK
2020-07-06_13:52:24_MDT [openHABian] Adding NodeSource repository to apt... OK
2020-07-06_13:52:29_MDT [openHABian] Installing NodeJS... OK
2020-07-06_13:52:48_MDT [openHABian] Beginning setup of the openHAB Log Viewer (frontail)... 
2020-07-06_13:52:48_MDT [openHABian] Installing openHAB Log Viewer (frontail)... OK
2020-07-06_13:53:02_MDT [openHABian] Configuring openHAB Log Viewer (frontail)... OK
2020-07-06_13:53:03_MDT [openHABian] Setting up openHAB Log Viewer (frontail) service... OK
2020-07-06_13:53:04_MDT [openHABian] Adding an openHAB dashboard tile for 'frontail'... OK
2020-07-06_13:53:11_MDT [openHABian] Installing ZRAM ...
2020-07-06_13:55:26_MDT [openHABian] Cleaning up ... 
2020-07-06_13:55:35_MDT [openHABian] Execution of 'openhabian-setup.sh unattended' completed.
2020-07-06_13:55:35_MDT [openHABian] Waiting for openHAB to become ready on openhab... OK
2020-07-06_13:55:35_MDT [openHABian] Visit the openHAB dashboard now: http://openhab:8080
2020-07-06_13:55:35_MDT [openHABian] To gain access to a console, simply reconnect.
2020-07-06_13:55:35_MDT [openHABian] First time setup successfully finished.

I did not, however test offline install as it won't work quite right without #984 being finished.

@holgerfriedrich
Copy link
Member

Just to let you know about the new GitHub functionality: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

@ecdye
Copy link
Member Author

ecdye commented Jul 7, 2020

Just to let you know about the new GitHub functionality: https://github.blog/changelog/2020-07-06-github-actions-manual-triggers-with-workflow_dispatch/

That will be amazing if it works, my one concern is that I don't know if it will let people without write access trigger them. (i.e. me). Lets test and find out, as that is a much more ideal solution.

@ecdye
Copy link
Member Author

ecdye commented Jul 7, 2020

Hard to tell if it works until is has been merged because the other test cases cause it to trigger. We may have to wait and see if it works as intended but it should.

@ecdye ecdye requested a review from holgerfriedrich July 7, 2020 16:00
@ecdye ecdye force-pushed the build branch 3 times, most recently from c870231 to 2a76f95 Compare July 8, 2020 20:36
@ecdye
Copy link
Member Author

ecdye commented Jul 8, 2020

Maybe you got me wrong. Why not check all .bash files - even if only used during CI....

Mostly because those are the only ones that we have previoously enforced shellcheck on. If I were to do them all I would probably need to fix several issues and that is not really inside the scope of this PR.

@holgerfriedrich
Copy link
Member

Maybe you got me wrong. Why not check all .bash files - even if only used during CI....

Mostly because those are the only ones that we have previoously enforced shellcheck on. If I were to do them all I would probably need to fix several issues and that is not really inside the scope of this PR.

I was basically referring to the one you just introduced. But if this is a problem, don't mind to leave it out for now.

The 'Build Image' action was primarily written by Holger Friedrich and
then edited for better functionality in the openHABian core repo.

This also contains other changes to the CI setup, such as moving the
ShellCheck checks out of Travis and instead enforcing them in GitHub
Actions.

Along with theses changes, some improvements to the build script and
install output were also made. These changes to the build script include
using cURL over wget for downloading needed files and also cloning the
openHABian repo in the build process rather than the install process.
This change was made because the repo must be present for offline
install to work, and as such it makes sense to provide it with the image
to reduce points of failure on install.

Fixes openhab#991

Co-authored-by: Holger Friedrich <[email protected]>
Co-authored-by: Markus Storm <[email protected]>
Signed-off-by: Ethan Dye <[email protected]>
@ecdye
Copy link
Member Author

ecdye commented Jul 9, 2020

Done

@ecdye ecdye requested a review from holgerfriedrich July 9, 2020 01:25
Copy link
Member

@holgerfriedrich holgerfriedrich left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's see how it works.
@mstormi ready to merge?

@mstormi mstormi merged commit 6342c92 into openhab:master Jul 9, 2020
@ecdye ecdye deleted the build branch August 2, 2020 03:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI-Pipeline Relating to CI
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Keep ShellCheck action in Travis?
3 participants