Introduction of Raspberry Pi OS #931

holgerfriedrich · 2020-05-29T21:14:08Z

Replace raspbian by raspios due to renaming of the OS
(beta) Added initial support for building 64-bit images

Resolves: #946

Signed-off-by: Holger Friedrich [email protected]

ecdye · 2020-05-31T01:04:33Z

RFC: Do we want to consider renaming the project, or do we keep our name and move on?

build-image/openhabian.pi-raspbian64.conf

mstormi · 2020-06-07T10:20:36Z

I'm currently enjoying the "offline installation" ~~dev~~use case #825 and while it's pretty hard to develop with (almost) no Internet access, this helped me identify the parts of the code we will need to rework for offline install to work (I hope, hard to test in this situation)

Will upload a PR when I'm back online. Meanwhile below the relevant code for you to include.
Can you insert that plus calls to install_zram_code and java_zulu_fetch Zulu8-32 into build.bash ?

That should pre-load the code versions that are current at the time of image build. Our current java and my new ZRAM code should work to install these from storage instead of Inet if Inet is down.
Make sure that /opt/zram and /opt/jdk are also included with the image.


install_zram_code() {
  mkdir -p "$1"
  /usr/bin/git clone -q --branch "$TAG" "$ZRAMGIT" "$1"

  cond_redirect apt-get install -y -q --no-install-recommends make
  cd "$1" || return 1
  /bin/sh ./install.sh
}

ZRamInstallLocation=/opt/zram
install_zram_code "$ZRamInstallLocation"

holgerfriedrich · 2020-06-07T15:16:05Z

@mstormi .

.xz.zip is caused by GitHub, all artifacts will be zipped during download. There seems no way to avoid it right now, but others are complaining about this as well. If we put the img file in the archive, you will have to wait for the img file to be compressed on the server side, this takes some time until the download starts.
64bit image is still experimental, there is not "lite" version available. GUI is now disabled by patching the standard systemd target.
pulling stuff in build.bash - no problem. BUT: compiling! This would require cross-compiling for the ARM platform (since I run the build on x86). Have you thought about creating deb packages for the zram build, this would ease the handling a lot?

Noticed some problems with during install, webif did not work and zulu8 64bit did not work. Trying the next build...

mstormi · 2020-06-07T15:32:11Z

* Have you thought about creating deb packages for the zram build, this would ease
  the handling a lot?

Yes but I want to avoid that. Just like the original zram author I'm no expert at building packages. And the compilation is not even for the ZRAM code itself but for a single tool (overlayfstools) which is in turn YET another upstream. That I don't want to mess with.
Then again you just made me aware that we would need to pre-fetch overlayfstools, too, if that's supposed to install offline. .... so probably I need to move that off the zram install routine, too
What's bad about compiling on image build ? Would relieve us from installing a compiler and needs to work anyway if we want true offline install capability.

PS: back in action with faster Internet

mstormi · 2020-06-08T10:05:14Z

did you set the number of tryUntil() attempts on "ping 9.9.9.9|| wget ...." to 60 from 10 ?
10 is way enough (every wget does retries already)

mstormi · 2020-06-08T17:30:19Z

Also remember to replace names in the docs

holgerfriedrich · 2020-06-08T20:05:34Z

@mstormi @ecdye willing to play around with the new 64bit image?
I just installed successfully on my Pi3b. A few points to clarify:
[ ] webif not tests, did not seem to work during last install
[ ] azul8-64 did not work with OH, complained about library issue. Switched to azul11-64 in latest image
[ ] graphical splash screen of the new OS still pops up during install before showing the command prompt
[ ] is_rapbian to be checked if it still works (confirmed that on 64bit it will identfy as debian, not raspbian) -> implications to be checked

@mstormi yes, more retries on slow WLAN APs....
@mstormi doc update done on pr branch, not applied to current image

holgerfriedrich · 2020-06-08T20:09:16Z

Yes but I want to avoid that. Just like the original zram author I'm no expert at building packages. And the compilation is not even for the ZRAM code itself but for a single tool (overlayfstools) which is in turn YET another upstream. That I don't want to mess with.

An alternative approach to cross compiling is cache the two git repos, and start compilation on the PI during installer run (first-boot.bash). Any special compile flags derived from the HW platform you compile on (e.g. optimizations only suitable for Pi3)?

mstormi · 2020-06-08T21:06:37Z

An alternative approach to cross compiling is cache the two git repos, and start compilation on the PI during installer run (first-boot.bash).

Yes that's what I'm out for. Check out #950 and try to integrate that with your code...can you call install_zram_code() during build to have the repo "cached", and include /opt/zram with the image ?
Also make sure git is pulled/installed in the image.

Any special compile flags derived from the HW platform you compile on (e.g. optimizations only suitable for Pi3)?

no, dead simple

mstormi · 2020-06-09T15:03:21Z

willing to play around with the new 64bit image?

where is it ? Link in #946 is still the old one I believe, at least boots into desktop

mstormi · 2020-06-09T15:20:44Z

I just interrupted install but on reboot it fails to continue on "apt-get --yes upgrade"
Manual execution of that resulted in "E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem."
After I did and rebootet, installation continued.
So while of course a user should not interrupt, and this problem probably is not specific to the new image, some users do like I did, and we should take care of that.
EDIT: forget it for this PR, I've created #952

mstormi · 2020-06-09T15:34:32Z

BTW I have added #950 (offline install enhancements) to the "testbuild" branch, so why not test that with Internet blocked.

Just did and it does not work because /boot/first-boot.bash (which is run) is different from /opt/openhabian/build-image/first-boot.bash (which is newer with my changes).
How's that? Is that copied on image build ?
If yes please use the testbuild branch for future image builds.

holgerfriedrich · 2020-06-09T21:18:55Z

@mstormi yes, build.bash is patching the file - but for RPi it is only adding the content of helpers.bash:
sed -i -e '1r functions/helpers.bash' $buildfolder/boot/first-boot.bash

Regarding the offline install: I would prefer to do things separately. Currently, I already merge 3 PRs + fixes to my master branch to build the image. I think we could start merging #890, for me the new installation worked fine in all of my tests. It will anyway only be rolled out when we release the new image.

mstormi · 2020-06-09T21:39:31Z

yes, build.bash is patching the file -but for RPi it is only adding the content of helpers.bash:

Then why is /boot/first-boot.bash so different from build-image/first-boot.bash ?

Re: offline install: ok, finish image build first, then we'll review what's left missing for offline install

Did you mean to say #890 is ready to be reviewed and merged ?

holgerfriedrich · 2020-06-10T05:48:20Z

Just did and it does not work because /boot/first-boot.bash (which is run) is different from /opt/openhabian/build-image/first-boot.bash (which is newer with my changes).
How's that? Is that copied on image build ?

Did you also apply the changes from this PR? It is mainly changing build.bash...?

mstormi · 2020-06-10T07:56:17Z

Did you also apply the changes from this PR? It is mainly changing build.bash...?

No. As far as I can see it does not change first-boot.bash or anything related to copying or creating that. I admit I've never built an image or changed anything about build.bash so am unaware of how it works. But /boot/first-boot.bash applies BEFORE git retrieves the current openhabian repo during the installation process. So it is as it is because it was built like that on image creation. Changes to repo such as mine cannot change that (they change the copy inside build-image but not the one in /boot).
Also irgendwas ist da noch komisch. Zumindest ich habe das noch nicht in Gänze verstanden.

Could you apply all of your image-build related changes (you spoke of 3 PRs but I don't know what these are) to the testbuild branch ? That's nothing other than today's master plus #950 . Thanks!

mstormi · 2020-06-10T08:02:23Z

FWIW, I tried the 64 bit image on my brand new RPi4. Note the last two lines of the output.
(I installed via Ethernet and did nothing else).
Possibly related to any of the open WiFi issues such as #912 ?

[email protected]'s password:
Linux openhab 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Tue Jun  9 17:59:22 2020

Wi-Fi is currently blocked by rfkill.
Use raspi-config to set the country before use.

mstormi · 2020-06-10T11:35:27Z

By the way, many of your commits have an invalid signature, please correct it
see https://github.com/openhab/openhabian/pull/953/checks

mstormi · 2020-06-10T12:33:51Z

ZRAM (or overlayfs-tools actually) does not compile both, online and offline, because of libattr1-dev package missing. Please add to image as it must be present for offline installs, too.

mstormi · 2020-06-10T19:53:13Z

I've been using both images on RPi 3 & 4 with no issues.
Only thing to catch the eye is #721 again - but #890 is not merged so that's to be expected.
Will merge that asap but wanted to have a cleanly forwarded stable first.

holgerfriedrich · 2020-06-10T21:13:36Z

@mstormi do you still have your 32bit RasPiOS system available?
Can you check

./functions/helpers.bash
is_debian
echo $?
is_raspbian
echo $?

Will try to build a rebased image tomorrow.

mstormi · 2020-06-10T22:03:06Z

debian = no (1), raspbian= yes (0)

mstormi · 2020-06-11T16:54:39Z

@mstormi yes, build.bash is patching the file - but for RPi it is only adding the content of helpers.bash:

I saw this results in the first TWO lines to start with #!.

mstormi · 2020-06-11T17:41:22Z

We also need these packages to be there offline:
python3-pip python3-wheel python3-setuptools
(and its dependencies)

mstormi · 2020-06-13T21:17:04Z

Hi Holger,
could you review #961 please and build another image to include that ?
What about the promised hook from #950 (comment) for all those packages we need to have available on offline installs ?

* Replace raspbian by raspios due to renaming of the OS * (beta) Added initial support for building 64-bit images * Set default runlevel to multi-user.target, disable auto-login * Update documentation Signed-off-by: Holger Friedrich <[email protected]>

holgerfriedrich · 2020-06-17T20:39:36Z

@mstormi I think we should go step by step. This changes basically only the download loacation and adds basic support for the 64bit beta image. The introduction of 64bit image will cause more issues (especially in external components) and should be handled in a separate issue. WDYT?

holgerfriedrich · 2020-06-17T20:40:26Z

@mstormi @ecdye new images with latest commits from today are available,
https://github.com/holgerfriedrich/openhabian/actions?query=workflow%3Abuild-image

ecdye

LGTM

build-image/first-boot.bash

ecdye · 2020-06-17T20:56:39Z

build-image/openhabian.pi-raspios64beta.conf

+
+# Java architecture mode
+# Valid arguments: "Adopt11", "Zulu8-32", "Zulu8-64", "Zulu11-32", or "Zulu11-64"
+java_opt=Zulu11-64


Java 11 default? Are you planning on this being the openHAB 3 recommended setup for a RPi3+?

See above, zulu8-64 did not work in my last test. This might have changed in the meantime, since java installation improved since then. Not tested yet.

ecdye · 2020-06-17T20:58:19Z

build.bash

+elif [ "$hw_platform" == "pi-raspios32" ] || [ "$hw_platform" == "pi-raspios64beta" ]; then
+  if [ "$hw_platform" == "pi-raspios64beta" ]; then
+    zipfile=raspios_lite_arm64_latest.zip
+    baseurl="https://downloads.raspberrypi.org/raspios_arm64/images/raspios_arm64-2020-05-28/2020-05-27-raspios-buster-arm64.zip"


You should add a note to change the link to a permanent redirect to the latest version when it becomes available.

sure. just the one beta image available as of now.

mstormi · 2020-06-17T21:57:15Z

The introduction of 64bit image will cause more issues (especially in external components) and should be handled in a separate issue. WDYT?

I'm fine to start with the 32bit image only.
We could offer the 64bit one marked experimental/unsupported. Or not at all for the time being ?

holgerfriedrich · 2020-06-17T21:59:24Z

I would out it into the code base but not explicitly mention it in the documentation.

mstormi

LGTM
feel free to merge yourself, I had my share of merges for ~~today~~ yesterday.

mstormi · 2020-06-17T22:10:45Z

build.bash

+    baseurl="https://downloads.raspberrypi.org/raspios_lite_armhf_latest"
+    bits=32
+  fi
+
  # Prerequisites
  echo_process "Checking prerequisites... "
  REQ_COMMANDS="git wget unzip crc32 dos2unix xz"


is that my hook for offline? If so add libattr1-dev

holgerfriedrich marked this pull request as draft May 29, 2020 21:14

holgerfriedrich changed the title ~~Introduction of Raspbian Pi OS~~ Introduction of Raspberry Pi OS May 29, 2020

holgerfriedrich force-pushed the pr-raspios branch from 2af4344 to 05f6c39 Compare May 29, 2020 21:26

holgerfriedrich added this to the v1.6 Hardware Image milestone May 31, 2020

holgerfriedrich force-pushed the pr-raspios branch 3 times, most recently from cbd77fd to 20adbbf Compare June 1, 2020 23:18

holgerfriedrich mentioned this pull request Jun 2, 2020

New image #946

Closed

13 tasks

mstormi reviewed Jun 2, 2020

View reviewed changes

build-image/openhabian.pi-raspbian64.conf Outdated Show resolved Hide resolved

holgerfriedrich force-pushed the pr-raspios branch 9 times, most recently from 4a8cebe to 339aac3 Compare June 7, 2020 07:00

holgerfriedrich force-pushed the pr-raspios branch from 339aac3 to 76d0f74 Compare June 8, 2020 19:57

This was referenced Jun 10, 2020

WiFi gone after HW upgrade RPi3 -> RPi4 #912

Closed

Begin Standardization and Rewrite #938

Merged

holgerfriedrich force-pushed the pr-raspios branch from 76d0f74 to 25037de Compare June 11, 2020 06:14

holgerfriedrich force-pushed the pr-raspios branch from 25037de to 4814fb0 Compare June 17, 2020 19:36

Introduction of Raspberry Pi OS

0a31499

* Replace raspbian by raspios due to renaming of the OS * (beta) Added initial support for building 64-bit images * Set default runlevel to multi-user.target, disable auto-login * Update documentation Signed-off-by: Holger Friedrich <[email protected]>

holgerfriedrich force-pushed the pr-raspios branch from 4814fb0 to 0a31499 Compare June 17, 2020 20:00

holgerfriedrich marked this pull request as ready for review June 17, 2020 20:35

holgerfriedrich requested a review from mstormi June 17, 2020 20:35

ecdye approved these changes Jun 17, 2020

View reviewed changes

mstormi approved these changes Jun 17, 2020

View reviewed changes

mstormi merged commit a80f0e4 into openhab:master Jun 18, 2020

holgerfriedrich deleted the pr-raspios branch August 11, 2020 16:41

Introduction of Raspberry Pi OS #931

Introduction of Raspberry Pi OS #931

Conversation

holgerfriedrich commented May 29, 2020 • edited by mstormi Loading

ecdye commented May 31, 2020

mstormi commented Jun 7, 2020 • edited Loading

holgerfriedrich commented Jun 7, 2020

mstormi commented Jun 7, 2020 • edited Loading

mstormi commented Jun 8, 2020

mstormi commented Jun 8, 2020

holgerfriedrich commented Jun 8, 2020 • edited Loading

holgerfriedrich commented Jun 8, 2020

mstormi commented Jun 8, 2020 • edited Loading

mstormi commented Jun 9, 2020 • edited Loading

mstormi commented Jun 9, 2020 • edited Loading

mstormi commented Jun 9, 2020 • edited Loading

holgerfriedrich commented Jun 9, 2020

mstormi commented Jun 9, 2020

holgerfriedrich commented Jun 10, 2020

mstormi commented Jun 10, 2020 • edited Loading

mstormi commented Jun 10, 2020 • edited Loading

mstormi commented Jun 10, 2020 • edited Loading

mstormi commented Jun 10, 2020 • edited Loading

mstormi commented Jun 10, 2020

holgerfriedrich commented Jun 10, 2020

mstormi commented Jun 10, 2020

mstormi commented Jun 11, 2020

mstormi commented Jun 11, 2020

mstormi commented Jun 13, 2020

holgerfriedrich commented Jun 17, 2020

holgerfriedrich commented Jun 17, 2020 • edited Loading

ecdye left a comment

Choose a reason for hiding this comment

ecdye Jun 17, 2020

Choose a reason for hiding this comment

holgerfriedrich Jun 17, 2020

Choose a reason for hiding this comment

ecdye Jun 17, 2020

Choose a reason for hiding this comment

holgerfriedrich Jun 17, 2020

Choose a reason for hiding this comment

mstormi commented Jun 17, 2020

holgerfriedrich commented Jun 17, 2020

mstormi left a comment

Choose a reason for hiding this comment

mstormi Jun 17, 2020

Choose a reason for hiding this comment

holgerfriedrich commented May 29, 2020 •

edited by mstormi

Loading

mstormi commented Jun 7, 2020 •

edited

Loading

mstormi commented Jun 7, 2020 •

edited

Loading

holgerfriedrich commented Jun 8, 2020 •

edited

Loading

mstormi commented Jun 8, 2020 •

edited

Loading

mstormi commented Jun 9, 2020 •

edited

Loading

mstormi commented Jun 9, 2020 •

edited

Loading

mstormi commented Jun 9, 2020 •

edited

Loading

mstormi commented Jun 10, 2020 •

edited

Loading

mstormi commented Jun 10, 2020 •

edited

Loading

mstormi commented Jun 10, 2020 •

edited

Loading

mstormi commented Jun 10, 2020 •

edited

Loading

holgerfriedrich commented Jun 17, 2020 •

edited

Loading