Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prefetch code and packages for offline install #984

Closed
holgerfriedrich opened this issue Jun 18, 2020 · 23 comments · Fixed by #1121
Closed

Prefetch code and packages for offline install #984

holgerfriedrich opened this issue Jun 18, 2020 · 23 comments · Fixed by #1121
Labels
base system Related to the base system

Comments

@holgerfriedrich
Copy link
Member

holgerfriedrich commented Jun 18, 2020

Refs: #825

For an "offline install", i.e. first boot of the openHABian image without internet access, we need to cache some code and packages during creation of the image.
I.e. cache ZRAM code repos, cache JAVA, cache essential packages for building ZRAM and all the dependencies.

As this is done during image creation, it cannot use the built-in mechanisms of the target image (apt, ect).

install_zram_code $buildfolder/root/opt/zram
java_zulu_fetch $buildfolder/root
...

List what is required including dependencies, only if not included in the Raspberry Pi OS lite image (make, gcc, git is fine....).

  • libattr1-dev
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 18, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
@holgerfriedrich holgerfriedrich added the base system Related to the base system label Jun 18, 2020
@mstormi mstormi mentioned this issue Jun 18, 2020
13 tasks
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 18, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 18, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 18, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 18, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
@holgerfriedrich
Copy link
Member Author

@mstormi zram did not work, we have an apt install in install_zram_code

we need to download the deb files manually from correct respian repo and install them during first-boot :-/
This is why I did not like the idea of offline install in the first place.....

zulu java seems to work, azul has the same problem as zram but even worse....

@mstormi
Copy link
Contributor

mstormi commented Jun 18, 2020

we have an apt install in install_zram_code

which is just the libattr1-dev that you've asked for (what to install upfront)

zulu java seems to work, azul has the same problem as zram but even worse....

what do you mean ? it works or not ?

@mstormi mstormi closed this as completed Jun 18, 2020
@mstormi
Copy link
Contributor

mstormi commented Jun 18, 2020

wrong click

@mstormi mstormi reopened this Jun 18, 2020
@holgerfriedrich
Copy link
Member Author

@mstormi We have to distinguish between build system (the one which executes build.bash) and target image. When I call install_zram_code, the library gets installed on the build system.
I have not way of executing apt of the mounted target image during install.
A simple but ugly way is to wget the correct deb file in install_zram_code and install it during first-boot (I was thinking to create a cache folder and installing all packages during first boot...)

@mstormi
Copy link
Contributor

mstormi commented Jun 18, 2020

I think @ecdye worked around exactly that for Zulu, check his code

@ecdye
Copy link
Member

ecdye commented Jun 19, 2020

No, for Zulu we use the API so we then avoid APT altogether. Honestly, I don't think that an elegant solution for offline install is possible. We may be able to fail gracefully and leave the user with a partial system but not full one.

Zulu offline may not even truly work because the install still has apt packages that it requires as prerequisites.

@mstormi
Copy link
Contributor

mstormi commented Jun 19, 2020

Zulu offline may not even truly work because the install still has apt packages that it requires as prerequisites.

which ones? That's Holger's intention of this issue to collect these.

@mstormi
Copy link
Contributor

mstormi commented Jun 19, 2020

there's an 'offline' competitor now: https://ubuntu.com/appliance/openhab

@holgerfriedrich
Copy link
Member Author

there's an 'offline' competitor now: https://ubuntu.com/appliance/openhab

Do you know who created this?
Maybe time to start a discussion in maintainers section if we should refocus our efforts.

@mstormi
Copy link
Contributor

mstormi commented Jun 19, 2020

Do you know who created this?

No. Canonical I believe.

Maybe time to start a discussion in maintainers section if we should refocus our efforts.

already did: https://github.com/orgs/openhab/teams/openhabian-maintainers/discussions/17

@ecdye
Copy link
Member

ecdye commented Jun 19, 2020

which ones? That's Holger's intention of this issue to collect these.

The ones listed in java_zulu_prerequisite() being libc, libstdc++, zlib1g

@mstormi
Copy link
Contributor

mstormi commented Jun 23, 2020

Holger, any news ?
#986 now hopefully fixed.

@holgerfriedrich
Copy link
Member Author

@mstormi No, sorry. Not even a chance to follow everything you are doing in the meantime. Busy days, so I assume this will take a bit longer....
I have the a really bad feeling when I think about the dependencies of dependencies... We cannot clone the whole apt package repo.... Hm, why not ;-)

@ecdye
Copy link
Member

ecdye commented Jun 24, 2020

Maybe because we want the resulting image to be a reasonable size. 😉

@mstormi
Copy link
Contributor

mstormi commented Jun 24, 2020

It cannot be that many dependencies packages, essentially it's libs only. And why not clone them all?
Come on let's try. This is the only remaining issue to get the image done, isn't it ?

holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 28, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 28, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 28, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 29, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch
* use apt of target image to fetch deb packages

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 30, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch
* use apt of target image to fetch deb packages
* verify signature of download images using key from raspberrypi.org website
* first-boot to install cached packages

Signed-off-by: Holger Friedrich <[email protected]>
holgerfriedrich added a commit to holgerfriedrich/openhabian that referenced this issue Jun 30, 2020
* initial functionality added to build.bash
* directory prefix for java_zulu_fetch
* use apt of target image to fetch deb packages
* verify signature of download images using key from raspberrypi.org website
* first-boot to install cached packages

Signed-off-by: Holger Friedrich <[email protected]>
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
@ecdye ecdye added this to the v1.6 Hardware Image milestone Sep 21, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
ecdye added a commit that referenced this issue Sep 21, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 21, 2020
ecdye added a commit that referenced this issue Sep 22, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 22, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 23, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 23, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 23, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 23, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 24, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 24, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 24, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 24, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 25, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 25, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 25, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 25, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 25, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 27, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 28, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 28, 2020
ecdye added a commit to ecdye/openhabian that referenced this issue Sep 30, 2020
mstormi pushed a commit that referenced this issue Oct 1, 2020
@ecdye ecdye unpinned this issue Oct 1, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
base system Related to the base system
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants