Brunch Rammus: Upgrade to ChromeOS 101 Stuck on Logo

[darethehair]

Hello! Today I tried to do a simple ChromeOS upgrade to 101 (?) on one of our re-purposed Asus CN60 Chromeboxes, and it no longer fully boots -- stuck on new 'dark' ChromeOS logo! Previous versions worked well for over a year.

[sebanc]

Did you update brunch at the same time or was it only chromeos ? (no need to in principle, it is just to know as I have no idea why this happened for now)

Stuck on the boot logo usually means that either compatible graphics were not found (which would be quite strange, maybe a bad commit in chromiumos mesa) or it can also mean that it booted but that the main display is on a different screen.
Would the latter be possible ?

[reicrnl]

I have the same issue. Also using Asus CN60 and its stucked at the Chrome OS logo after updating ChromeOS.

[darethehair]

Did you update brunch at the same time or was it only chromeos ? (no need to in principle, it is just to know as I have no idea why this happened for now)

No, the version of Brunch was the same -- just the upgrade to ChromeOS. I confirmed this by creating a new bootable version on a USB stick -- composed of the R100 version (April 2nd) Brunch and the new 101 version of ChromeOS -- and got the same black-background ChromeOS logo freeze.

Stuck on the boot logo usually means that either compatible graphics were not found (which would be quite strange, maybe a bad commit in chromiumos mesa) or it can also mean that it booted but that the main display is on a different screen. Would the latter be possible ?

In order to check if a dual-display has inadvertently been set, it will take me more time to test (need to find my display port cable).

In desperation to returning this Chromebox back to a service state (and a local Thrift store) I will probably temporarilty try ChromeOS Flex.

[sebanc]

I think I found the source of the issue:
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/d6224e0a6a87f531651472ba6d27154f7fe4b8e4

They migrated rammus from i965 to iHD mesa driver. Unfortunately, it seems iHD driver is incompatible with your gpu for some reason. At some point this bug might be fixed by Google but it is hard to say...

For now it seems skl devices such as "cave" board are still using the i965 driver so you could switch to this recovery. Otherwise, indeed ChromeOS Flex can be a solution.

[darethehair]

I think I found the source of the issue: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/d6224e0a6a87f531651472ba6d27154f7fe4b8e4

They migrated rammus from i965 to iHD mesa driver. Unfortunately, it seems iHD driver is incompatible with your gpu for some reason. At some point this bug might be fixed by Google but it is hard to say...

For now it seems skl devices such as "cave" board are still using the i965 driver so you could switch to this recovery. Otherwise, indeed ChromeOS Flex can be a solution.

I'll certainly give it a try. BTW, I was able to find a displayport cable, but no 'alternate' display was being fed to -- which makes sense in this case.

[bybenjamin]

can you deal with the problem here please @sebanc
#1544

[darethehair]

FYI: Unfortunately, the 'cave' version of ChromeOS appears to have the same problem as the 'rammus' one did i.e. frozen on the ChromeOS boot logo :(

I have successfully installed Flex, though I very much wish to be able to return to Brunch in the future if possible -- so get the added benefit of Android apps.

[Earlybirdly]

Updated my fine working r100 stable brunchbook (hp Chromebook 14-q030sg) to rammus 101 only and got permanently stuck at the new chromeOS logo as well. This seems to be a serious issue. Is there a solution in sight?

Luckily TTY was already available at this state so that by chance I was able to rollback to rammus 100 without losing anything.

[sebanc]

I have been trying to figure out this issue but for now I don't really understand it... As "cave" recovery does not boot, the issue is not related to the mesa driver change, however I still think this issue is related to mesa but only found this suspicious commit in R101 and I have no idea how it could break things on your chromebooks:
https://chromium.googlesource.com/chromiumos/overlays/chromiumos-overlay/+/dc32a5db762001772f108907a113039347819b45%5E%21/#F7

In case it would not be related to mesa, I had a look at the boards configuration changes / rootfs changes but did not notice anything specific which might explain this.

What could be interesting would be to boot r101 in verbose mode with the kernel parameter "module_blacklist=i915" to see if there is anything specific that shows up in the log. If someone can provide a photo of the console messages, it could help identifying the source of this issue.

[darethehair]

What could be interesting would be to boot r101 in verbose mode with the kernel parameter "module_blacklist=i915" to see if there is anything specific that shows up in the log. If someone can provide a photo of the console messages, it could help identifying the source of this issue.

I don't think (?) that the generated GRUB2 entries during a Brunch install include a menuentry for verbose/debug mode anymore, do they? I just see a "--class" for "brunch" and "brunch-settings". If I knew what menuentry to add, I could give your suggestion a try.

[sebanc]

You can enable verbose mode in the settings menu (just before the bootsplash selection if I remember correctly).

Othewise you can press "e" when you see the grub menu and change:
if [ -z $verbose ] -o [ $verbose -eq 0 ]; then
to
if [ ! -z $verbose ]; then

[darethehair]

When I create a bootable Brunch USB stick (rammus image) and use the 'Settings' menu to add the blacklist entry for i915, and debugging activated, I get screens like this (first shows the strange 'headless' messages, second shows the repeating final messages):

What is strange (and it might be my fault somehow) is that when I try my preferred method of booting from GRUB2 menuentries, I get a message to the effect "the boot partition was not found, falling back to shell..."

Update: I think that Rammus 101 does eventually leave the boot logo screen (?) but starts to run into other problems (e.g. network issues, frozen/missing mouse) eventually causing non-operation.

[sebanc]

Thanks for the verbose mode photos !

Both screens show usual messages for devices with unsupported GPUs (which is normal in this case as we blacklisted the i915 driver). However, It confirms that there are no unexpected error messages (notably no "invalid opcodes" messages or such) which could have lead to a different theory.

Even if rammus does eventually leave the boot logo (I guess after a very long time), the other issues are most likely caused by the underlying graphics issue preventing other kernel modules from actually loading.

Looking at the chromeos subreddit, I noticed a few threads which look a bit similar so there are chances that Google might have to fix it, notably this one:
https://www.reddit.com/r/chromeos/comments/uxhb3d/lenovo_100e_chromebook_2nd_gen_v101_update_failing/

[bybenjamin]

I had the same problem. Try it with kernel 5.15

[sebanc]

Btw, regarding "what is strange (and it might be my fault somehow) is that when I try my preferred method of booting from GRUB2 menuentries, I get a message to the effect "the boot partition was not found, falling back to shell...".

This is generally a grub config issue, if you post your grub config and the format of the partition on which you installed the image I might be able to help.

[bybenjamin]

Hi, it won't go into sleep mode. Can you help. I leave the dmesg and group outputs.
dmesg.txt
grub.cfg.txt

[darethehair]

Btw, regarding "what is strange (and it might be my fault somehow) is that when I try my preferred method of booting from GRUB2 menuentries, I get a message to the effect "the boot partition was not found, falling back to shell...".

This is generally a grub config issue, if you post your grub config and the format of the partition on which you installed the image I might be able to help.

Thanks! I have used the GRUB2 technique for a long time now, so I am confused as to what may be going wrong (though the 'new style' of suggested generated GRUB2 code is a bit overwhelming compared to the original 'short' style). In this case, all I changed was the name/location 'img_path' variable to the location of the IMG file on my USB stick (or will be the HD device):

menuentry "ChromeOS" --class "brunch" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	if [ -z $verbose ] -o [ $verbose -eq 1 ]; then
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path \
			console= vt.global_cursor_default=0 brunch_bootsplash=$brunch_bootsplash quiet
	else
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	fi
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

menuentry "ChromeOS (settings)" --class "brunch-settings" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	linux (loop,7)/kernel boot=local noresume noswap loglevel=7 options= chromeos_bootsplash= edit_brunch_config=1 \
		cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

[darethehair]

I had the same problem. Try it with kernel 5.15

Sadly, for me, on the ASUS CN60, changing the kernel to 5.15 made no difference :(

[sebanc]

Btw, regarding "what is strange (and it might be my fault somehow) is that when I try my preferred method of booting from GRUB2 menuentries, I get a message to the effect "the boot partition was not found, falling back to shell...".
This is generally a grub config issue, if you post your grub config and the format of the partition on which you installed the image I might be able to help.

Thanks! I have used the GRUB2 technique for a long time now, so I am confused as to what may be going wrong (though the 'new style' of suggested generated GRUB2 code is a bit overwhelming compared to the original 'short' style). In this case, all I changed was the name/location 'img_path' variable to the location of the IMG file on my USB stick (or will be the HD device):

menuentry "ChromeOS" --class "brunch" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	if [ -z $verbose ] -o [ $verbose -eq 1 ]; then
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path \
			console= vt.global_cursor_default=0 brunch_bootsplash=$brunch_bootsplash quiet
	else
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	fi
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

menuentry "ChromeOS (settings)" --class "brunch-settings" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	linux (loop,7)/kernel boot=local noresume noswap loglevel=7 options= chromeos_bootsplash= edit_brunch_config=1 \
		cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

The config looks correct, could you verify if the uuid is correct by running "sudo blkid" on the partition where the brunch image is installed and looking at the "PARTUUID" value ?

[darethehair]

Btw, regarding "what is strange (and it might be my fault somehow) is that when I try my preferred method of booting from GRUB2 menuentries, I get a message to the effect "the boot partition was not found, falling back to shell...".
This is generally a grub config issue, if you post your grub config and the format of the partition on which you installed the image I might be able to help.

Thanks! I have used the GRUB2 technique for a long time now, so I am confused as to what may be going wrong (though the 'new style' of suggested generated GRUB2 code is a bit overwhelming compared to the original 'short' style). In this case, all I changed was the name/location 'img_path' variable to the location of the IMG file on my USB stick (or will be the HD device):

menuentry "ChromeOS" --class "brunch" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	if [ -z $verbose ] -o [ $verbose -eq 1 ]; then
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path \
			console= vt.global_cursor_default=0 brunch_bootsplash=$brunch_bootsplash quiet
	else
		linux (loop,7)$kernel boot=local noresume noswap loglevel=7 options=$options chromeos_bootsplash=$chromeos_bootsplash $cmdline_params \
			cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	fi
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

menuentry "ChromeOS (settings)" --class "brunch-settings" {
	img_path=/boot/isos/chromeos_rammus.img
	img_uuid=7bf33843-9e9a-48be-a30e-36e64be6e9e4
	search --no-floppy --set=root --file $img_path
	loopback loop $img_path
	source (loop,12)/efi/boot/settings.cfg
	linux (loop,7)/kernel boot=local noresume noswap loglevel=7 options= chromeos_bootsplash= edit_brunch_config=1 \
		cros_secure cros_debug img_uuid=$img_uuid img_path=$img_path
	initrd (loop,7)/lib/firmware/amd-ucode.img (loop,7)/lib/firmware/intel-ucode.img (loop,7)/initramfs.img
}

The config looks correct, could you verify if the uuid is correct by running "sudo blkid" on the partition where the brunch image is installed and looking at the "PARTUUID" value ?

Ah! I think I might know what is going on here:

1. The partition where the Brunch IMG was initially created:

/dev/sda3: LABEL="DATA" UUID="bc6d4db4-f8c3-4f16-9009-c21ca392d1db" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="DATA" PARTUUID="7bf33843-9e9a-48be-a30e-36e64be6e9e4"

2. The partition on my USB stick where I copied the IMG in order to boot Brunch on other machines:

/dev/sdb3: LABEL="ISO" UUID="53ab46bc-4188-433a-9905-4af2e8a56f91" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="ISO" PARTUUID="401e6cfc-7d4f-4435-b434-731757444bd2"

The older method (?) did not utilize the UUID in order to verify (?) the Brunch image. Since I don't want to 'create' the Brunch image on my USB stick, I guess I need to change the 'img_uuid' to match the new destination (USB stick) that I copy it to? If the 'img_path' already states where the image is stored, is the 'img_uuid' really needed?

[sebanc]

Yes you need to change the uuid in that case.

Previously brunch used "img_part=/dev/xxx" which has been replaced by uuid as windows creates new partitions after some updates which change the "/dev/xxx" value whereas uuid remains unchanged.

The thing is Grub is able to loop through devices and find the file /boot/isos/chromeos_rammus.img but I still need to send the information the initramfs which for many reasons could not do the same efficiently.

At some point I found a function in grub which was able to identify the partuuid and send it to the initramfs automatically but turns out ubuntu and fedora grub did not implement that function so I had to revert this.

[kk-junk]

My C740 is stuck after upgraded to 101. How can I rollback to 100? Is it possible to do it without powerwash or re-image?

[Earlybirdly]

Luckily here I was able to rollback without a powerwash, @kk-junk. The easiest way should be using the TTY-shell, which should already be present when stuck on the logo screen:

• press ctrl + alt + F2 (->) (a shell with US-keyboard layout opens)
• sudo chromeos-update -r ~/Downloads/recovery.bin (manual update routine, replace "~/Downloads/recovery bin" with the name and location of your actual Chrome 100 recovery file)
• restart

In case, this doesn't work, I was also successful by rewriting just the necessary partitions from the original images manually using dd in a shell of a Live-USB and then letting brunch rebuild itself. Would be great, if @sebanc could provide a version of his chromeos-update script for this use case (updating a different drive with a stuck system). Actually even greater, if we could get v101 to start properly. Is there any solution in sight?

All this is probably limited to the case, that a new ChromeOS version hasn't yet upgraded any user data to a possible new version format when it gets stuck. Here it seems to work.

[kk-junk]

Thanks @Earlybirdly, I can restore the system to v100 using your instruction. Now, I have v101 on USB drive and is testing it again. This time it boot pass the boot chromos screen and initialized the wifi network. However, it does not load the add user screen. I can start the guest mode, but it does not load anything. Just timeout on any network request..
Not sure what is going on. Hope @sebanc can find a solution for those old Chomebook.

KK

[darethehair]

UPDATE: Today I tried 'Brunch r103 stable 20220721' on my Acer C740 Chromebook (was running Brunch OK but then this issue began) in order to see if the situation had improved. Yes, it had, but not perfectly i.e. using this new version of Brunch with the latest Rammus recovery image (chromeos_14816.99.0_rammus_recovery_stable-channel_mp-v2.bin) allows me to successfully boot and progress to the point of 'Who would you like to add to this Chromebook?' question -- at which time I pick myself ('You') and the I receive a spinning wheel for a few minutes, and then it returns back to this question.

So close to success! Wish that it could complete :(

EDIT: Looks like the same as #1563

[sebanc]

@darethehair Could you launch chromeos and go through setup until the spinning wheel appears, then press CTRL+ALT+F2, login as "root", run "dmesg" and post a photo here ?

[darethehair]

Sebanc: Here you go -- starting just before the bad messages start appearing (hopefully this will paste properly)...

[sebanc]

Thanks ! I think the tpm issues are normal on a chromebook with a custom firmware but to verify this, when you have the time, could you post the output of "cat /var/log/messages | grep -i tpm" ?

Also, I see that you have 2 veth adapters, are you connected to internet via wifi or ethernet ?

[darethehair]

I am not sure that the two screenshots are sufficient for showing all the errors, so check out this dropbox file containing a full true listing of dmesg that I captured onto a USB stick:

https://www.dropbox.com/s/1amdd86so9i5ywa/brunch_dmesg.txt?dl=0

Maybe I should start it over, since it doesn't really start from the beginning?

[darethehair]

The /var/log/messages grep that you were asking for:

https://www.dropbox.com/s/evk3uw2aovz8k40/varlogmessages_tpm.txt?dl=0

P.S. I am connected just via wifi in this case.

[sebanc]

Could you try to boot with "module_blacklist=tpm_tis" added on the kernel command line ?

[darethehair]

Here is a dmesg that includes the very start, and progresses through multiple crash dumps:

https://www.dropbox.com/s/py7resaxplsixkd/brunch_dmesg2.txt?dl=0

[darethehair]

Dmesg with 'tpm_tis' blacklisted:

https://www.dropbox.com/s/u2py2lpeggh45fz/brunch_dmesg3.txt?dl=0

[sebanc]

I really don't see what is causing this, if you have not already tried, could you try kernel 5.10 ?

[darethehair]

Sorry, please refresh my memory -- how do I control which kernel I am using while booting from a USB stick?

[sebanc]

Try to run "sudo edit-brunch-config" from tty2

[darethehair]

Tried kernel 5.10. Still had same external behavior, but dmesg was a lot cleaner:

https://www.dropbox.com/s/i2utdcmg418ruit/brunch_dmesg4_510.txt?dl=0

[sebanc]

Thanks for the new log !
Unfortunately there are no real issues there which makes this issue difficult to debug...

Could you try to get the "/var/log/messages" file with kernel 5.10 when you have the time ?

[darethehair]

/var/log/messages using kernel 5.10:

https://www.dropbox.com/s/4p3stdh8kmn354w/varlogmessages_510.txt?dl=0

Take note that there are multiple boot sessions in there. The relevant ones (for you) are probably the '2022-07-23T19:**' ones at the end.

[sebanc]

Thanks again for the new log. Unfortunately there is nothing that really catches my attention aside from the chromeos crashes that I cannot relate to something specifically.

Maybe you could try the below things:

• Remove /etc/chrome_dev.conf file and see if it helps
• Try switching to dev_channel, enable updates in the brunch config and run "sudo update_engine_client --channel=dev-channel --update"

I will keep thinking about it but for now I am a bit lost with this issue

[mikealejandro121109]

I had the same problem. Try it with kernel 5.15

Sadly, for me, on the ASUS CN60, changing the kernel to 5.15 made no difference :(

Neither did for me as well I'm still stucked here with the 118 rammus

[darethehair]

I had the same problem. Try it with kernel 5.15

Sadly, for me, on the ASUS CN60, changing the kernel to 5.15 made no difference :(

Neither did for me as well I'm still stucked here with the 118 rammus

I am surprised and curious how you were able to run all the way up to 118, but in any case I was forced to move to ChromeOS Flex on my Haswell-based Chromebooks/boxes, which is OK but I miss the Android app support.

[abbasabidi85]

EDIT: anyone facing dgpu issue should use any linux distro to live boot and run the command sudo lspci and note down their pcie address of dgpu it will be in this format XX:XX.X I was making the mistake and writing it down as XX:XX:X, for example mine was 01:00.0 after getting this I used it in kernel command line parameters as remove_dgpu=0000:01:00.0 and voila it worked.

I'm using rammus v121 on i5-7200U with AMD dgpu and it gets stuck on ChromeOS logo and then in a 5 to 10 minutes it goes to black but with ChromeOS slightly visible. I tried kernel command line parameters fbcon=map:0 , fbcon=map:1 , disable_dgpu , remove_dgpu and I also tried to access TTY using Ctrl + Alt + F2 but to no avail.

I also use opencore they have specific boot-args to disable dgpu, something like that would be very good.

I'm gonna now try rammus v10 😕