bcm2835-audio HDMI device is not present on Pi 4 running with KMS and VC4 HDMI devices disabled

[KonstaT]

I'm experiencing a similar issue to #4634 that wasn't fixed with 214bf6b.

When running on Pi 4 with KMS and disabling VC4 HDMI devices (i.e. setting dtoverlay=vc4-kms-v3d,noaudio in /boot/config.txt) bcm2835-audio HDMI device is not present as it should be.

Running 5.10.74 kernel with latest firmware (https://github.com/raspberrypi/firmware/tree/d2ab264b688a223e23029ffbba86820c8c6cb329) I get:

[    3.899252] ALSA device list:
[    3.902277]   #0: bcm2835 Headphones

Same kernel but downgrade the firmware to https://github.com/raspberrypi/firmware/tree/d06e0ef85c22825a4491497a1d2a4787471f011b I get:

[   10.422612] ALSA device list:
[   10.425629]   #0: bcm2835 HDMI 1
[   10.428908]   #1: bcm2835 Headphones

This would be the expected behavior. This doesn't work with any newer firmware (bisected this to something that changed between https://github.com/raspberrypi/firmware/tree/d06e0ef85c22825a4491497a1d2a4787471f011b and https://github.com/raspberrypi/firmware/tree/19272ccd69049aaf42c78a235a0bf37dbabd5ea7)

[6by9]

When running vc4-kms-v3d ALL configuration of the HDMI block is done by the kernel.
HDMI audio is inserted into the blanking periods of the video signal, and is therefore dependent on the video mode in use. It also requires signalling in the infoframes as to exactly what audio format is being sent.

bcm2835-audio uses the firmware to handle the audio routing. With vc4-kms-v3d and the kernel in charge of the HDMI block, mode selection, and infoframes, it is not possible for the firmware to correctly insert audio, therefore it can't be supported.
The behaviour under 5.10.74 would therefore be correct - you've asked for no audio, and you've got exactly that.

[pelwell]

@mripard Interestingly, I get no picture when the noaudio parameter is used on Pi 4 with top-of-tree kernel (f814bfc):

dtoverlay=vc4-kms-v3d,noaudio

[   27.991647] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out
[   38.869297] [drm:drm_atomic_helper_wait_for_dependencies [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out
[   49.108196] [drm:drm_atomic_helper_wait_for_flip_done [drm_kms_helper]] *ERROR* [CRTC:76:crtc-3] flip_done timed out

[KonstaT]

@6by9
Any explanation why this worked before?
It's also not what I'd expect reading https://github.com/raspberrypi/linux/blob/rpi-5.10.y/arch/arm/boot/dts/overlays/vc4-kms-v3d-pi4-overlay.dts#L183 and 5ad79a1.

@pelwell
I can reproduce that maybe once in ~20 reboots. Seems to be pretty random. This is running Android on Pi 4.

[6by9]

It may work up until the point the resolution is changed significantly from that which the firmware chose. At that point the configuration for when audio packets are to be inserted into the stream will go awry.

I'd say the overlay is wrong and it should leave brcm,disable-hdmi; set should vc4-kms-v3d be loaded at all.

[pelwell]

By default the Pi 4 vc4-kms-v3d overlay disables the old firmware-based audio driver. When the noaudio parameter was added it was made to re-enable (or rather not disable) the firmware driver. Leaving it permanently disabled allows my display to work - I think this is a change that needs to be made.

[pelwell]

See #4652.

[pelwell]

It's probably not the outcome you were expecting, but with #4652 merged this can be closed.

[KonstaT]

IMO #4652 and 5ad79a1 is not entirely accurate either because the firmware does something (that I have no way of knowing) and 'bcm2835 HDMI 1' never gets enabled regardless if you do something on kernel side with brcm,disable-hdmi or not. So, it seems this was a won't fix. :(

I guess if I ever want to switch over to 5.10 kernel on Android I need to implement some changes on the Android audio HAL to use the vc4-hdmi-0/1 devices. Just to clarify that these actually work with KMS? And can you point me to some documentation how these are different from the ALSA devices bcm2835-audio (or various audio DACs) create?

[6by9]

If it used to work, then that was more luck than anything else. Two independent processors trying to control HDMI and HDMI audio is going to fail in some conditions, so not something that you work around.

rpi-5.10.y supports vc4-fkms-v3d where the HDMI display pipeline remains under the control of the firmware, and therefore using snd_bcm2835 for HDMI audio is valid.

Yes the audio support from vc4-kms-v3d does work, BUT it requires userspace to provide IEC958 formatted data. ALSA can do this for you, but you need the correct config file.

[KonstaT]

Well, it works fine on 5.4 kernel and up to this point with 5.10 as well.

FKMS vs. KMS is another situation that's lacking documentation. It's not possible to use FKMS with Android graphics HALs.

Thanks, I'll look into if I can configure the audio to use IEC958 formatted data on Android.

And another somewhat unrelated question now that I have you attention. There soon could be a codec component for Android to use V4L2 API for hardware video decoding/encoding. To clarify, H.264 decoder on Pi 4 is stateful and H.265 (HEVC) is stateless, right? Does either of these depend on the firmware (i.e. use of FKMS) or do these also work with KMS?

[pelwell]

Unrelated questions tend to get missed or ignored - open a new issue.

[6by9]

Well, it works fine on 5.4 kernel and up to this point with 5.10 as well.

Sorry, the action taken is the correct action.
We are learning all the time during development as well. With things like HDMI you build up from the simple things (ie video), then add audio, infoframes, HDR, CEC, etc. During that development process there may be times where things are incorrect, and those get rectified later (now in the case of this HDMI audio issue).

FKMS vs. KMS is another situation that's lacking documentation. It's not possible to use FKMS with Android graphics HALs.

It isn't? Why not? They both expose exactly the same V3D block, and the DRM devices have almost exactly the same capabilities. A question for the forums though, not here.

Thanks, I'll look into if I can configure the audio to use IEC958 formatted data on Android.

The required ALSA conf file ships with Raspberry Pi OS. If Android doesn't use ALSA then you'll have to deal with it within Android.

And another somewhat unrelated question now that I have you attention. There soon could be a codec component for Android to use V4L2 API for hardware video decoding/encoding. To clarify, H.264 decoder on Pi 4 is stateful and H.265 (HEVC) is stateless, right? Does either of these depend on the firmware (i.e. use of FKMS) or do these also work with KMS?

H264 stateful uses the firmware. HEVC stateless is purely kernel.
Sorry, we don't support Android, and last time I looked the codec APIs were all based around OpenMAX. Mapping that onto the stateful API won't be too hard, but mapping onto the stateless will be quite a job.

[KonstaT]

Thank you for taking the time to reply. I'll open a topic in 'Advanced users' section in the forum if I have further questions as I don't expect you to follow the Android section.

Android now has https://android.googlesource.com/platform/external/v4l2_codec2/ and AOSP reference board developers are working on this at least on Amlogic and Qualcomm hardware. It only supports stateful V4L2 API. And since H.264 requires FKMS this leaves me no hw decoders to use.

[6by9]

It only supports stateful V4L2 API. And since H.264 requires FKMS this leaves me no hw decoders to use.

No it doesn't. Stateful H264 decode works fine under vc4-kms-v3d.

[solskogen]

Could this be related to raspberrypi/Raspberry-Pi-OS-64bit#157 ?

[pelwell]

No unless you were using "dtoverlay=vc4-kms-v3d,noaudio".