2-way audio does not work with certain Hikvision cameras

[pergolafabio]

Checklist:

• I updated to the latest version available
• I cleared the cache of my browser

Release with the issue:
Specific PR release: #1291
But it also happens with 5.2.0 and Frigate addon

Last working release (if known): Unknown
Browser and Operating System:
This happens on all browsers mac/chrome/companion app

Description of problem:

Using the config below, two way audio works, toggling the MIC button on/off again, always stops/starts the two way audio...
But the camera is also a doorbell, when a call is received, we need to first mute the call, by sending a service to it, to make the ringing sound stop, afterwards we press the mic button to start talking... so far so good.. Then i press the mic button to stop the audio, this also works... Next on the second call, we do the same thing, we send the service to the doorbell to stop the ringing, but the when we press the MIC button, the two way audio isnt initiated... this is my issue
What we need todo, is to refresh the page where the frigate card is located, after the refresh, when we then press again the MIC button, it seems to work....

I dont know why, or whats the cause, or why i need to refresh the page?

frigate card config:

        - type: custom:frigate-card
          cameras:
            - camera_entity: camera.deurbel
              live_provider: go2rtc
              go2rtc:
                url: https://user:[email protected]:8123
                stream: deurbel
                modes:
                  - webrtc
                  ......

go2rtc config:

streams:
  deurbel:
    - rtsp://user:[email protected]:554/Sms=24.unicast
    - isapi://user:[email protected]:80/

I also checked the go2rtc addon logs, no isses there... so not sure if its go2rtc or frigate card related, difficult to tell..
What is happening on a refresh of the view? whats different between a refresh of the view and toggling the mic button?

What i also see, when you do a refresh of the page, the mic button is not immediately visible, it takes a second, and then its there... Is there maybe a way to force that refresh? That could maybe solve my issue, alltough still dont know the root cause...

Edit:

My guess, go2rtc is using ISAPI to use two way audio on hikvision camera's, I think the audio session is not closed when pressing the mute button? When refreshing the page it starts a new two way audio session? The audio session is always open, regardless the mic button is muted or not??
That way, on an incoming call the audio session is closed, bur frigate thinks it's still open?

[Daz42]

Just to add to the above (I've been working with @pergolafabio, I have a Hikvision Doorbell and Internal Intercom. I am using the Frigate add-on, Frigate Integration and Hass card to use the camera, speaker and microphone from the Hikvision Doorbell. When using this all through the Frigate Hass card, it all initially works well and I get 2 way audio.

However, when I take the following steps things go wrong:

1. Checked that 2 way audio working
2. Press doorbell button
3. Pressed the HA entity indoor button for answer call (Hikvision Doorbell Entity)
4. Pressed the HA entity indoor button for hangup call (Hikvision Doorbell Entity)
5. 2 way audio not working anymore
6. Restarted Frigate / reload Hass card
7. 2 way audio back again

I'm happy to do any testing.

[pergolafabio]

a quick update, i tested the WebRTC card to rule out go2rtc, and that card works reliable
everytime i click the mute/unmute icon to switch streams, it works

is it maybe possible to compare the mic button from frigate to behave also like the webrtc card?

[dermotduffy]

My guess, go2rtc is using ISAPI to use two way audio on hikvision camera's, I think the audio session is not closed when pressing the mute button?

Correct.

When refreshing the page it starts a new two way audio session? The audio session is always open, regardless the mic button is muted or not??

Unless you use the always_connected option, the stream will be converted to a 2-way session the first time the microphone is unmuted. It will automatically disconnect (convert back to without 2-way audio) after disconnect_seconds have elapsed.

Next on the second call, we do the same thing, we send the service to the doorbell to stop the ringing, but the when we press the MIC button, the two way audio isnt initiated... this is my issue

So it all works fine, until to call an unrelated service and then the microphone unmute doesn't work anymore? How are you calling the unrelated service? Can you provide your full card config?

Thank you!

[pergolafabio]

Hey , thnx for feedback already, appreciate it! Below full config
As you can see, im using the dev card version, where i can point to camera without frigate addon/integration

As you can see, i have extra 2 buttons ( see also screenshot) a phone / hangup button
That phone buttons calls a script, so that when someone is actually pressing the doorbell button, i need to stop the ringing and afterwards inittiate the two way audio

But i also see this new PF : #1359
That means i can combine the answer + mic button into one, thnx already for that, will test that out later!

But anyway, back to issue, you are talking about always_connected , i tried it with true/false, but ddint make any difference... But i wasnt aware of the disconnect_seconds!! I was always testing making calls after each other, within the 60 seconds

What is the best approach for this? probably setting this back to "false" , so that means, on incoming call, if i wait longer then 60 seconds for next call , it should work? What happens if the stream is active? I mean if the lovelave card is active? Does it trigger the 60 sec disconnected? Or does it only disconnect if i browse away from the page?

That being said, maybe that new PR could be helpfull with the automation? Can i define for example, if press the hangup button or if i toggle again the mic button, that i force the disconnect?

But whats still strange, i dont know if the disonnect_seconds will solve it... When the issue occured, i could toggle the mic button as many times if i want, it never worked again sending two way audio, untill i did a refresh of the lovelace card...
Shouldnt toggling the mic button , disconnect any open audio sessions anway?

        - type: custom:frigate-card
          cameras:
            - camera_entity: camera.deurbel
              live_provider: go2rtc
              go2rtc:
                url: https://user:[email protected]:8123
                stream: deurbel
          menu:
            style: outside
            position: bottom
            buttons:
              microphone:
                enabled: true
                type: toggle
              screenshot:
                enabled: false
              download:
                enabled: false
              fullscreen:
                enabled: false
              snapshots:
                enabled: false
              timeline:
                enabled: false
              media_player:
                enabled: false
              clips:
                enabled: false
              live:
                enabled: false
              cameras:
                enabled: false
              frigate:
                enabled: false
              camera_ui:
                enabled: false
          live:
            microphone:
              always_connected: true
            auto_mute: []
            auto_unmute: []
            controls:
              builtin: true
              title:
                mode: none
            layout:
              fit: fill
          elements:
            - type: custom:frigate-card-menu-icon
              icon: mdi:volume-high
              tap_action:
                - action: custom:frigate-card-action
                  frigate_card_action: unmute
            - type: custom:frigate-card-menu-icon
              icon: mdi:volume-off
              tap_action:
                - action: custom:frigate-card-action
                  frigate_card_action: mute
            - type: custom:frigate-card-menu-icon
              icon: mdi:phone
              tap_action:
                - action: call-service
                  service: script.answer_call
                - action: custom:frigate-card-action
                  frigate_card_action: unmute
                - action: custom:frigate-card-action
                  frigate_card_action: microphone_unmute
            - type: custom:frigate-card-menu-icon
              icon: mdi:phone-hangup
              tap_action:
                - action: custom:frigate-card-action
                  frigate_card_action: microphone_mute
            - type: custom:frigate-card-menu-icon
              icon: mdi:door-open
              hold_action:
                - action: call-service
                  service: switch.turn_on
                  data:
                    entity_id: switch.ds_kd8003_door_relay_0
          dimensions:
            aspect_ratio_mode: static
            aspect_ratio: '16:9'

[pergolafabio]

For quick testing, i changed my card to this setting:

          live:
            microphone:
              always_connected: false
              disconnect_seconds: 15

Indeed after 15 sec, the mic is disconnected, i can turn on mic and speak again, so that setting is working

However , my issue is still present, if i have a real call coming in, i drop the doorbell sound, but then i enable mic, but i cant speak... if i then wait 15 sec, the mic is off, i turn it on again, i still cant speak

The only solution, is to browse away from the lovelace card or a a refresh of the page, then i can speak again

There must be something different in refreshing the page and the disconnect feature, i think the dsconnect feature doesnt disconnect enough? Whats the difference between your disconnect feature and refreshing the page?

[felipecrs]

@pergolafabio maybe you should try again with #1362.

[pergolafabio]

Worth a shot, I will try later indeed!

[pergolafabio]

Tried the new dev , but still doesnt work as expected for me :-)
Whenever the doorbell was ringing, i cant use two way audio anymore
Need to close companion app or do a hard refresh on the card to make it work again

[pergolafabio]

hey @dermotduffy , any thing i can test or do ? thnx

[dermotduffy]

hey @dermotduffy , any thing i can test or do ? thnx

Sorry, as I don't these cameras, I have no way to test and I'm not sure what is going on or what else to suggest I'm afraid. There are a few other 2-way audio related issues in the backlog that I will look into that could impact this, so I'll leave this issue open in case they help / in case someone else manages to make some progress to figure out what's going on.

[pergolafabio]

Ok, no problem
Maybe it's worth a look how the webrtc card works? Cause when I test that card pressing the mic/mute button every time works on that card..
Maybe they use a different approach to stop/start audio?

[dermotduffy]

Maybe it's worth a look how the webrtc card works? Cause when I test that card pressing the mic/mute button every time works on that card..

Sounds like a great plan. Thanks.

[pergolafabio]

Thnx!

If you want, I can give you access to my intercom, so you can test it remote

[pergolafabio]

hey @dermotduffy were you able to have a look at this issue again?

[pergolafabio]

is it maybe related to this issue?

#1389

maybe the connection is indeed still open, and only works again after a new hard refresh of the page?

[felipecrs]

Maybe this will fix this issue: AlexxIT/go2rtc#967

[pergolafabio]

Aha, that's interesting!! I keep an eye on it, that could be it indeed

[pergolafabio]

@felipecrs , thnx for pointing that issue, it fixes the issue, seems indeed related to go2rtc , and indeed a fix is underway
can close this one now...

thxn for all help

[felipecrs]

I would suggest you mention you tried the fix in go2rtc repo, so that the owner can be more confident to merge it.

[pergolafabio]

hmm, he didnt create an addon repo, not sure how i can test a specifc PR , since its not merged yet in alex master

[pergolafabio]

Hey @dermotduffy , i did some digging and traced it with wireshark
The difference with the WEBRTC card, toggling the mic button, initiates a new stream
With your card, toggling the MIC button just mutes/unmutes the stream

Is there a way/service to restart the stream with your card? That could solve the issue

thnx

[felipecrs]

@pergolafabio does not go2rtc 1.9.0 solves the issue?

[pergolafabio]

Yes and no, toggling the mic button on frigate, doesn't start a new stream session, I wiresharked this

Webrtc card does it, not frigate card

[pergolafabio]

Hey @dermotduffy

You said:
FYI: The Frigate card does restart the stream on the first press of the unmute button

Well, that's exactly the issue, that's why I only hear two way audio the first time I use the card :-)
Is there some service to force to restart the stream again when toggling the mic button again?

This is very specific for Hikvision, in normal circumstances , it's not an issue , I can toggle the mic button as many times I want, I always have two way audio.. but this device is also an intercom... So when someone is making an intercom event by ringing the device, the two way audio is initiated to another client... That's why the audio is closed on the frigate card.. in order to get audio back, I need to restart the stream, that will send an isapi command (close audio) to the camera, that command is now included in the go2rtc addon since 1.9.0

[f1d094]

Just to chime-in as a non-doorbell Hikvision camera user: It is important to me that functionality not change when "always-connected" is enabled. More specifically that the mute button does not close/restart the stream. We use this camera as door intercom and often have some back-and forth with whoever is there. Muting/Unmuting multiple times is typical.

Using AlexxxIT's WebRTC card restarts the stream on every mute/unmute. The Muting/Unmuting process is slow on the camera-end and this quickly breaks things. The ISAPI interface cannot keep up.

What I would propose is an alternate button for enable/disable mic (via stopping/starting ISAPI stream) as opposed to mute/unmute which simply mutes the client-side mic, which is instantaneous and expected behavior for any audio client/server setup.

[f1d094]

As additional thought: Being "always-connected" should really already be the default behavior. If there is a microphone configured, the expectation is that use of that microphone should be instantaneous. Tearing down/re-building the stream is going to cause a noticeable lag. This is particularly evident when using this remotely on a mobile device...it just doesn't work.

As long as there is an "always-connected" option, the solution will work for our use case, but it took a couple month's worth of headaches before I was lucky enough that @dermotduffy gave me a heads up in another thread. Anyone newly trying this out and lacking the skills to do tcpdumps and review them via wireshark will likely find the solution unworkable.

Food for thought.

[felipecrs]

As additional thought: Being "always-connected" should really already be the default behavior. If there is a microphone configured, the expectation is that use of that microphone should be instantaneous.

This triggers the browser to think the microphone is in use, and on Android this means also highlighting the status bar item.

For example, I only use microphone 0.01% of the times I open my dashboard, so IMO it should not be made the default.

Instead, maybe investing time into somehow making attaching the microphone to an existing stream session on the fly would be a better replacement for always_connected, without requiring a stream restart.

[f1d094]

maybe investing time into somehow making attaching the microphone to an existing stream session

@felipecrs: This is what @dermotduffy was discussing earlier. This is not possible. Nor is it correct from a development perspective. If the microphone is set to "enabled" in the config, it should be enabled. In this context that explicitly means "always-connected". Otherwise the "mute" button should be changed to "Disabled/Enabled". It is no longer a mute button.

If you personally only use the microphone rarely, then that by definition is an edge case and should carry the edge-case configuration requirement vs the expectation that if microphone is set to "enabled" that it is enabled and will remain enabled until explicitly set as "disabled", and the mute button actually means "mute" the same as how a mute button works on a phone, stereo, or any other audio device.

On your stereo, when you hit mute, the music keeps playing...but it just stops coming out of the speakers. When you hit mute on your phone, it does not disconnect the call...it simply prevents audio from being transmitted. The configuration you are recommending is that every time someone hits "mute" the phone hangs up and a new phone call needs to be made when the user wants to speak again. It might happen much faster here...especially if you are on the same network...but tearing down and rebuilding the connection on every press of the mute button just doesn't make sense.

With the current setup the expected behavior is obfuscated. The microphone is sluggish, like a child's walkie-talkie, and the connection becomes unusable regularly without any explanation...because the mute button is causing the connection to be torn-down/re-built on every press. Whereas if "always-connected" was the default, then you might have a microphone enabled on your desktop but you at least have an indicator of what is going on and can then look for the setting to change...and can then enable/disable the microphone entirely as you wish.

There is a value to have a new/special button to enable/disable the microphone entirely...but it should not be the mute button.

[f1d094]

I should also note that tear-down/re-connection also means the video channel. They are coupled.

In no context does anyone expect that hitting "mute" means restarting the video.

[felipecrs]

Thanks, I'm sold.

[Martinvdm]

I have the same camera. Didn’t try the frigate card yet but in the go2rtc webrtc card I have some issue also. In this card it is recommended to do a second stream with mic and a first stream without it if you want to enable/disable your mic.
while the first stream is just working fine, the second stream, with enabled audio is crashing the iOS app and reconnection the app completely. I don’t know why. I will try the frigate card soon. Is there any advise or latest example config how to start?

[felipecrs]

I am using a Hikvision doorbell exclusively through the Frigate Card for a month by now already.

The setup is pretty much the same I have in: https://github.com/felipecrs/dahua-vto-on-home-assistant

It works flawlessly.

[pergolafabio]

For me I still need to send the "close audio" command if I unmute the second time.... But the issue is only present on a real incoming call, to another indoor station, seems the audio channel is then probably occupied...

I still need somekind of hard refresh on the card, like the webrtc card does

[spn91]

I have the same issue with the frigate-card. First session is always working. If I mute the mic on the client with which the first session was started, the browser tab still shows that the mic is in use for a few minutes. As long as the mic status icon is shown on this client, I can't use the mic on another client.

If using the UI of frigate HA addon and triggering the mic there, I can see how the whole streams reloads (within a few ms) and really disconnects. Is there a possibility to implement such a feature, or is it maybe already implemented? Just an action which can be triggered when pressing a button, which reloads the stream and thus also ends the 2-way-audio session.

[felipecrs]

There has been a microphone_disconnect action you can use to disconnect the microphone. You can create a button to call it, like I did in felipecrs/dahua-vto-on-home-assistant.