Sometimes model do not respond until user speak some another sentence

[zelitors]

I'm using a custom instruction in the model instead of defaults instructions suggestions, and using the model in Brazil Portuguese language.

Sometimes the model freezer in a 'thinking' state and do not respond to user. After some seconds waiting for a response when user says 'keep going' or 'hello, you still there', the model response and keep the chat flow.

This issue is not persistent, in the most case this situation does not happen and the chat (user/model audios) flow and fluence occurs perfectly.

I double-checked my network stability, avoiding any input audio noising and OpenAI API quota to ensure that it is not a network, audio, or quota limit problem.

Follow an example video when the problem occurs:
https://github.com/user-attachments/assets/b1a72b96-feb5-4f7c-8bb6-6e5daef50293

There is anyone with a similar issue?
This is a LiveKit or OpenAI issue?

Regards,
Joselito Ribeiro

[davidzhao]

are you using OpenAI Realtime API? can you turn on detailed debug logs (--dev) and include the latest logs when the agent freezes?

[yousri-sellami]

I think I'm facing the same problem. I am using VoicePipelineAgent with Deepgram STT. It's a bit hard to reproduce. The easiest way I found to reproduce is to successively interrupt the agent until the issue occurs. After X interruptions, it always end up being stuck in thinking state.

As you can see in the logs, it stays more than 30s in thinking state. VAD and STT seem to be working fine, but the LLM generation is somehow blocked.