feat: audio support

[varshith15]

This PR aims to add audio support to comfystream. It adds the following functionalities

1. Instead of invoking queue_prompt after each frame push, it's now run in a loop and queue connects stream track and comfyui workflow, audio and video get to server at different rates to invoking queue_prompt after pushing each frame isn't feasible

2. Can select 3 kinds of permutations with two tracks video and audio (video only, audio only, video + audio)

3. Ability to run multiple prompts concurrently -- the reason for this is we don't want one workflow(prompt) to affect another prompt (this occurs mostly due to buffering which is mostly needed in audio workflows), also if two workflows are not dependent, they should be run on separate threads for concurrency

Future Work:

1. Explore one-to-many queuing system as it is required in cases where multiple workflows might be consuming from the same queue

2. Perfect video, audio alignment when the gap between them is larger than the buffering capacity of webRTC

3. Have a data channel to send text or other datatypes (ASR etc) from server to UI

4. How to handle generative workflows -- cases where there is no input just output

[yondonfu]

@varshith15

Thanks for the PR! A couple requests:

1. Please provide a high level overview of the changes and expected behavior which will make review easier.
2. Please rebase off the latest main as resolving the conflicts will make review easier.

[varshith15]

@yondonfu it was supposed to be a draft PR (have to try to plug in the UI part, rebase changed a bunch of things), my bad

[UD1sto]

Hey @varshith15 ! I am very interested in contributing to your efforts for audio support. Would you like to discuss and coordinate?

[varshith15]

hey @UD1sto, yes ofcourse, please find me on discord, let's discuss on there

[eliteprox]

Thanks for this amazing feature @varshith15! I tested this successfully with an audio-only stream after making a few necessary changes. I re-based the branch and pushed it to audio_support_main for easier review, see https://github.com/yondonfu/comfystream/compare/audio_support_main?expand=1.

Please see the review comments below and rebase (or force push from 598ed4e), then re-request review when ready

[eliteprox]

The audio streaming works great, but while testing video, there was an async or race-condition type of issue with tensor_utils. Remarkably, the async worker queue creates a frame-skipping effect (which is desired). See attached stack trace.

message(13).txt

[eliteprox]

Thank you for your contribution @varshith15! I've fully tested this change with audio support. This change also implements frame skipping for video workflows. There is only one issue remaining related to simultaneous audio/video inference. Currently users may run an audio or video workflow, but not both at the same time. This is in scope to be resolved following the hiddenswitch version upgrade #37

[rickstaa]

@eliteprox Thanks a ton for reviewing and merging this! 🙌

Just a small request for next time—would it be possible to use a combined commit message when squashing? The individual commit history is already in the pull request, and having a single message would make it much easier to navigate, especially in the terminal.

Thanks again!

[eliteprox]

@eliteprox Thanks a ton for reviewing and merging this! 🙌

Just a small request for next time—would it be possible to use a combined commit message when squashing? The individual commit history is already in the pull request, and having a single message would make it much easier to navigate, especially in the terminal.

Thanks again!

Thank you for the feedback, this PR had several UI conflicts as it aged. I used a squash commit to re-base and squash the commits before squashing again to main. Apologies for the long commit message, I'll be mindful about that going forward.