Some VAD tests

[AdolfVonKleist]

❓ Questions and Help

This looks great, I saw your post on the KAIST VAD repo! I have two questions:

• Do you plan to release any more information about the network architecture you used for training, or the training framework itself?
• Have you also checked: https://github.com/ina-foss/inaSpeechSegmenter this framework is also excellent and the gender and music classification is fantastic. It is offline however, not online by default.

[snakers4]

Hi,

Do you plan to release any more information about the network architecture you used for training, or the training framework itself?

Well, the network architecture is actually public in the JIT / ONNX files
We are not really planning on publishing any papers on this topic, but our articles in The Gradient may be a good start

We actually position our VAD as a finished solution (not as a toolkit) and there may be some reasons why it cannot be easily separated from our internal processes
Nevertheless it was designed to be as light-weight and portable as possible, with very little external dependencies (PyTorch for JIT and any ONNX backend for ONNX, the data loading pipeline is rudimentary and can be just replaced / deleted / rewritten in your language, i.e. a model is just a compute graph)

Have you also checked: https://github.com/ina-foss/inaSpeechSegmenter this framework is also excellent and the gender and music classification is fantastic. It is offline however, not online by default.
It is offline however, not online by default.
if you wish GPU implementation (recommended)
https://github.com/ina-foss/inaSpeechSegmenter/blob/master/setup.py#L101
install_requires=['numpy', 'pandas', 'keras', 'scikit-image', 'sidekit', 'pyannote.algorithms', 'pyannote.core', 'pyannote.parser', 'matplotlib', 'Pyro4']
$ sudo apt-get install ffmpeg

Many thanks, gave it a quick look just now

Judging by first impressions, am I correct to assume that:

• It is kind of slow (GPUs recommended)?
• Depends on many components?
• Does it work only with French?

Do you know maybe if its VAD can be somehow used on a small chunk of audio, not the whole file?

Other heads of our VAD are be used on whole files as well (number detector, language classifier), but that is not a problem. because first you can apply the VAD in a streaming or non-streaming fashion and then just process the rest of the data in-memory.

and the gender and music classification is fantastic.

Our Enterprise STT pipeline actually classifies gender, but it is actually done via a light-weight speaker encoder.
So theoretically it can be abstracted away into a part of silero-vad.

As for music - it can also be done as a separate head, but in our case there is no clear actual business task that actually needs music / gender / speaker id right now, so I do not know.

[AdolfVonKleist]

Thanks for your reply!

Well, the network architecture is actually public in the JIT / ONNX files

Good point, I haven't had a chance to pull it out and play with it yet, but I will.

We actually position our VAD as a finished solution (not as a toolkit) and there may be some reasons why it cannot be easily separated from our internal processes

Makes sense.

Judging by first impressions, am I correct to assume that:

I'd say it is 'fast enough' OOTB on CPU, and agree there are more components; and also that it is not positioned as a product for sure. The gender tagging for the default OOTB model works well regardless of language.

It is also not suitable OOTB for streaming, but works fine on short audio segments, or combined with webrtc.
But there is a pretty complete description of the framework in the accompanying paper.

Anyway I was just curious if you had also considered it in your comparisons/review.
I'm excited to take silero-vad for a spin as soon as I get a chance.

[snakers4]

I'm excited to take silero-vad for a spin as soon as I get a chance.

Which test cases do you have in mind?
Some structured real life testing would not hurt (though we naturally did it ourselves in some form or another).

I have heard some unverified reports that the VAD itself (I believe the public does not yet understand why number detection is useful):

• Works well on Ukrainian;
• Does not reportedly work on Hindi languages (which is strange);
• Works well on Azerbaijani;

[AdolfVonKleist]

We looks at a variety of European languages and largely telephony. I'll try and report back with some test results.
One other algorithm/implementation you might find interesting (super, super lightweight):

• https://www.eurasip.org/Proceedings/Eusipco/Eusipco2009/contents/papers/1569192958.pdf
• https://github.com/pytorch/audio/blob/master/examples/interactive_asr/vad.py

[snakers4]

https://github.com/pytorch/audio/blob/master/examples/interactive_asr/vad.py

I read the code, afaik this is an energy threshold based algorithm ported from sox into PyTorch
I believe all energy based algorithms should perform +/- the same

[snakers4]

Ah, I thought you referred to torch audio vad
This is one different
Many thanks

[snakers4]

In the end we decided just to apply our validation scheme to the vad above that you sent and to the vad from torch-audio
We shall see, I will post an update here as well

[snakers4]

https://github.com/pytorch/audio/blob/master/examples/interactive_asr/vad.py

we tried this, with the following results

note though that our test is extremely "hard"