Better stop

[TheBest-Name]

There have been a lot of times where LVA I started talking and I kept wanting to say stop.

This is a two-part issue.

1. The stop word does not do as much as you would expect. I would have expected it to stop any music playing. It would also stop any alarms and announcements or anything really that's still going. It seems to stop some things, but not really everything.

2. When LVA doesn't catch what you're trying to say, it just repeats and repeats and repeats. The problem is when you're watching TV or something, it can trigger and then you're just stuck hearing it talk through your show, and when you tell it to stop, it still just keeps repeating and says that there's no music playing. It would be really nice that it would have better stop detection. Just the ability to know when it should just stop using context clues. If it hears a conversation between two people, it should know that it's not a part of it and it should be stopped unless it's specifically referenced or it should be able to see that it's not needed and be stopped if there's nothing specifically calling for it.

[omaramin-2000]

The stop word only stops the playback responses/announcements and timers but not media player to avoid halting playing music lightly (e.g, false triggers) and so on. For the limited detection, this is something related to handling the background noise that we're working on there: #250

[TheBest-Name]

Wait, so the stop is also supposed to stop the responses. Because there were many times where it kept talking and I told it to stop many times and it just didn't seem to. Could that be my audio because it was talking or was that something I was missing?

[TheBest-Name]

Wait, when you say limited detection, are you talking about the stop word or trying to detect when it should stop replying.

[florian-asche]

The stop word is currently speaking for my language, german, not working really good. That might be changed when we add a sensitivity option for it. When waiting for speak i was not able to use the stop word. Maybe it is missing there.
As said in some other comments from me i really would like to refactor the code. What would help me there is a simple documentation how the future application could look like.

Wait for wake word
  detect speech
    use stop word to abort
    duck music when detected and unduck on end.
Play music
    music can only be stopped by HA
Timers
    set by HA
    ends
      use stop word to abort

something like that.

[TheBest-Name]

I guess you also need to include announcements too.
Also, if my communications feature request goes through, then it'll also need to be used to stop communications there. #248

[rolyantrauts]

Its what wakeword are for to give the unique sentence, so you don't get false activations.
Its should be wakeword_stop than just stop.
It just has to be that way to stop false activations as stop is so common...
Stop I think with open/micro wakewords provided is actually another class in the wakeword model and currently activates as a single word but likely could be adapted to a cascade function 1-wakeword->2-stop.
Currently i think it doesn't, but the code could be made easily to require a wakeword before stop is accepted as stop.

I use a custom wakeword not open/micro with BoWWServer as wakeword is just wakeword doesn't even have VAD as it used purely as a broadcast switch where the whole sentence with wakeword is sent upstream so industry standard Silero, more compute VAD can run.
Parakeet2HA isn't just ASR its a second stage authoritative wakeword and even though you could run without wakeword it is supposed to be authoritative expecting a wakeword-stop command.

Google from last play keeps the ASR alive for an extended 30 sec period to capture, even without voice, just a stop command in this period, stops, but also without being triggered 'stop' is just too common and too easy to trigger otherwise. That 30sec grace period minimises false stops that also could be implemented.

[aryanhasgithub]

AEC is more effective then NS and AGC for wake word detection I tried with and without AEC there was a noticeable improvement in % recognized

[rolyantrauts]

I haven't looked but presume its just the web_rtc AEC that is the same in pulse_audio. That works great, much better than speex, but over a certain level it just fails on arm at least, as never did try x86.
Maybe its had an update, but was never great on arm, for some reason maybe compute?
Haven't tried it since the rhasspy forums.
@TheBest-Name The NS in web_rtc is even stinkier than RnNoise after a google, but RnNoise is a step up but still the granddaddy of ML NS.

Still though a voice pipeline is an end2end architecture so if you haven't trained a wakeword by augmenting the training dataset with earlier filters especially ML that have a signature often they can reduce accuracy.

DTLN is the next step up from RnNoise and have a plugin in here https://github.com/rolyantrauts/PiDTLN2/tree/main/plugin
DeepFilterNet is a step up from DTLN but will not run on a Pizero2/P3 https://github.com/Rikorose/DeepFilterNet

The WebTRC AEC has a ML element but the main filter is DSP for linear AEC so only the residual non linear left over gets a secondary ML treatment, so prob doesn't effect much.

You might be better doing what VoicePE, Respeaker-Lite, Sat1 does where NS is only for wakeword and ASR just gets AGC.
#99 (reply in thread)
Presume being Xmos hardware its more difficult to to finetune your wakeword / ASR models.

ASR/Wakeword models if you present signatures that where not part of training, can drop accuracy hugely.
DeepFilterNet is an amazing NS but feed Whisper without finetuning and recognition becomes poor even though for a human it sounds brilliant the levels of noise it can cope with.
I never did try RtxVoice but presume is similar.

Finetune your models to create a end2end architecture you can get really great results, just throw any model/DSP you can get varying results with some being worse than just a mic and ASR and presume that is why VoicePE, Respeaker-Lite and Sat1 just have gain control and bypass all else and its up to the Dev to prove it works different to VoicePE / Respeaker-lite as they do seem to use the same firmware.

I have been retraining DTLN with sparsity training as for audio F32 is obviously better than Int8 and the above was some analysis for the max noise level I could add to the dataset, which at epoch22 is about -10dB SNR to try and stay at a STOI level of at least 7 for the wakeword I would train. Approx 3x louder than the speech, where after denoising its still intelligiable.

Its the same as when you create a custom wakeword with own voice as works great with own voice, but not so great with anyone else.
Same with speech enhancement when I get round to training the wakeword it will work great with DTLN but not other SE as they all have different signatures.

[rolyantrauts]

Everything is pretty much a complex compromise, but one of the most effective is just simple physics of the inverse square law.
If you double the distance from a point source (e.g., from 1 meter to 2 meters), the intensity of the sound is reduced to one-quarter (1/4) of its original value.

I will mention this one more time as a normal layout in a room of any music lover is x2 well positioned speakers, because our media is recorded in stereo.
If you just provide BT or some form of wireless audio you get the best and nicest sounding audio of any as audiophiles have been doing for decades.

The most common room formats, lounges, bedrooms means its very easy to expect common locations to be able to put a mic much nearer to the voice and have some distance to speakers.
Every meter / 3 foot you get -6db, which is pretty huge as in the above I should of used RMS power as -10dB of the above filter is x10 less RMS noise than speech, amplitude its x3.
Combining the effect of distance and a filter can be huge as at 2m / 6foot of me sat in front of my big TV I am getting -12dB purely from distance and another 10dB from the filter x160 less RMS noise than speech.
We measure sound in dB where unity is at 1 meter as the closer you get to the mic you get the same but this time you get approx +6db every time you halve the distance.
At that standard broadcast distance 25-30mm it prob near +12db which to my speakers might be x2500 less RMS noise than voice.

It might be cool to stick a microphone on top of a speaker, but as any audio engineer will tell you its utterly stupid, if you don't need to. There is no reason when you can create ceiling/wall mikes to make a smart speaker that you don't have access to the highly binned parts and the tech resources big tech have.
Its very easy for opensource to make a linux-voice-assistant, its really hard to make a smart-speaker and why would you unless you have commercial intentions.

The standard type room layout of wall, standing or ceiling speakers is a massive boost to say a mic and speaker on a desk as the vibrations of the speaker travel through the medium of the desk, especially bass frequencies and end up getting transferred to the mic.
This is resonant distortion and is a very different signal to the audio at the DAC of your soundcard and it totally kills linear AEC as the ref signal has very little match to the incoming audio its trying to process.

When I was testing WebRTC AEC with speakers and mics and enclosure types, I gave up as it worked great at low volume, but anytime I ramped up the volume to music listening levels, I could even see it in Audacity as the bass would seep these huge spikes into the incoming mic audio, irrespective of rubber insulation and all manner of attempts. Having speakers and mics in very close proximity exponentially makes the noise problem worse and you start to marvel at some of the truly astounding engineering that goes into these commercial smart speakers that without the economies of scale or dumping for a loss, its prob not possible without ridiculous cost for the average maker.
I understand more now about linear AEC how its quite low compute, exceptional vulnerable to any clock drift or non-linear input (distortion) and how WebRTC AEC for most devices is perfect as its mostly a linear filter with a small ML residual echo cleaner. The further your mic and speaker and the more you can isolate resonance the better it works, but soon as you get any distortion above the level the residual ML can cope with (quite low secondary function) it starts to fail.

Google / Amazon and likes have purchase power to create what are called differential mic arrays that can capture a full frequency audio that standard array need multiple geometries to capture.
Small spacings to capture the high frequencies and wider spacing for low frequencies and this extends beyong the voice frequencies so that noise can be cancelled.
The mics generally available from the likes of digikey, RS and else where are really good mics maybe -26db sensitivity, 64dB SNR and quite high Acoustic Overload Point (AOP) of 110dB across the frequency ranges.
They can quite easily make those early large multi geometry arrays, but even though close they are useless for making small differential arrays, even though sensitivity might be higher than those Google/Amazon use.
Its the SNR of ultra-low noise mics 70dB+ with very high Acoustic Overload Point (AOP) of maybe 130dB, but where by being big and economies of sale comes in they will have the manufacturers binning devices for them at extremely close tolerances and also component placing in manufacture.
We are talking fractional tolerances of +-1 moving to +-0.5, fairly big bumps in SNR and AOP where big tech will have a full range signal from a differential small array that is above the noise floor, whilst smaller vendors will not.

Its like the Sat1 that has a small array 4 mic square geometry, but not the mics or purchase power to enable it.
Its skipped a generation with array geometry, but its mics are gen1 devices.
This doesn't mean opensource can not do mic arrays, just that certain geometries are not available as we don't have access to dictate binning from a manufacturer, to get those ultra tight tolerances of differential arrays.
You just go back to the older style star arrays where a central mic to outside will give the small dimension's of the high frequencies and the long diagonals the low frequencies of relatively low compute MVDR arrays.
Everything is a compromise of components and manufacture processes available, number of mics, geometry and placement.

I am not sure what to do with a 44mm small square array that doesn't have mics capable to be a differential array.
The 2 mic Xmos alg does something very different which is almost like AEC where one mic is used a reference based on the TDOA between the other and maybe the tflite model the use is just a simple vad to differentiate between noise and voice.
I don't think its MVDR or beamforming as the 62mm 2mic diagonal due to physics would create a very limited frequency range.
I do have one idea where the 2mic arrangement continues to work local and the 4mic upstreamed as it would be interesting what x2 Sat1 hats could do positioned deliberately to have different coverage of a room, so at least 1 is in a good proximity to recognise wakeword and then centrally upstream to create this 8 mic wide array...

There are a ton of things you can do with some lateral thought and its just having the time and knowledge to implement it.
Like there is much opensource linear AEC even non-linear all needing testing to effectiveness and how much compute it takes.
I keep meaning to try low compute linear AEC centrally that should not be possible due to clockdrift, but keep looking at the ultra tight sync of SendSpin and its Kalman timings that are sub sample micro-seconds, that actually maybe its possible...

That WebRTC works great as long as you don't create a smart speaker or have it loud so sound moves through materials and creates a lot of distortion so the linear part fails.
The is no such thing as one Alg is better than another as with WebRTC the NS is just a simple stationary noise filter , even more lightweight than RnNoise and there is a whole load of different NS to chose from.

That a single standard mic closer to you, with a standard hi-fi speaker arrangement with some relatively simple low end algs can outperform a $70 array dev kit, if the dev kit is used in smart speaker configuration.
So its not even what it is, what it processing it does, its also how you use it and where you place it to how well a voice system will work.

Welcome to the wonderful and complex world of audio engineering.
For many users the low cost of a single mic and the simplicity to try and keep speakers/noise and mic as far apart as possible and as close to voice, is likely the simplest and lowest cost option.
That its also easy to have multiple providing differing area covering the best stream, that can be selected as its likely it will give the best wakeword hit probability.
Distributed almost conference like systems have never been explored by big tech as Alexa was initially designed as a voice device to lock you into the Amazon shopping ecosphere, but actually mics on a speaker is one of the most stupid and hard devices to create, stupid as there are many alternatives where you do not need to.

I have MS and its been a 26 year struggle of 3 exacerbations, constant pain, co-ordination and cognitive problems, where this little hobby has given me something to do over the last 7 years.
Dunno about Webrtc AEC as its pretty good, but the kicker is when you have you music blasting and have mic and speaker in close proximity as your shouting 'stop' it can be that is when it decides to fail.
Haven't checked it for a while, think likely it has better latency than it did on Pulseaudio, but could do the same of failing when you need it and working great the rest of the time, due to the environment of the time.

Even DTLN as have definitely made some improvements and my ML ability is prob above average, or at least the effort I put into it is :)
GTCRN is prob better as its filterbank methods are less harsh than DTLN's masking and likely retains a more original voice pattern. You invest time get to know something, start to understand it and something newer comes along.
Third party noise for a voice system for all its commands is more common than a 'stop' when playing and why I haven't paid as much attention to AEC for a number of years. Your mic device doesn't need to be the device that emits audio and when like that a filter can process all noise whilst AEC only processes what is playing.

The Pizero2 is not so great either as its neon instructions support int8 or f32 but no Int16/F16. The later Cortex-A55 boards have Neon Int16/f16 support but also Ml instructions so prob something like the Radxa Zero3 which is Pi4 like perf due to the Ml instructions and improvements is likely approaching double that due to native Int16/f32.
Prob should be trying to create a Int16 quantised GTCRN model, but hey, people know Pi.
Might even run DeepFilterNet if quantised to Int16.

[TheBest-Name]

I just came on and saw the posts.
I'm just curious, it sounds like you are an audio engineer.

The one question I have though is if you do put the microphone apart from the speaker, wouldn't the latency of the sound traveling through the air also cause some issue with being able to remove it? Other than just passing it through a noise filter?

Also, when you say, "The NS in web_rtc is even stinkier than RnNoise after a google, but RnNoise is a step up but still the granddaddy of ML NS.". What are you actually saying about it? Are you saying that one would be better for this use case than another?

Thank you.

[rolyantrauts]

Not an audio engineer, have had a background, but far more of late reading papers over 7 years and ML audio engineering.
Inverse square law is basic high school physics, but always played about with electronics and audio gear, bands...
The latency of sound is not the problem, especially when saying having the mic close to voice and far from noise.
the oddly named smart speaker is the opposite, with mic ontop of noise and far from voice.

Yeah its all a trade off usually compute, the NS in webtrc looks like some simple filter, very low compute.
RnNoise is a combo filter / ML Jean-Marc Valin published in 2017–2018 that fused DSP with a RNN network to create what was one of the 1st opensource lightweight, low-latency, real-time noise suppression algs.
2017 was the birth to quite a few Ml frameworks so it was very cutting edge at the time.
As the frameworks evolved and practitioners, Ml has become dominant over DSP.
Much isn't opensource but we have a lot now in the openspace, much are just academic models file based that likely could be made realtime.
bcresnet which wasn't streaming and something I keep meaning to go back to as the FFT I did something stupid and didn't do any windowing / overlap with the FFT and will fix one time.
Irrespective its quite a good example as its a StateOfArt wakeword opensource where the creators publish for academic reason, but don't supply the code ready to use for realtime streaming.
There are quite a lot of similar really good ML that with a bit of geekery and effort can be made really useful for use, if you don't decide to do something dumb.
TSE to https://github.com/rrbluke/NRES

Guess its Github as it knows what I have been looking at and publishing so it pushes relevant trending repo's, as it has over the last 7 years.

But yeah there is always better and its a bit like LLM's as they tend to get bigger and need more compute, often someone comes out with anew idea that exceeds the last StateOfArt or creates a super efficient tiny model that is almost as good as the fat heavy ones.
The 1st link above TSE (Targeted Speech enhancement) is what Google/Amazon do as they have dropped most algs to try and cancel unknown noise in favour of extracting a known voice.
They just have super optimised models to run on there custom AI accelerators (NPU for all purposes) and are owners of super efficient kind that keep closely to their chests.

We have had a revolution in ML models of all types in the last 20 years, but yeah the NS in WebRTC is sort of in the ML dark ages, with RnNoise being an upgrade, but might be wrong as it was of little interest, so didn't spend much time researching it.

You just get a feel for things as the above TSE even if pasted 113.24 MACs (G/s) / 12.68M params, its posts superb state-of-art results but the MAC's is an ooof that prob mini-pc / gpu, but SpEx+, maybe not PiZero2 but looks interesting, then you go on the hunt will it quantise, does it run causal / realtime or can it be made to...

[TheBest-Name]

Yes, it does seem like it is kind of a lot.
One question I just thought of is if you could train a machine learning model to get rid of the signature that it provides.
Because I remember you were saying that wake words and also stop words and those kinds of models. Can have a hard time detecting when passed through a filter because it gets the signature. Well, can you take a noise filter and make it so its filter gets trained out of it?

I'm just curious because it also seems like that alone would also help in just this context of this issue, I'm not sure.

Also, if you had to choose one for this project, which do you think would be better? The Google/Amazon approach of just extracting a known voice like you were saying or a noise filter.

Honestly, that wouldn't be just good for the stop, that would also be good for the assistant in general. At least I would imagine if it could lock onto a voice and then make it so that the TV or other things wouldn't talk to it, that might help a lot. Though, I'm not sure of all the stuff, so that's why I'm asking.

[rolyantrauts]

You dont' have to remove the signature as its the same as presenting a wakeword model with a volume input that it has never seen before that was not in its training data.
You just make sure what a model see's is what the training data is. Its that simple, train in the signature.
There are generative models that can fill in and clean voice, they are the very big and fat types.

Google/Amazon you have the best engineers in the world with R&D funding and resources of a top 5 world tech corporation.
Amazon almost managed to get a monopoly on the home voice market and Google has kicked their ass since then as near all tech evolution in current Gen5 from microphone array type to TSE, AI accelerators was a Google 1st. Much of this was to reduce cost of cloud services and do things local, as at one stage Alexa was leaking $10Bn annual.
About the only thing Amazon was 1st is form factor as the new to be released Google home is also going to be a sphere and its the prefect shape for a desk mount microphone to minimise reflections and capture 3D audio with minimal and maximum obstruction. The last part might confuse but the over hang of the sphere helps reflect away reflections from table tops, but they are not sphere's for aesthetic reasons, these things are engineered to ultra levels of sophistication and it would be utterly stupid to ignore them.

They use small differential arrays that give a full frequency audio that are very resilient to noise and yes TSE (Targetted Voice Extraction) original focus was the cocktail party problem of double-talk. The biggest problem for voice devices is not what they are playing its what other devices are playing and its often human voices.
Its a sort of hybrid ML source seperation where the voice-embeddings of a user provide the target to separate, the method to extract known voice than cancel unknown noise has also in all areas of voice posted SoTa scores.
If you look at that https://github.com/HaoFengyuan/X-TF-GridNet?tab=readme-ov-file#results that plucking out voice in noise x120 RMS SNR and even though prob not the best metric PESQ of 3.77 is an absolutely miraculous clarity, Short-Time Objective Intelligibility (STOI) is prob a better metric.

If some did actually actually pay attention you might not get cul-de-sacs such as having gen 2 microphones on a gen 4 array geometry that leaves you without a paddle.
I have spent a ton of time, researching and you can learn a lot from how the generations changed as leading edge voice tech evolved, but they are trying to create commercial devices to process local speech so they can use local compute and local energy than to use cloud services that failed to capitalise revenue.

The biggest thing you can learn is that the standalone device is massively complex, 11 years ago the 1st Alexa was released and design and production has stretched 2 of the largest tech companies to there limits.
What is strange when it so hard and costly to create the device and most available opensource does not have the R&D and resources to convert and quantise to run on the latest low energy embedded AI accelerators that opensource has a fixation with cloning them.
There is a lot of delusional aspiration to be one of the names in voice tech and the approach and choice of device has likely stagnated progress for 5 years.
Opensource like HA is a homelab, geek paradise where there is a plethora of available cutting edge opensource from academia, but very little will run in the compute of the device, but yeah TSE and tons of great voice tech exists that like HA will run as a server, but is ignored because IMO its $ aspirational delusion that they can produce a device, which so far has been a 5 year total fail.

[TheBest-Name]

What you said about picking out the correct voices out of a room, it's definitely one of the problems I was having because it would start talking when the TV was going and other things were happening, even though what the TV was saying wasn't even close to the wake word. In fact, it was farther away than what I have to do when I get right up to it and I say the wake word and it doesn't listen. But then the TV doesn't even say anything close to it, and it pops right on.

That was one of the reasons why I was saying to have the LLM be able to decide what actually constitutes talking to the assistant. It was actually to stop that exact scenario where the TV wasn't even talking to it, or where someone else in the room wasn't even talking to it and it activates.

[rolyantrauts]

Its even more accurate than that as it knows the voice profile of the user and so can differentiate the users voice of the profiles it stores and extract the voice.
If you look at https://github.com/HaoFengyuan/X-TF-GridNet?tab=readme-ov-file#results
That is voice extracted from noise x120 louder than voice with huge clarity aka the PESQ score.
TSE is its own thing, its not a LLM and its currently one of the most active areas in voice research.
There is a list there of some, ready working as there are many opensource repo's on github but you have to have the resources of Google and amazon to create highly quantised custom versions to run on embedded AI accelerators with extremely low cost as actually I think the Google Nest audio from memory is a Cortex-A53 with an embedded AI accelerator (NPU).
There are already custom highly efficient models to do this, not highly efficient where they are running on a Pi3 with a powerful embedded NPU but very much in the N100 and up mini PC area and why concentrating on a device and not the server is just so stupid and has stagnated opensource voice for 4 years.

https://community.rhasspy.org/t/thoughts-for-the-future-with-homeassistant-rhasspy/4055/2
4 years and equally bemused at current direction in HA, we don't need LLM's as the tech and opensource already exists, its just ignored unless it can be refactored and presented as own.
It why I have been giving DTLN some love again as after all the effort one member of the community invested for it to be ignored like that is extremely bad form.

Even mics and arrays, it would seem all about control as soon as you bake a geometry into a pcb its fixed and no longer open, the claim that this is open-source hardware is just ridiculous.
This is especially so when it has gen 2 mics and a array geometry of later gen small differential array mics, as you scratch your head that at best you likely could make a far from optimised MVDR, that might be not improvement of the the 2 Mic Xmos provided algs. things is it doesn't even provide them easily on I@S and have to learn a whole new microcontroller ecpsphere dev just to present them to make the whole xmos microcontroller a PDM to I2S convertor that costs $70!
That for me the purchase power to dictate binning and control tolerances for differential arrays doesn't matter that much, as I don't need to create a commercial all-on-one device and with access to gen 2 components I can make a Gen 2 mic array.
Mic arrays as Google/Amazon clearly demonstrate are designed for the methods and algs in use and not the other way round.
This is because of the physical properties of sound.
So going back to Gen 2 like the echo dot with its 7 mic star array that obviously screams MVDR the center mic to outside mics gives the high frequency band and the long diagonals gives the low frequencies.
The alg enforced the geometries of the array, so its a completely naive statement to say ready made fixed geometry microphones are opensource designs as soon as cast onto PCB they become hard and closed.
I have been wondering about a compromise of a 4 mic 'Y' star for MVDR use where I can use gen 2 components that are easily available, because I don't intend to put it in the harsh environment of directly ontop of a speaker.
All opensource has ever needed was a 8 channel ADC that has analogue differential mems daughter boards that from 2 to 8 mics, you can make your own geometries, algs with out stupid LED rings or dacs and amps, so that it is about choice and truly open on how its employed.
I made a compromise on mics as even a 4 channel TDM ADC of lower cost would be kind of nice, hence the thoughts on a 'Y' star array, but there are solid reasons why Amazon went for a 7 mic star for MVDR.