professor b: U u u u  , I meant , you know , this end of the world , yeah , is really what I meant , phd c: Oh . phd c: I did some experim  , just a few more experiments before I had to , go away for the w well , that week . professor b: Great ! phd c: Was it last week or whenever ?  , so what I was started playing with was the th again , this is the HTK back - end . And , I was curious because the way that they train up the models , they go through about four sort of rounds of of training . And in the first round they do  , I think it 's three iterations , and for the last three rounds e e they do seven iterations of re - estimation in each of those three . And so , you know , that 's part of what takes so long to train the the the back - end for this . It 's this program called H E professor b: But in HTK , what 's the difference between , a an inner loop and an outer loop in these iterations ? phd c: OK . So what happens is , at each one of these points , you increase the number of Gaussians in the model . phd c: And so , in the final one here , you end up with ,  for all of the the digit words , you end up with , three mixtures per state , professor b: Yeah . So I had done some experiments where I was I I want to play with the number of mixtures . phd c: But , I wanted to first test to see if we actually need to do this many iterations early on . phd c: And so , I I ran a couple of experiments where I reduced that to l to be three , two , two , five , I think , and I got almost the exact same results . So , I I think m it only took something like , three or four hours to do the full training , professor b: As opposed to ? phd f: Good . phd c: as opposed to wh what , sixteen hours or something like that ?  , it takes you have to do an overnight basically , the way it is set up now . phd c: So , even we don't do anything else , doing something like this could allow us to turn experiments around a lot faster . professor b: And then when you have your final thing , do a full one , so it 's phd c: And when you have your final thing , we go back to this .  , it 's like one little text file you edit and change those numbers , and you don't do anything else . phd c: So it 's a very simple change to make and it doesn't seem to hurt all that much . phd a: So you you run with three , two , two , five ? That 's a phd c: So I  , I I have to look to see what the exact numbers were . phd c: I I thought was , like , three , two , two , five , phd a:   Oh , the other thing that I did was , I compiled the HTK stuff for the Linux boxes . So , you can now run your experiments on that machine and you can run five at a time and it runs , as fast as , you know , five different machines . phd c: So , I 've forgotten now what the name of that machine is but I can I can send email around about it . phd c: And so we 've got it now HTK 's compiled for both the Linux and for , the Sparcs .  , you have to make you have to make sure that in your dot CSHRC , it detects whether you 're running on the Linux or a a Sparc and points to the right executables .  , and you may not have had that in your dot CSHRC before , if you were always just running the Sparc . phd c: I can I can tell you exactly what you need to do to get all of that to work . phd c: So , together with the fact that we 've got these faster Linux boxes and that it takes less time to do these , we should be able to crank through a lot more experiments . phd c: So after I did that , then what I wanted to do was try increasing the number of mixtures , just to see ,  see how how that affects performance . In fact , you could do something like keep exactly the same procedure and then add a fifth thing onto it phd c:   grad e: So at at the middle o where the arrows are showing , that 's you 're adding one more mixture per state , phd c:  - huh . It goes from this  , try to go it backwards this at this point it 's two mixtures per state . And I think what happens here is professor b: Might be between , shared ,  shared variances or something , phd c: Yeah . phd c: there because they start off with , an initial model which is just this global model , and then they split it to the individuals . I don't know yet the what happened Tuesday , but the points that they were supposed to discuss is still , things like the weights ,  professor b: Oh , this is a conference call for , Aurora participant sort of thing . professor b: Do you know who was who was since we weren't in on it , do you know who was in from OGI ? Was was was Hynek involved or was it Sunil phd a: I have no idea . professor b: or ? phd a: Mmm , I just professor b: Oh , you don't know . So the points were the the weights how to weight the different error rates that are obtained from different language and and conditions . phd a: Some people are arguing that it would be better to have weights on  well , to to combine error rates before computing improvement .  , and the fact is that for right now for the English , they have weights they they combine error rates , but for the other languages they combine improvement . And right now actually there is a thing also , that happens with the current weight is that a very non - significant improvement on the well - matched case result in huge differences in in the final number . phd c: How should that be done ?  , it it seems like there 's a simple way phd a:   phd c: Th - they 're professor b: But the but , the other thing phd a: In professor b: I don't know I haven't thought it through , but one one would think that each It it 's like if you say what 's the what 's the best way to do an average , an arithmetic average or a geometric average ? phd c:   professor b: So phd c: Well , it seems like they should do , like , the percentage improvement or something , rather than the absolute improvement . But the question is , do you average the relative improvements or do you average the error rates and take the relative improvement maybe of that ? phd a: Yeah . professor b: And the thing is it 's not just a pure average because there are these weightings . And so when you average the the relative improvement it tends to to give a lot of of , importance to the well - matched case because the baseline is already very good and , i it 's phd c: Why don't they not look at improvements but just look at your av your scores ? You know , figure out how to combine the scores phd a:   phd c: with a weight or whatever , and then give you a score here 's your score . professor b: Well , that 's what he 's seeing as one of the things they could do . professor b: It 's just when you when you get all done , I think that they pro I m I I wasn't there but I think they started off this process with the notion that you should be significantly better than the previous standard . professor b: And , so they said " how much is significantly better ? what do you ? " And and so they said " well , you know , you should have half the errors , " or something , " that you had before " . professor b: i i it does seem like it 's more logical to combine them first and then do the phd a: Combine error rates and then professor b: Yeah . When when you combine error rate it tends to give more importance to the difficult cases , and some people think that professor b: Oh , yeah ? phd a: well , they have different , opinions about this . Some people think that it 's more important to look at to have ten percent imp relative improvement on well - matched case than to have fifty percent on the m mismatched , and other people think that it 's more important to improve a lot on the mismatch and So , bu phd c: It sounds like they don't really have a good idea about what the final application is gonna be . professor b: Well , you know , the the thing is that if you look at the numbers on the on the more difficult cases , if you really believe that was gonna be the predominant use , none of this would be good enough . professor b: whereas you sort of with some reasonable error recovery could imagine in the better cases that these these systems working . So , I think the hope would be that it would  , it would work well for the good cases and , it would have reasonable reas soft degradation as you got to worse and worse conditions . I I guess what I 'm  , I I was thinking about it in terms of , if I were building the final product and I was gonna test to see which front - end I 'd I wanted to use , I would try to weight things depending on the exact environment that I was gonna be using the system in . professor b: I th phd c: So if if they don't know , doesn't that suggest the way for them to go ?  , you assume everything 's equal .  , y y  , you professor b: Well , I I think one thing to do is to just not rely on a single number to maybe have two or three numbers , phd c: Yeah . professor b: and and and say here 's how much you ,  you improve the ,  the the relatively clean case and here 's or or well - matched case , and here 's how here 's how much you , phd c:   professor b: So , I guess what you would do in practice is you 'd try to get as many , examples of similar sort of stuff as you could , and then , phd c: Yeah . professor b:  So the argument for that being the the the more important thing , is that you 're gonna try and do that , but you wanna see how badly it deviates from that when when when the ,  it 's a little different . phd c: So professor b: phd c: so you should weight those other conditions v very you know , really small . That 's a that 's a that 's an arg phd c: that 's more of an information kind of thing . professor b: that 's an ar Well , that 's an argument for it , but let me give you the opposite argument . professor b: are you gonna have w  , examples with the windows open , half open , full open ? Going seventy , sixty , fifty , forty miles an hour ? On what kind of roads ? phd c:   professor b: I I I think that you could make the opposite argument that the well - matched case is a fantasy . professor b: I think the thing is is that if you look at the well - matched case versus the po you know , the the medium and the and the fo and then the mismatched case , we 're seeing really , really big differences in performance . You wouldn't like that as soon as you step outside You know , a lot of the the cases it 's is phd c: Well , that 'll teach them to roll their window up . professor b: in these cases , if you go from the the , I don't remember the numbers right off , but if you if you go from the well - matched case to the medium , it 's not an enormous difference in the in the the training - testing situation , and and and it 's a really big performance drop . professor b: You know , so ,  Yeah ,  the reference one , for instance this is back old on ,  on Italian  , was like six percent error for the well - matched and eighteen for the medium - matched and sixty for the for highly - mismatched .  , and , you know , with these other systems we we helped it out quite a bit , but still there 's there 's something like a factor of two or something between well - matched and medium - matched . And so I think that if what you 're if the goal of this is to come up with robust features , it does mean So you could argue , in fact , that the well - matched is something you shouldn't be looking at at all , that that the goal is to come up with features that will still give you reasonable performance , you know , with again gentle degregra degradation , even though the the testing condition is not the same as the training . professor b: So , you know , I I could argue strongly that something like the medium mismatch , which is you know not compl pathological but  , what was the the medium - mismatch condition again ? phd a: it 's Yeah . Medium mismatch is everything with the far microphone , but trained on , like , low noisy condition , like low speed and or stopped car and tested on high - speed conditions , I think , like on a highway and professor b: Right . phd a: So professor b: So it 's still the same same microphone in both cases , phd a: Same microphone but Yeah . But the way they have it now , it 's I guess it 's it 's They they compute the relative improvement first and then average that with a weighting ? phd a: Yeah . professor b: so , u i since they have these three categories , it seems like the reasonable thing to do is to go across the languages and to come up with an improvement for each of those . professor b: Just say " OK , in the in the highly - matched case this is what happens , in the m the ,  this other m medium if this happens , in the highly - mismatched that happens " . professor b: I think that that I I I gather that in these meetings it 's it 's really tricky to make anything ac make any policy change because everybody has has , their own opinion phd a:   Yeah , but there is probably a a big change that will be made is that the the baseline th they want to have a new baseline , perhaps , which is , MFCC but with a voice activity detector . So they want to have at least fifty percent improvement on the baseline , but w which would be a much better baseline . phd a: And if we look at the result that Sunil sent , just putting the VAD in the baseline improved , like , more than twenty percent , professor b:   phd a: which would mean then then mean that fifty percent on this new baseline is like , well , more than sixty percent improvement on on o e e  professor b: So nobody would be there , probably . professor b: So whose VAD is Is is this a ? phd a: they didn't decide yet . I guess i this was one point of the conference call also , but mmm , so I don't know .  , it 's not that the design of the VAD isn't important , but it 's just that it it it does seem to be i  , a lot of work to do a good job on on that and as well as being a lot of work to do a good job on the feature design , phd a: Yeah . Per - e s s someone told that perhaps it 's not fair to do that because the ,  to make a good VAD you don't have enough to with the the features that are the baseline features . phd c: Wha - what do you mean ? phd a: Yeah , if i professor b: So y so you m s Yeah , but Well , let 's say for ins see , MFCC for instance doesn't have anything in it , related to the pitch . So suppose you 've that what you really wanna do is put a good pitch detector on there and if it gets an unambiguous phd c: Oh , oh . professor b: if it gets an unambiguous result then you 're definitely in a in a in a voice in a , s region with speech . phd c: So there 's this assumption that the v the voice activity detector can only use the MFCC ? phd a: That 's not clear , but this e professor b: Well , for the baseline . professor b: So so if you use other features then y But it 's just a question of what is your baseline . Right ? What is it that you 're supposed to do better than ? phd c: I g Yeah . professor b: And so having the baseline be the MFCC 's means that people could choose to pour their ener their effort into trying to do a really good VAD phd c: I don't s But they seem like two separate issues . phd c: Right ?  professor b: Unfortunately there 's coupling between them , which is part of what I think Stephane is getting to , is that you can choose your features in such a way as to improve the VAD . professor b: You should do both phd c: Right ? professor b: and and I I think that this still makes I still think this makes sense as a baseline . professor b: you know , we had the MFCC 's before , lots of people have done voice activity detectors , phd a:   professor b: you might as well pick some voice activity detector and make that the baseline , just like you picked some version of HTK and made that the baseline .  , and if one of the ways you make it better is by having your features be better features for the VAD then that 's so be it . professor b: But , at least you have a starting point that 's  , cuz i i some of the some of the people didn't have a VAD at all , I guess . professor b: then they they looked pretty bad and and in fact what they were doing wasn't so bad at all . And if it turns out that you can't improve on that , well , then , you know , nobody wins and you just use MFCC .  , it seems like , it should include sort of the current state of the art that you want are trying to improve , and MFCC 's , you know , or PLP or something it seems like reasonable baseline for the features , and anybody doing this task , is gonna have some sort of voice activity detection at some level , in some way . They might use the whole recognizer to do it but rather than a separate thing , but but they 'll have it on some level . phd c: It seems like whatever they choose they shouldn't , you know , purposefully brain - damage a part of the system to make a worse baseline , or professor b: Well , I think people just had phd c: You know ? professor b: it wasn't that they purposely brain - damaged it . professor b: And and then when the the the proposals actually came in and half of them had V A Ds and half of them didn't , and the half that did did well and the half that didn't did poorly . So what happened since , last week is well , from OGI , these experiments on putting VAD on the baseline . And these experiments also are using , some kind of noise compensation , so spectral subtraction , and putting on - line normalization , just after this . So I think spectral subtraction , LDA filtering , and on - line normalization , so which is similar to the pro proposal - one , but with spectral subtraction in addition , and it seems that on - line normalization doesn't help further when you have spectral subtraction . phd c: Is this related to the issue that you brought up a couple of meetings ago with the the musical tones phd a: I phd c: and ? phd a: I have no idea , because the issue I brought up was with a very simple spectral subtraction approach , phd c: Mmm . phd a: and the one that they use at OGI is one from from the proposed the the the Aurora prop  , proposals , which might be much better . And what 's happened here is that we so we have this kind of new , reference system which use a nice a a clean downsampling - upsampling , which use a new filter that 's much shorter and which also cuts the frequency below sixty - four hertz , professor b: Right . professor b: When you say " we have that " , does Sunil have it now , too , phd a: I No .  , it seems to improve on the well - matched case , but it 's a little bit worse on the mismatch and highly - mismatched  when we put the neural network . And with the current weighting I think it 's sh it will be better because the well - matched case is better . professor b: But how much worse since the weighting might change how how much worse is it on the other conditions , when you say it 's a little worse ? phd a: It 's like , fff , fff  , ten percent relative . That 's phd a:  - y w when I say it 's worse , it 's not it 's when I I  , compare proposal - two to proposal - one , so , r  , y putting neural network compared to n not having any neural network . phd a: because it has  , this sixty - four hertz cut - off , clean downsampling , and ,  what else ?  , yeah , a good VAD . I I j  ,  pr professor b: But the latencies but you 've got the latency shorter now . phd f: Isn't it phd a: And so professor b: So it 's better than the system that we had before . phd a: And then I took this system and , mmm , w  , I p we put the old filters also . So we have this good system , with good VAD , with the short filter and with the long filter , and , with the short filter it 's not worse .  professor b: But what you 're saying is that when you do these So let me try to understand . professor b: that , on the i things are somewhat better , in proposal - two for the well - matched case and somewhat worse for the other two cases . professor b: So does ,  when you say ,  So The th now that these other things are in there , is it the case maybe that the additions of proposal - two over proposal - one are less im important ? phd a: Yeah . Then we tried , to do something like proposal - two but having , e using also MSG features . So basically we try to , find good features that could be used for voicing detection , but it 's still ,  on the ,  t phd f: Oh , well , I have the picture . phd a: we w basically we are still playing with Matlab to to look at at what happened , phd c: What sorts of phd f: Yeah . phd a: and phd c: what sorts of features are you looking at ? phd f: We have some phd a: So we would be looking at , the variance of the spectrum of the excitation , phd f: this , this , and this . So the So basically the spectrum of the excitation for a purely periodic sig signal shou sh professor b: OK . Yeah , w what yo what you 're calling the excitation , as I recall , is you 're subtracting the the ,  the mel mel mel filter , spectrum from the FFT spectrum . phd a: So we have the mel f filter bank , we have the FFT , so we just professor b: So it 's it 's not really an excitation , phd a: No . phd f: We have here some histogram , phd a: E yeah , phd f: but they have a lot of overlap . So , well , for unvoiced portion we have something tha that has a mean around O point three , and for voiced portion the mean is O point fifty - nine . phd c: How did you get your voiced and unvoiced truth data ? phd a: We used , TIMIT and we used canonical mappings between the phones phd f: Yeah . We , use TIMIT on this , phd a: and phd f: for phd a: th Yeah . phd f: But if we look at it in one sentence , it apparently it 's good , I think . phd a: It seems quite robust to noise , so when we take we draw its parameters across time for a clean sentence and then nois the same noisy sentence , it 's very close . There could be also the ,  something like the maximum of the auto - correlation function or which phd c: Is this a a s a trained system ? Or is it a system where you just pick some thresholds ? Ho - how does it work ? phd a: Right now we just are trying to find some features . Hopefully , I think what we want to have is to put these features in s some kind of ,  well , to to obtain a statistical model on these features and to or just to use a neural network and hopefully these features w would help phd c: Because it seems like what you said about the mean of the the voiced and the unvoiced that seemed pretty encouraging . phd c: Well , y I I don't know that I would trust that so much because you 're doing these canonical mappings from TIMIT labellings . phd c: Right ? So , really that 's sort of a cartoon picture about what 's voiced and unvoiced . phd c: i it it may be that that you 're finding something good and that the variance is sort of artificial because of how you 're getting your truth . But another way of looking at it might be that  , what w we we are coming up with feature sets after all . So another way of looking at it is that  , the mel cepstru mel spectrum , mel cepstrum , any of these variants , give you the smooth spectrum . By going back to the FFT , you 're getting something that is more like the raw data . So the question is , what characterization and you 're playing around with this another way of looking at it is what characterization of the difference between the raw data and this smooth version is something that you 're missing that could help ? So , looking at different statistical measures of that difference , coming up with some things and just trying them out and seeing if you add them onto the feature vector does that make things better or worse in noise , where you 're really just i i the way I 'm looking at it is not so much you 're trying to f find the best the world 's best voiced - unvoiced , classifier , phd c:   professor b: but it 's more that , you know , try some different statistical characterizations of that difference back to the raw data phd c: Right . The the more obvious is that that well , using the th the FFT , you just it gives you just information about if it 's voiced or not voiced , ma mainly ,  . phd a: this is why we we started to look by having sort of voiced phonemes professor b: Well , that 's the rea w w what I 'm arguing is that 's Yeah . professor b: But in in reality , it 's you know , there 's all of this this overlap and so forth , grad e: Oh , sorry . professor b: and But what I 'm saying is that may be OK , because what you 're really getting is not actually voiced versus unvoiced , both for the fac the reason of the overlap and and then , th you know , structural reasons , like the one that Chuck said , that that in fact , well , the data itself is that you 're working with is not perfect . professor b: So , what I 'm saying is maybe that 's not a killer because you 're just getting some characterization , one that 's driven by your intuition about voiced - unvoiced certainly , phd a:   professor b: but it 's just some characterization of something back in the in the in the almost raw data , rather than the smooth version . professor b: And your intuition is driving you towards particular kinds of , statistical characterizations of , what 's missing from the spectral envelope . professor b: obviously you have something about the excitation , and what is it about the excitation , and , you know and you 're not getting the excitation anyway , you know . So so I I would almost take a  , especially if if these trainings and so forth are faster , I would almost just take a  , a scattershot at a few different ways of look of characterizing that difference and , you could have one of them but and and see , you know , which of them helps . phd c: So i is the idea that you 're going to take whatever features you develop and and just add them onto the future vector ? Or , what 's the use of the the voiced - unvoiced detector ? phd a: I guess we don't know exactly yet . Th phd c: It 's not part of a VAD system that you 're doing ? phd f: No . phd a:  Yeah , it could be ,  it could be a neural network that does voiced and unvoiced detection , phd c:   professor b: But each one of the mixture components  , you have , variance only , so it 's kind of like you 're just multiplying together these , probabilities from the individual features within each mixture . So it 's so , it seems l you know phd c: I think it 's a neat thing .  , I know that , people doing some robustness things a ways back were were just doing just being gross and just throwing in the FFT and actually it wasn't wasn't wasn't so bad .  , so it would s and and you know that i it 's gotta hurt you a little bit to not have a a spectral ,  a s a smooth spectral envelope , so there must be something else that you get in return for that phd a:   phd c: So how does  , maybe I 'm going in too much detail , but how exactly do you make the difference between the FFT and the smoothed spectral envelope ? Wha - wh i i  , how is that ,  ? phd a: we just How did we do it up again ? phd f: we distend the we have the twenty - three coefficient af after the mel f filter , phd a:   phd f: And i the interpolation i between the point is give for the triang triangular filter , the value of the triangular filter and of this way we obtained this mode this model speech . phd a: S professor b: So you essentially take the values that th that you get from the triangular filter and extend them to sor sort of like a rectangle , that 's at that m value . phd a: So we have we have one point for one energy for each filter bank , phd f: mmm Yeah , it 's linear . phd a: which is the energy that 's centered on on on the triangle phd f: Yeah . At the n at the center of the filter phd c: So you you end up with a vector that 's the same length as the FFT vector ? phd a: Yeah . And I think the variance is computed only from , like , two hundred hertz to one to fifteen hundred . phd a: Above ,  it seems that Well , some voiced sound can have also , like , a noisy part on high frequencies , and But professor b: Yeah . phd a: Well , it 's just professor b: No , it 's makes sense to look at low frequencies . phd c: So this is  , basically this is comparing an original version of the signal to a smoothed version of the same signal ? phd f: Yeah . So i so i i this is  , i you could argue about whether it should be linear interpolation or or or or zeroeth order , but but phd c:  - huh . professor b: at any rate something like this is what you 're feeding your recognizer , typically .  , so the mel cepstrum is the is the is the cepstrum of this this , spectrum or log spectrum , phd a: So this is Yeah . professor b: whatever it You - you 're subtracting in in in power domain or log domain ? phd a: In log domain . But , anyway ,  and that 's phd c: So what 's th  , what 's the intuition behind this kind of a thing ? I I don't know really know the signal - processing well enough to understand what what is that doing . What happen if what we have have what we would like to have is some spectrum of the excitation signal , professor b: Yeah . phd a: And the way to do this is that well , we have the we have the FFT because it 's computed in in the in the system , and we have the mel filter banks , phd c:   phd a: and so if we if we , like , remove the mel filter bank from the FFT , we have something that 's close to the excitation signal . phd a: It 's something that 's like a a a train of p a pulse train for voiced sound phd c: OK . So do you have a picture that sh ? phd a: So - It 's Y phd c: Is this for a voiced segment , phd a: yeah . phd c: this picture ? What does it look like for unvoiced ? phd f: Yeah . This is phd f: but between the frequency that we are considered for the excitation phd a: Right . phd c: So , does does the periodicity of this signal say something about the the phd f: Fifteen p phd a: So it 's Yeah . professor b: to first order what you 'd what you 're doing  , ignore all the details and all the ways which is that these are complete lies .  , the the you know , what you 're doing in feature extraction for speech recognition is you have , in your head a a a a simplified production model for speech , phd c:   professor b: in which you have a periodic or aperiodic source that 's driving some filters . phd a: Do you have the mean do you have the mean for the auto - correlation ? professor b: first order for speech recognition , you say " I don't care about the source " . professor b: The filters roughly act like a ,  a ,  a an overall resonant you know , f some resonances and so forth that th that 's processing excitation . professor b: So if you look at the spectral envelope , just the very smooth properties of it , you get something closer to that . professor b: And the notion is if you have the full spectrum , with all the little nitty - gritty details , that that has the effect of both , phd c: Yeah . professor b: And so this is saying , well , if you really do have that sort of vocal tract envelope , and you subtract that off , what you get is the excitation . And I call that lies because you don't really have that , you just have some kind of signal - processing trickery to get something that 's kind of smooth . professor b: That 's why I was going to the why I was referring to it in a more a more , conservative way , when I was saying " well , it 's yeah , it 's the excitation " . This moved in the professor b: So so , stand standing back from that , you sort of say there 's this very detailed representation . professor b: but whenever you smooth you lose something , so the question is have you lost something you can you use ? phd c: Right . professor b: probably you wouldn't want to go to the extreme of just ta saying " OK , our feature set will be the FFT " , cuz we really think we do gain something in robustness from going to something smoother , but maybe there 's something that we missed . professor b: And then you go back to the intuition that , well , you don't really get the excitation , but you get something related to it . professor b: And it and as you can see from those pictures , you do get something that shows some periodicity , in frequency , phd c:   professor b: so , phd c: So you don't have one for unvoiced picture ? phd f: not here . professor b: But presumably you 'll see something that won't have this kind of , regularity in frequency , in the phd a: But Yeah . phd c: And so you said this is pretty doing this kind of thing is pretty robust to noise ? phd a: It seems , yeah . The mean is different with it , because the the histogram for the the classifica phd a: No , no , no . But th the kind of robustness to noise phd f: Oh ! phd a: So if if you take this frame , from the noisy utterance and the same frame from the clean utterance phd f:  . Cool ! phd f: I have here the same frame for the clean speech phd c: Oh , that 's clean . phd f: the same cle phd c: Oh , OK phd f: But they are a difference . phd a: Yeah , that 's phd f: Because here the FFT is only with two hundred fifty - six point phd c: Oh . phd a: because if we use the standard , frame length of of , like , twenty - five milliseconds , what happens is that for low - pitched voiced , because of the frame length , y you don't really have you don't clearly see this periodic structure , professor b:   phd a: Yeah , but it 's the same frame and phd c: Oh , it 's that time - frequency trade - off thing . Well ,  it looks better , but , the thing is if if ,  if you 're actually asking you know , if you actually j  , need to do place along an FFT , it may be it may be pushing things . professor b: And and ,  phd c: Would you would you wanna do this kind of , difference thing after you do spectral subtraction ? phd a: maybe . The spectral subtraction is being done at what level ? Is it being done at the level of FFT bins or at the level of , mel spectrum or something ? phd a: I guess it depends . professor b: how are they doing it ? phd a: How they 're doing it ? Yeah . Filter bank , phd a: no ? It 's on the filter bank , phd f: yeah . So we 'll perhaps try to convince OGI people to use the new the new filters and Yeah .  , has has anything happened yet on this business of having some sort of standard , source , phd a: not yet professor b: or ? phd a: but I wi I will call them and professor b: OK . phd a: now they are I think they have more time because they have this well , Eurospeech deadline is over phd c: When is the next , Aurora deadline ? phd a: and It 's , in June . professor b: Early June , late June , middle June ? phd a: I don't know w professor b:  .  , and he 's been doing all the talking but but these he 's he 's ,  phd f: Yeah .  , but has he pretty much been talking about what you 're doing also , and ? phd f: Oh , I I am doing this . I 'm sorry , but I think that for the recognizer for the meeting recorder that it 's better that I don't speak . phd f: Because professor b: You know , we 'll get we 'll get to , Spanish voices sometime , and we do we want to recognize , you too . phd f: After the after , the result for the TI - digits on the meeting record there will be foreigns people . phd c: Y professor b: We like we we 're we 're w we are we 're in the , Bourlard - Hermansky - Morgan , frame of mind . So it 's  , anything to talk about ? grad d: N  , not not not much is new . So when I talked about what I 'm planning to do last time , I said I was , going to use Avendano 's method of , using a transformation , to map from long analysis frames which are used for removing reverberation to short analysis frames for feature calculation .  , but , I decided not to do that after all because I I realized to use it I 'd need to have these short analysis frames get plugged directly into the feature computation somehow professor b:   grad d: and right now I think our feature computation is set to up to , take , audio as input , in general . So I decided that I I 'll do the reverberation removal on the long analysis windows and then just re - synthesize audio and then send that . Right ? grad d: or or even if I 'm using our system , I was thinking it might be easier to just re - synthesize the audio , professor b: Yeah ? grad d: because then I could just feacalc as is and I wouldn't have to change the code .  , longer - term if it 's if it turns out to be useful , one one might want to do something else , grad d: Right . professor b: but  , in in other words , you you may be putting other kinds of errors in from the re - synthesis process . But anyway it sounds like a reasonable way to go for a for an initial thing , and we can look at at exactly what you end up doing and and then figure out if there 's some something that could be be hurt by the end part of the process . So that 's That was it , huh ? grad d: That Yeah , e That 's it , that 's it . I went off on a little tangent this past week , looking at , modulation s spectrum stuff , and and learning a bit about what what ,  what it is , and , the importance of it in speech recognition . And I found some some , neat papers , historical papers from , Kanedera , Hermansky , and Arai . grad e: And they they did a lot of experiments where th where , they take speech and , e they modify the ,  they they they measure the relative importance of having different , portions of the modulation spectrum intact . grad e: And they find that the the spectrum between one and sixteen hertz in the modulation is ,  is im important for speech recognition . professor b: And and , the the MSG features were sort of built up with this notion grad e: Yeah . professor b: But , I guess , I thought you had brought this up in the context of , targets somehow . professor b: But i m grad e:  professor b: i it 's not  , they 're sort of not in the same kind of category as , say , a phonetic target or a syllabic target grad e: Mmm . professor b: or a grad e: I was thinking more like using them as as the inputs to to the detectors . professor b: So maybe , le phd c: Should we do digits ? professor b: let 's do digits <doc-sep> , I can say about just q just quickly to get through it , that Dave and I submitted this ASRU .  , basically we 're dealing with rever reverberation , and , when we deal with pure reverberation , the technique he 's using works really , really well .  , and when they had the reverberation here , we 'll measure the signal - to - noise ratio and it 's , about nine DB . professor b: phd a: You mean , from the actual , recordings ? professor b: a fair amount of phd d: k phd a: It 's nine DB ? professor b: Yeah .  , I know that when you figured out the filters that we 're using for the Mel scale , there was some experimentation that went on at at ,  at OGI .  , but one of the differences that we found between the two systems that we were using , the the Aurora HTK system baseline system and the system that we were the the  , other system we were using , the  , the SRI system , was that the SRI system had maybe a , hundred hertz high - pass . professor b: For some reason , Dave thought it was twenty , phd d:  So the , center would be somewhere around like hundred professor b: but . professor b: But do you know , for instance , h how far down it would be at twenty hertz ? What the how much rejection would there be at twenty hertz , let 's say ? phd d: At twenty hertz . professor b: Yeah , any idea what the curve looks like ? phd d: Twenty hertz frequency Oh , it 's it 's zero at twenty hertz , right ? The filter ? phd c: Yea - actually , the left edge of the first filter is at sixty - four . professor b: It 's actually set to zero ? What kind of filter is that ? phd c: Yeah . professor b: Is this oh , from the from phd c: It This is the filter bank in the frequency domain that starts at sixty - four . professor b: Oh , so you ,  so you really set it to zero , the FFT ? phd d: Yeah , phd c: Yeah . So I wonder , is it @ @ Was there their experimentation with , say , throwing away that filter or something ? And ,  phd d: throwing away the first ? professor b: Yeah . professor b: Right , but the question is , whether sixty - four hertz is is , too , low . phd d: I t I think I 've tried a hundred and it was more or less the same , or slightly worse . professor b: On what test set ? phd d: On the same , SpeechDat - Car , Aurora . professor b: phd d: So it was professor b: and on and on the , TI - digits also ? phd d: No , no , no . That 'd be something to look at sometime because what , eh , he was looking at was performance in this room . professor b: Would that be more like Well , you 'd think that 'd be more like SpeechDat - Car , I guess , in terms of the noise .  , the signal - to - noise ratio , you know , looks a fair amount better if you if you high - pass filter it from this room . professor b: So , the main the the phd a: So that 's on th that 's on the f the far field ones though , right ? Yeah . phd a: So wha what is ,  what 's causing that ? professor b: Well , we got a a video projector in here , and ,  which we keep on during every every session we record , phd a: Yeah . professor b: which , you know , I I w we were aware of phd a:  - huh . professor b: But but ,  So , those are those are major components , I think , phd a: I see . professor b: but , it ,  I guess , I maybe I said this last week too but it it it really became apparent to us that we need to to take account of noise . And , so I think when when he gets done with his prelim study I think one of the next things we 'd want to do is to take this , noise , processing stuff and and , synthesize some speech from it . phd a: When are his prelims ? professor b: And then  , I think in about , a little less than two weeks .  , let 's see , this is the sixteenth , seventeenth ? Yeah , I don't know if he 's before It might even be in a week . I I guessed that they were gonna do it some time during the semester professor b: week and a half . phd a: but they 'll do it any time , huh ? professor b: They seem to be Well , the semester actually is starting up . phd a: Is it already ? professor b: Yeah , the semester 's late late August they start here .  , that that was sort of one  , the overall results seemed to be first place in in in the case of either , artificial reverberation or a modest sized training set . And But if you had a a really big training set , a recognizer , system that was capable of taking advantage of a really large training set I thought that One thing with the HTK is that is has the as we 're using the configuration we 're using is w s is being bound by the terms of Aurora , we have all those parameters just set as they are . So even if we had a hundred times as much data , we wouldn't go out to , you know , ten or t or a hundred times as many Gaussians or anything .  , so , that ,  that seemed to be So , if you have that that better recognizer that can that can build up more parameters , and if you , have the natural room , which in this case has a p a pretty bad signal - to - noise ratio , then in that case , the right thing to do is just do u use speaker adaptation . But I think that that would not be true if we did some explicit noise - processing as well as , the convolutional kind of things we were doing . phd d: That sample was released only yesterday or the day before , right ? phd a: No Well , I haven't grabbed that one yet . phd d: Oh , there is another short sample set phd a: There was another short one , yeah . phd a: And so I haven't grabbed the latest one that he just , put out yet . professor b: Is there any word yet about the issues about , adjustments for different feature sets or anything ? phd a: No , I I d You asked me to write to him and I think I forgot to ask him about that . phd d: Cuz they have it phd a: Maybe I 'll send it to the list . phd d: Cuz they have , already frozen those in i insertion penalties and all those stuff is what I feel . phd d: And , on that , they have run some experiments using various insertion penalties and all those phd a: And so they 've picked the values . professor b: For r w what test set ? phd d: p the one that they have reported is a NIST evaluation , Wall Street Journal . professor b: But that has nothing to do with what we 're testing on , right ? phd c:   So they 're , like  So they are actually trying to , fix that those values using the clean , training part of the Wall Street Journal . phd d: they want to train it and then this they 're going to run some evaluations . professor b: So they 're set they 're setting it based on that ? phd d: Yeah . So now , we may come back to the situation where we may be looking for a modification of the features to account for the fact that we can't modify these parameters . professor b:  but it 's still worth , I think , just since you know , just chatting with Joe about the issue . Do you think that 's something I should just send to him professor b:  phd a: or do you think I should send it to this there 's an a m a mailing list .  , we 're , you know , certainly willing to talk about it with everybody , but I think I think that , it 's probably best to start talking with him just to phd a: OK . professor b:  @ @ you know , it 's a dialogue between two of you about what you know , what does he think about this and what what you know what could be done about it . professor b: if you get ten people in involved in it there 'll be a lot of perspectives based on , you know , how phd a: Yeah . professor b: but if if if there is any , way to move in a way that would that would , you know , be more open to different kinds of features . But if if ,  if there isn't , and it 's just kind of shut down and and then also there 's probably not worthwhile bringing it into a larger forum where where political issues will come in . phd d: Because he there was some mail r saying that it 's may not be stable for Linux and all those . phd d: SUSI phd a: Yeah , SUSI or whatever it was phd d: yeah . professor b: but , I noticed , just glancing at the , Hopkins workshop , web site that ,  one of the thing I don't know Well , we 'll see how much they accomplish , but one of the things that they were trying to do in the graphical models thing was to put together a a , tool kit for doing ,  r  , arbitrary graphical models for , speech recognition . professor b: So And Jeff ,  the two Jeffs were phd a: Who 's the second Jeff ? professor b:  Oh , do you know Geoff Zweig ? phd a: No . professor b: so he did he did his PHD on dynamic Bayes - nets , for for speech recognition . He had some continuity built into the model , presumably to handle some , inertia in the in the production system , and ,  phd a:  .  , so it 's exactly the same approach , but the features that the VAD neural network use are , MFCC after noise compensation . professor b: What was it using before ? phd c: Before it was just P L phd d:  phd c: So . phd c: Yeah , phd d: Yeah , yeah , yeah , phd c: noisy noisy features . phd c:  This is what we get after This So , actually , we , yeah , here the features are noise compensated and there is also the LDA filter .  , and then it 's a pretty small neural network which use , nine frames of of six features from C - zero to C - fives , plus the first derivatives . phd a: Is that nine frames u s  , centered around the current frame ? Or phd c: Yeah . professor b: S so , I 'm I 'm sorry , there 's there 's there 's how many how many inputs ? phd c: So it 's twelve times nine . So I guess about eleven thousand parameters , which actually shouldn't be a problem , even in in small phones . phd a: So , I 'm I 'm s so what is different between this and and what you phd c: It should be OK . So the previous syst It 's based on the system that has a fifty - three point sixty - six percent improvement . The only thing that changed is the n a p eh a es the estimation of the silence probabilities . phd c:  So it 's it 's not bad , but the problem is still that the latency is too large . professor b: What 's the latency ? phd c: Because  the the latency of the VAD is two hundred and twenty milliseconds . And , the VAD is used  , i for on - line normalization , and it 's used before the delta computation . So if you add these components it goes t to a hundred and seventy , right ? professor b: I I 'm confused . You started off with two - twenty and you ended up with one - seventy ? phd c: With two an two hundred and seventy . phd c: If Yeah , if you add the c delta comp delta computation professor b: Oh . I the is this are these twenty - millisecond frames ? Is that why ? Is it after downsampling ? or phd c: The two - twenty is one hundred milliseconds for the  No , it 's forty milliseconds for t for the , cleaning of the speech . phd c: Yeah , and there is the delta at the input which is , professor b: One hundred milliseconds for smoothing . phd c:  phd d: It 's like forty plus forty plus professor b: And then forty phd c: Mmm . phd c: Yeah , there are twenty that comes from There is ten that comes from the LDA filters also . phd d: If you are using professor b:  phd c: Plus the frame , phd d: t If you are using three frames phd c: so it 's two - twenty . phd d: If you are phrasing f using three frames , it is thirty here for delta . So it 's like s five , six cepstrum plus delta at nine nine frames of professor b: And then ten milliseconds for phd d: Fi - There 's an LDA filter . professor b: ten milliseconds for LDA filter , and t and ten another ten milliseconds you said for the frame ? phd c: For the frame I guess . I computed two - twenty Yeah , well , it 's I guess it 's for the fr the professor b: OK . And then there 's delta besides that ? phd c: So this is the features that are used by our network and then afterwards , you have to compute the delta on the , main feature stream , professor b: OK . phd c: which is  , delta and double - deltas , which is fifty milliseconds . No , the after the noise part , the forty the the other hundred and eighty Well , Wait a minute . Some of this is ,  is ,  is in parallel , isn't it ?  , the LDA Oh , you have the LDA as part of the V D -  , VAD ? Or phd c: The VAD use , LDA filtered features also . professor b: so the delta at the end is how much ? phd c: It 's fifty . So phd c: But well , we could probably put the delta , before on - line normalization . It should not that make a big difference , phd a: What if you used a smaller window for the delta ? phd c: because phd a: Could that help a little bit ?  , I guess there 's a lot of things you could do to phd c: Yeah . phd c: Yeah , professor b: So phd c: but , nnn professor b: Yeah . So if you if you put the delta before the , ana on - line If Yeah phd c:   phd c: Cuz i professor b: And then y then you don't have that additive phd c: Yeah , phd d: Yep . phd c: cuz the time constant of the on - line normalization is pretty long compared to the delta window , professor b: OK . And you ought to be able to shove tw ,  sh  pull off twenty milliseconds from somewhere else to get it under two hundred , right ?  phd a: Is two hundred the d professor b: The hundred milla phd c:   It could be eighty and and probably do @ @ phd c: Yeah , phd a: i a hun phd c: yeah . phd a:  Wh - what 's the baseline you need to be under ? Two hundred ? professor b: Well , we don't know . professor b: if it 's two if if it 's ,  if it 's two - fifty , then we could keep the delta where it is if we shaved off twenty . If it 's two hundred , if we shaved off twenty , we could we could , meet it by moving the delta back . phd a: So , how do you know that what you have is too much if they 're still deciding ? professor b: we don't , but it 's just  , the main thing is that since that we got burned last time , and you know , by not worrying about it very much , we 're just staying conscious of it . professor b: And so , th  , if if if a week before we have to be done someone says , " Well , you have to have fifty milliseconds less than you have now " , it would be pretty frantic around here . professor b:  phd a: But still , that 's that 's a pretty big , win . And it doesn't seem like you 're in terms of your delay , you 're , that professor b: He added a bit on , I guess , because before we were we were had were able to have the noise , stuff , and the LVA be in parallel . phd c: Well , but I think the main thing , maybe , is the cleaning of the speech , which takes forty milliseconds or so . phd d: Yeah , the LDA LDA we don't know , is , like is it very crucial for the features , right ? phd c: No . professor b: so you could start pulling back , phd d: S s h professor b: but phd d: Yeah , professor b: But I think you have phd d: l professor b: you have twenty for delta computation which y now you 're sort of doing twice , right ? But yo w were you doing that before ? phd c: Mmm . Well , in the proposal , the input of the VAD network were just three frames , I think . professor b: So , what you have now is fort  , forty for the the noise , twenty for the delta , and ten for the LDA . That 's seventy milliseconds of stuff which was formerly in parallel , phd c:  professor b: right ? So I think , phd c:   professor b: you know , that 's that 's the difference as far as the timing , right ? phd c: Yeah . professor b: and you could experiment with cutting various pieces of these back a bit , but  , we 're s we 're not we 're not in terrible shape . professor b: It 's it 's not like it 's adding up to four hundred milliseconds or something . phd a: Where where is this where is this fifty - seven point O two in in comparison to the last evaluation ? professor b: Well , it 's I think it 's better than anything , anybody got . Yeah , and r and phd c: It would phd d: Yeah , so this is this is like the first proposal .  , I 'm sure other people working on this are not sitting still either , but phd a: Yeah . professor b: but but , the important thing is that we learn how to do this better , and , you know . So , our ,  Yeah , you can see the kind of kind of numbers that we 're having , say , on SpeechDat - Car which is a hard task , cuz it 's really ,  I think it 's just sort of sort of reasonable numbers , starting to be . Yeah , even for a well - matched case it 's sixty percent error rate reduction , professor b: Yeah . So actually , this is in between what we had with the previous VAD and what Sunil did with an IDL VAD . Which gave sixty - two percent improvement , right ? phd d: Yeah , it 's almost that . phd a: What was that ? Say that last part again ? phd c: So , if you use , like , an IDL VAD , for dropping the frames , phd d: o o Or the best we can get . phd c: the best that we can get i That means that we estimate the silence probability on the clean version of the utterances . phd a: So that would be even That wouldn't change this number down here to sixty - two ? phd c: Yeah . So you you were get phd c: If you add a g good v very good VAD , that works as well as a VAD working on clean speech , phd a: Yeah . phd c: then you wou you would go phd a: So that 's sort of the best you could hope for . professor b: And ,  and sixty - two with the the , you know , quote , unquote , cheating VAD . phd c: yeah , the next thing is , I started to play Well , I don't want to worry too much about the delay , no . And the other stream is the output of a neural network , using as input , also , these , cleaned MFCC .  phd a: Those are th those are th what is going into the tandem net ? phd c: I don't have the comp Mmm ? phd a: Those two ? phd c: So there is just this feature stream , the fifteen MFCC plus delta and double - delta . phd a: Yeah ? phd c: so it 's makes forty - five features that are used as input to the HTK . professor b: cuz then it has one part that 's discriminative , phd c: Yeah . Right now it seems that i I just tested on SpeechDat - Car while the experiment are running on your on TI - digits . Well , it improves on the well - matched and the mismatched conditions , but it get worse on the highly mismatched .  , professor b: y phd c: like , on the well - match and medium mismatch , the gain is around five percent relative , but it goes down a lot more , like fifteen percent on the HM case . professor b: You 're just using the full ninety features ? phd c:  The professor b: Y you have ninety features ? phd c: i I have ,  From the networks , it 's twenty - eight . It 's i i i It 's because it 's what we did for the first proposal . phd c: But we have to for sure , we have to go down , because the limit is now sixty features .  phd a: So , it seems funny that I don't know , maybe I don't u quite understand everything , but that adding features I guess I guess if you 're keeping the back - end fixed . But I guess if you 're keeping the number of Gaussians fixed in the recognizer , then professor b: Well , yeah . professor b: But , just in general , adding information Suppose the information you added , well , was a really terrible feature and all it brought in was noise . professor b: Right ? So so ,  Or or suppose it wasn't completely terrible , but it was completely equivalent to another one feature that you had , except it was noisier . professor b: Right ? In that case you wouldn't necessarily expect it to be better at all . I 'm just surprised that you 're getting fifteen percent relative worse on the wel professor b:  - huh . professor b: So , " highly mismatched condition " means that in fact your training is a bad estimate of your test . professor b: So having having , a g a l a greater number of features , if they aren't maybe the right features that you use , certainly can e can easily , make things worse . If you have if you have , lots and lots of data , and you have and your your your training is representative of your test , then getting more sources of information should just help . professor b: So I wonder , Well , what 's your what 's your thought about what to do next with it ? phd c: I don't know . I 'm surprised , because I expected the neural net to help more when there is more mismatch , as it was the case for the professor b:   phd c: Yeah , it 's the same training set , so it 's TIMIT with the TI - digits ' , noises , added . phd c:  professor b: Well , we might  , we might have to experiment with ,  better training sets . professor b: I The other thing is , before you found that was the best configuration , but you might have to retest those things now that we have different The rest of it is different , right ? So , For instance , what 's the effect of just putting the neural net on without the o other other path ? phd c:   professor b: You don't necessarily know what phd a: What if you did the Would it make sense to do the KLT on the full set of combined features ? Instead of just on the phd c: Yeah . The reason I did it this ways is that in February , it we we tested different things like that , so , having two KLT , having just a KLT for a network , or having a global KLT . phd c: And phd a: So you tried the global KLT before phd c: Well phd a: and it didn't really phd c: Yeah . phd c: The differences between these configurations were not huge , but it was marginally better with this configuration . professor b: And I guess if the These are all so all of these seventy - three features are going into , the ,  the  . professor b: And is are i i are are any deltas being computed of tha of them ? phd c: Of the straight features , yeah . phd c: So , yeah , maybe we can add some context from these features also as Dan did in in his last work . i Yeah , but the other thing I was thinking was , now I lost track of what I was thinking . phd a: What is the You said there was a limit of sixty features or something ? phd c:   phd a: What 's the relation between that limit and the , forty - eight  , forty eight hundred bits per second ? professor b: Oh , I know what I was gonna say . phd a: So I I I don't understand , phd c: The f the forty - eight hundred bits is for transmission of some features . phd a: because i  , if you 're only using h phd c: And generally , i it s allows you to transmit like , fifteen , cepstrum . professor b: The issue was that , this is supposed to be a standard that 's then gonna be fed to somebody 's recognizer somewhere which might be , you know , it it might be a concern how many parameters are use u used and so forth . What I was going to say is that , maybe maybe with the noise removal , these things are now more correlated . So you have two sets of things that are kind of uncorrelated , within themselves , but they 're pretty correlated with one another . professor b: And , they 're being fed into these , variants , only Gaussians and so forth , and and , phd c:   professor b: so maybe it would be a better idea now than it was before to , have , one KLT over everything , to de - correlate it . phd d: What are the S N Rs in the training set , TIMIT ? phd c: It 's , ranging from zero to clean ? Yeah . So we found this this ,  this Macrophone data , and so forth , that we were using for these other experiments , to be pretty good . professor b: So that 's i after you explore these other alternatives , that might be another way to start looking , is is just improving the training set . professor b: we were getting , lots better recognition using that , than Of course , you do have the problem that , u i we are not able to increase the number of Gaussians , or anything to , to match anything . So we 're only improving the training of our feature set , but that 's still probably something . phd a: So you 're saying , add the Macrophone data to the training of the neural net ? The tandem net ? professor b: Yeah , that 's the only place that we can train . professor b: We can't train the other stuff with anything other than the standard amount , phd a: Right .  ,  phd a: What what was it trained on again ? The one that you used ? phd c: It 's TIMIT with noise . phd c: So , yeah , it 's rather a small professor b: How big is the net , by the way ? phd c: it 's , five hundred hidden units . And professor b: And again , you did experiments back then where you made it bigger and it and that was that was sort of the threshold point . @ @ ? phd d: So is it is it though the performance , big relation in the high ma high mismatch has something to do with the , cleaning up that you that is done on the TIMIT after adding noise ? phd c:  phd d: So it 's i All the noises are from the TI - digits , phd c: Yeah . phd d: right ? So you i phd c:  They k  phd d: Well , it it 's like the high mismatch of the SpeechDat - Car after cleaning up , maybe having more noise than the the training set of TIMIT after clean s after you do the noise clean - up . phd d: So it had like all these different conditions of S N Rs , actually in their training set of neural net . phd d: But after cleaning up you have now a different set of S N Rs , right ? phd c: Yeah . phd d: And is it something to do with the mismatch that that 's created after the cleaning up , like the high mismatch phd c: You mean the the most noisy occurrences on SpeechDat - Car might be a lot more noisy than phd d:   professor b: but , you 're saying Yeah , the noisier ones are still going to be , even after our noise compensation , are still gonna be pretty noisy . phd d: Yeah , so now the after - noise compensation the neural net is seeing a different set of S N Rs than that was originally there in the training set . phd d: And that SNR may not be , like , com covering the whole set of S N Rs that you 're getting in the SpeechDat - Car . professor b: Right , but the SpeechDat - Car data that you 're seeing is also reduced in noise by the noise compensation . phd c: Well , if the initial range of SNR is different , we the problem was already there before . phd c: Because Mmm professor b: Yeah , it depends on whether you believe that the noise compensation is equally reducing the noise on the test set and the training set .  professor b: Right ?  , you 're saying there 's a mismatch in noise that wasn't there before , phd d:  . professor b: but if they were both the same before , then if they were both reduic reduced equally , then , there would not be a mismatch . professor b: So , this may be Heaven forbid , this noise compensation process may be imperfect , but . I I just that could be seen from the TI - digits , testing condition because , the noises are from the TI - digits , right ? Noise phd c: Yeah . So phd d: So cleaning up the TI - digits and if the performance goes down in the TI - digits mismatch high mismatch like this phd c: Clean training , yeah . professor b: the Macrophone data , I think , you know , it was recorded over many different telephones . I don't think there 's anybody recording over a car from a car , but I think it 's it 's varied enough that if if doing this adjustments , and playing around with it doesn't , make it better , the most  , it seems like the most obvious thing to do is to improve the training set .   , what we were  the condition It it gave us an enormous amount of improvement in what we were doing with Meeting Recorder digits , even though there , again , these m Macrophone digits were very , very different from , what we were going on here . But it was just I think just having a a nice variation in acoustic conditions was just a good thing . phd c: Yeah , actually to s eh , what I observed in the HM case is that the number of deletion dramatically increases . phd c: When I added the num the neural network it doubles the number of deletions . Yeah , so I don't you know how to interpret that , but , mmm professor b: Yeah . phd c: t phd a: And and did an other numbers stay the same ? Insertion substitutions stay the same ? phd c: They p stayed the same , phd a: Roughly ? phd c: they maybe they are a little bit  , lower . But professor b: Did they increase the number of deletions even for the cases that got better ? phd c:   professor b: And it Remind me again , the " highly mismatched " means that the phd c: Clean training and professor b: sorry ? phd c: It 's clean training Well , close microphone training and distant microphone , high speed , I think . professor b: Close mike training phd c: Well The most noisy cases are the distant microphone for testing . phd c: The feature are the same except that professor b: that 's right , that 's right .  phd a: Well that that says that , you know , the ,  the models in in , the recognizer are really paying attention to the neural net features . professor b: But , yeah , actually the TIMIT noises are sort of a range of noises and they 're not so much the stationary driving kind of noises , right ? It 's it 's pretty different . " phd c: " Subway " , right ? and phd d: " Street " or " Airport " or something . phd c: So it 's mostly Well , " Car " is stationary , professor b:   phd c: " Babble " , it 's a stationary background plus some voices , professor b:   professor b: Well , I I think that if you run it Actually , you maybe you remember this . When you in in the old experiments when you ran with the neural net only , and didn't have this side path , with the the pure features as well , did it make things better to have the neural net ? phd c:   professor b: Was it about the same ?  , w i phd c: It was b a little bit worse . professor b: So , until you put the second path in with the pure features , the neural net wasn't helping at all . as soon as we added LDA on - line normalization , and all these things , then professor b: They were doing similar enough things . Well , I still think it would be k sort of interesting to see what would happen if you just had the neural net without the side thing . phd c: Yeah , professor b: And and the thing I I have in mind is , maybe you 'll see that the results are not just a little bit worse . You know ? And ,  But if on the ha other hand , it 's , say , somewhere in between what you 're seeing now and and and , what you 'd have with just the pure features , then maybe there is some problem of a of a , combination of these things , or correlation between them somehow . professor b: If it really is that the net is hurting you at the moment , then I think the issue is to focus on on , improving the the net . professor b: So what 's the overall effe  , you haven't done all the experiments but you said it was i somewhat better , say , five percent better , for the first two conditions , and fifteen percent worse for the other one ? But it 's but of course that one 's weighted lower , phd c: Y yeah , oh . That 's not that bad , but it was l like two percent relative worse on SpeechDat - Car . phd d: Well , it will overall it will be still better even if it is fifteen percent worse , because the fifteen percent worse is given like f w twenty - five point two five eight . So the so the worst it could be , if the others were exactly the same , is four , phd d: Is it like professor b: and and , in fact since the others are somewhat better phd d: Yeah , so it 's four . Is i So either it 'll get cancelled out , or you 'll get , like , almost the same . phd a: In the ,  a lot of the ,  the Hub - five systems , recently have been using LDA . So there 's the the LDA is is right there before the H M phd d: Yeah . phd a: So , you guys are using LDA but it seems like it 's pretty far back in the process . The LDA that you saying is , like , you take a block of features , like nine frames or something , and then do an LDA on it , phd a: Yeah . phd a: it 's you know , you 're just basically i phd d: Yeah , so this is like a two d two dimensional tile . And the LDA that we are f applying is only in time , not in frequency high cost frequency . So it 's like more like a filtering in time , rather than doing a r phd a: Ah . So what i what about ,  i u what i w  , I don't know if this is a good idea or not , but what if you put ran the other kind of LDA , on your features right before they go into the  ? phd d: it phd c:   phd a: But  , w but the other features that you have , th the non - tandem ones , phd c:  . Well , in the proposal , they were transformed u using PCA , but phd a:  - huh . professor b: The a the argument i is kind of i in and it 's not like we really know , but the argument anyway is that , we always have the prob  , discriminative things are good . professor b: they 're good because you you you learn to distinguish between these categories that you want to be good at distinguishing between . It PAC - PCA low - order PCA throws away pieces that are  , maybe not not gonna be helpful just because they 're small , basically . professor b: But , the problem is , training sets aren't perfect and testing sets are different . So you f you you face the potential problem with discriminative stuff , be it LDA or neural nets , that you are training to discriminate between categories in one space but what you 're really gonna be g getting is is something else . professor b: And so , Stephane 's idea was , let 's feed , both this discriminatively trained thing and something that 's not . So you have a good set of features that everybody 's worked really hard to make , phd a: Yeah . professor b: and then , you you discriminately train it , but you also take the path that that doesn't have that , phd a:  - huh . And that that seem So it 's kind of like a combination of the  , what , Dan has been calling , you know , a feature  , you know , a feature combination versus posterior combination or something . It 's it 's , you know , you have the posterior combination but then you get the features from that and use them as a feature combination with these these other things . And that seemed , at least in the last one , as he was just saying , he he when he only did discriminative stuff , i it actually was was it didn't help at all in this particular case . But by having them both there The fact is some of the time , the discriminative stuff is gonna help you . professor b: And some of the time it 's going to hurt you , phd a: Right . professor b: and by combining two information sources if , you know if if phd a: So you wouldn't necessarily then want to do LDA on the non - tandem features because now you 're doing something to them that professor b: That i i I think that 's counter to that idea . But if that 's the hypothesis , at least it would be counter to that hypothesis to do that . professor b: and in principle you would think that the neural net would do better at the discriminant part than LDA .  , we ,  we were getting ready to do the tandem , stuff for the Hub - five system , and , Andreas and I talked about it , and the idea w the thought was , " Well , yeah , that i you know th the neural net should be better , but we should at least have  , a number , you know , to show that we did try the LDA in place of the neural net , so that we can you know , show a clear path . phd a: You know , that you have it without it , then you have the LDA , then you have the neural net , and you can see , theoretically . professor b: or tha that 's a phd a: That 's what that 's what we 're gonna do next as soon as I finish this other thing . phd a: it everybody believes it , professor b: Oh , no it 's a g phd a: but you know , we just professor b: No , no , but it might not not even be true . professor b: it 's it 's it 's it 's it 's a great idea .  , one of the things that always disturbed me , in the the resurgence of neural nets that happened in the eighties was that , a lot of people Because neural nets were pretty easy to to use a lot of people were just using them for all sorts of things without , looking at all into the linear , versions of them . professor b: And , people were doing recurrent nets but not looking at IIR filters , and You know , so I think , yeah , it 's definitely a good idea to try it . phd a: Yeah , and everybody 's putting that on their systems now , and so , I that 's what made me wonder about this , professor b: Well , they 've been putting them in their systems off and on for ten years , phd a: but . professor b: but but but , phd a: Yeah , what  is it 's it 's like in the Hub - five evaluations , you know , and you read the system descriptions and everybody 's got , you know , LDA on their features . phd c: It 's the transformation they 're estimating on Well , they are trained on the same data as the final  are . Cuz they don't have these , you know , mismatches that that you guys have . phd a: So that 's why I was wondering if maybe it 's not even a good idea . professor b: part of why I I think part of why you were getting into the KLT Y you were describing to me at one point that you wanted to see if , you know , getting good orthogonal features was and combining the the different temporal ranges was the key thing that was happening or whether it was this discriminant thing , right ? So you were just trying I think you r  , this is it doesn't have the LDA aspect but th as far as the orthogonalizing transformation , you were trying that at one point , right ? phd c:   phd d: So , yeah , I 've been exploring a parallel VAD without neural network with , like , less latency using SNR and energy , after the cleaning up . So what I 'd been trying was ,  After the b after the noise compensation , n I was trying t to f find a f feature based on the ratio of the energies , that is , cl after clean and before clean . So that if if they are , like , pretty c close to one , which means it 's speech . And if it is n if it is close to zero , which is So it 's like a scale @ @ probability value . So I was trying , with full band and multiple bands , m ps  separating them to different frequency bands and deriving separate decisions on each bands , and trying to combine them .  , the advantage being like it doesn't have the latency of the neural net if it if it can professor b:   phd d: just like Which means that it 's it 's doing a slightly better job than the previous VAD , professor b:   phd d: so ,  professor b: But i d I 'm sorry , phd d: so u professor b: does it still have the median filter stuff ? phd d: It still has the median filter . professor b: So it still has most of the delay , phd d: So professor b: it just doesn't phd d: Yeah , so d with the delay , that 's gone is the input , which is the sixty millisecond . professor b: Well , w i phd d: At the input of the neural net you have this , f nine frames of context plus the delta . phd d: so the delay is only the forty millisecond of the noise cleaning , plus the hundred millisecond smoothing at the output . So the the di the biggest The problem f for me was to find a consistent threshold that works well across the different databases , because I t I try to make it work on tr SpeechDat - Car professor b:   phd d: and it fails on TI - digits , or if I try to make it work on that it 's just the Italian or something , it doesn't work on the Finnish . So there are there was , like , some problem in balancing the deletions and insertions when I try different thresholds . phd d: So The I 'm still trying to make it better by using some other features from the after the p clean up maybe , some , correlation auto - correlation or some s additional features of to mainly the improvement of the VAD . professor b: Now this this this , " before and after clean " , it sounds like you think that 's a good feature . That that , it you th think that the ,  the i it appears to be a good feature , right ? phd d:   phd c: Yeah , eventually we could could just phd d: Yeah , so Yeah , so that 's the Yeah . phd d: Because they did that itself phd c: Then you don't have to worry about the thresholds and phd d: There 's a threshold and Yeah . So if we if we can live with the latency or cut the latencies elsewhere , then then that would be a , good thing . professor b: anybody has anybody you guys or or Naren , somebody , tried the , second th second stream thing ?  . phd d: Oh , I just I just h put the second stream in place and ,  ran one experiment , but just like just to know that everything is fine . phd d: So it was like , forty - five cepstrum plus twenty - three mel log mel . phd d: And and , just , like , it gave me the baseline performance of the Aurora , which is like zero improvement . phd d: So I just tried it on Italian just to know that everything is But I I didn't export anything out of it because it was , like , a weird feature set . Well , what I think , you know , would be more what you 'd want to do is is is , put it into another neural net . phd d: The  , other thing I was wondering was , if the neural net , has any because of the different noise con unseen noise conditions for the neural net , where , like , you train it on those four noise conditions , while you are feeding it with , like , a additional some four plus some f few more conditions which it hasn't seen , actually , phd c:   phd d:  instead of just h having c  , those cleaned up t cepstrum , sh should we feed some additional information , like The the We have the VAD flag .  , should we f feed the VAD flag , also , at the input so that it it has some additional discriminating information at the input ? phd c:  -  !  professor b: Wh -  , the the VAD what ? phd d: We have the VAD information also available at the back - end . phd d: So if it is something the neural net is not able to discriminate the classes professor b: Yeah . phd d:  Because most of it is sil  , we have dropped some silence f We have dropped so silence frames ? professor b:   So , by having an additional , feature which says " this is speech and this is nonspeech " , it certainly helps in some unseen noise conditions for the neural net . phd a: What Do y do you have that feature available for the test data ? phd d: Well , we have we are transferring the VAD to the back - end feature to the back - end . phd d: Which is which is certainly giving a phd a: So you 're saying , feed that , also , into the neural net . The other thing you could do is just , p modify the , output probabilities of the of the , neural net , tandem neural net , based on the fact that you have a silence probability . professor b: So you have an independent estimator of what the silence probability is , and you could multiply the two things , and renormalize . professor b: But but ,  phd c: Yeah , so maybe , yeah , when phd a: But in principle wouldn't it be better to feed it in ? And let the net do that ? professor b: Well , u Not sure . " Or you can say , " It 's silence ! Go away ! "  , i Doesn't ? I think I think the second one sounds a lot more direct . So , what if you then ,  since you know this , what if you only use the neural net on the speech portions ? professor b: Well , phd c: That 's what phd a: Well , I guess that 's the same . professor b: Yeah , y you 'd have to actually run it continuously , phd a: But   , train the net only on professor b: but it 's @ @ Well , no , you want to train on on the nonspeech also , because that 's part of what you 're learning in it , to to to generate , that it 's it has to distinguish between . phd a: But  , if you 're gonna if you 're going to multiply the output of the net by this other decision , would then you don't care about whether the net makes that distinction , right ? professor b: Well , yeah . Now the only thing that that bothers me about all this is that I I I The the fact i i It 's sort of bothersome that you 're getting more deletions . But So I might maybe look at , is it due to the fact that  , the probability of the silence at the output of the network , is , professor b: Is too high . So maybe So phd c: If it 's the case , then multiplying it again by i by something ? phd d: It may not be it professor b: Yeah . phd d: Yeah , it it may be too it 's too high in a sense , like , everything is more like a , flat probability . phd d: So , like , it 's not really doing any distinction between speech and nonspeech phd c: yeah . phd a: Be interesting to look at the Yeah , for the I wonder if you could do this . But if you look at the , highly mism high mismat the output of the net on the high mismatch case and just look at , you know , the distribution versus the the other ones , do you do you see more peaks or something ? phd c: Yeah . phd c: or professor b: But I bu phd c: It it seems that the VAD network doesn't Well , it doesn't drop , too many frames because the dele the number of deletion is reasonable . But it 's just when we add the tandem , the final MLP , and then professor b: Yeah . Now the only problem is you don't want to ta I guess wait for the output of the VAD before you can put something into the other system , phd c: u professor b: cuz that 'll shoot up the latency a lot , right ? Am I missing something here ? phd c: But phd d:   But but I I guess phd a: But if you were gonna put it in as a feature it means you already have it by the time you get to the tandem net , right ? phd d: well . professor b: It 's kind of done in  , some of the things are , not in parallel , but certainly , it would be in parallel with the with a tandem net . So maybe , if that doesn't work ,  But it would be interesting to see if that was the problem , anyway . And and and then I guess another alternative would be to take the feature that you 're feeding into the VAD , and feeding it into the other one as well . professor b:  But that 's Yeah , that 's an interesting thing to try to see , if what 's going on is that in the highly mismatched condition , it 's , causing deletions by having this silence probability up up too high , phd c:   phd c: So , m professor b: Cuz Well , the V A if the VAD said since the VAD is is is right a lot ,  phd c: Yeah . Yeah , and the other thing Well , there are other issues maybe for the tandem , like , well , do we want to , w  n Do we want to work on the targets ? Or , like , instead of using phonemes , using more context dependent units ? phd a: For the tandem net you mean ? phd c: Well , I 'm Yeah . phd c: I 'm thinking , also , a w about Dan 's work where he he trained a network , not on phoneme targets but on the  state targets . professor b: Problem is , if you are going to run this on different m test sets , including large vocabulary , phd c: Yeah . I was just thinking maybe about , like , generalized diphones , and come up with a a reasonable , not too large , set of context dependent units , and and Yeah . But I d I d it it i it 's all worth looking at , phd c:   professor b: but it sounds to me like , looking at the relationship between this and the speech noise stuff is is is probably a key thing . phd a: So if ,  if the , high mismatch case had been more like the , the other two cases in terms of giving you just a better performance , how would this number have changed ? phd c:   If if i phd a: y Like sixty ? professor b: Well , we don't know what 's it 's gonna be the TI - digits yet . If you extrapolate the SpeechDat - Car well - matched and medium - mismatch , it 's around , yeah , maybe five . phd c: Well , it 's around five percent , because it 's s Right ? If everything is five percent . phd c: I d I d I just have the SpeechDat - Car right now , so phd a: Yeah . phd c: It 's running it shou we should have the results today during the afternoon , phd a:  . Well  So I won't be here for phd a: When When do you leave ? professor b: I 'm leaving next Wednesday .  , phd a: But you 're professor b: so I phd a: are you you 're not gonna be around this afternoon ? professor b: Yeah . professor b: This afternoon  Oh , right , for the Meeting meeting ? Yeah , that 's just cuz of something on campus . But , yeah , so next week I won't , and the week after I won't , cuz I 'll be in Finland .  , phd a: What 's September sixth ? professor b: and  , that 's during Eurospeech .  Right ? So it 'll be a few weeks , really , before we have a meeting of the same cast of characters . And and then  , we 'll start up again with Dave and Dave and Barry and Stephane and us on the , twentieth . Thirteenth ? About a month ? phd a: So , you 're gonna be gone for the next three weeks or something ? professor b: I 'm gone for two and a half weeks starting starting next Wed - late next Wednesday . Is that right ? professor b: I won't it 's probably four because of is it three ? Let 's see , twenty - third , thirtieth , sixth . And the the third one won't probably won't be a meeting , cuz cuz , Su - Sunil , Stephane , and I will all not be here . So it 's just , the next two where there will be there , you know , may as well be meetings , phd a: OK . And then starting up on the thirteenth , we 'll have meetings again but we 'll have to do without Sunil here somehow . phd a: When is the evaluation ? November , or something ? professor b: Yeah , it was supposed to be November fifteenth <doc-sep>professor c: OK , what are we talking about today ? phd b: I don't know . phd a: Oh , this was the , talk where they were supposed to try to decide phd b: To to decide what to do , phd a: Ah , right .  , so  , let 's let 's assume for right now that we 're just kind of plugging on ahead , phd b: Yeah . professor c: because even if they tell us that , the rules are different , we 're still interested in doing what we 're doing .  , well , we 've a little bit worked on trying to see , what were the bugs and the problem with the latencies . phd d: To improve phd b: So , We took first we took the LDA filters and , we designed new filters , using  recursive filters actually . professor c: So when you say " we " , is that something Sunil is doing or is that ? phd b: I 'm sorry ? professor c: Who is doing that ? phd b: us . phd b: So we took the filters the FIR filters and we designed , IIR filters that have the same frequency response . phd b: So they had two filters , one for the low frequency bands and another for the high frequency bands . And the low frequency band has sixty - four milliseconds of delay , and the high frequency band filter has something like eleven milliseconds compared to the two hundred milliseconds of the IIR filters . So we have the filters but we still have to implement a routine that does recursive filtering professor c: OK . phd b: and professor c: You you had a discussion with Sunil about this though ? phd b: No . professor c: right ? So so you need to discuss with him what we 're doing , phd b: Yeah .  , I yeah , I don't know if th that 's what they were trying to They were trying to do something different like taking ,  well , using filter that takes only a past professor c: Right . I think that the the fact that we we did that with had that thing with the latencies was indicative of the fact that there wasn't enough communication . Well , there is w one , remark about these filters , that they don't have a linear phase . phd b: Well , I don't know , perhaps it perhaps it doesn't hurt because the phase is almost linear but .  , and so , yeah , for the delay I gave you here , it 's it 's , computed on the five hertz modulation frequency , which is the mmm , well , the most important for speech so . professor c: So that would be , a reduction of a hundred and thirty - six milliseconds , phd d: The low f f phd b: Yeah . professor c: which ,  What was the total we ended up with through the whole system ? phd b: Three hundred and thirty . professor c: So that would be within ? phd b: Yeah , but there are other points actually , which will perhaps add some more delay . Is that some other other stuff in the process were perhaps not very  perf well , not very correct , like the downsampling which w was simply dropping frames . phd b: so we will try also to add a nice downsampling having a filter that that professor c:  - huh .  , because wh when when we look at the LDA filters , well , they are basically low - pass but they leave a lot of what 's above twenty - five hertz . phd b: and so , yeah , this will be another filter which would add ten milliseconds again . phd b: yeah , and then there 's a third thing , is that , basically the way on - line normalization was done  , is just using this recursion on on the  , on the feature stream , professor c: Yeah .  , and when we look at this filter actually it has a delay of eighty - five milliseconds . If we want to be very correct , so if we want to the estimation of the mean t t to to be well , the right estimation of the mean , we have to t to take eighty - five milliseconds in the future . We would be at six so , sixty - five , plus ten , plus for the downsampling , plus eighty - five for the on - line normalization . So it 's professor c: phd b: plus plus eighty for the neural net and PCA . phd b: So it would be around two hundred and forty so , well , professor c: Just just barely in there . phd a: What 's the allowable ? professor c: Two - fifty , unless they changed the rules . phd a: What were they thinking of changing it to ? professor c: But phd b: Yeah . professor c: well the people who had very low latency want it to be low  , very very very narrow , latency bound . professor c: Unfortunately we 're the main ones with long latency , but phd a: Ah ! professor c: But , phd b: Yeah , and basically the best proposal had something like thirty or forty milliseconds of latency . professor c: Yeah , so they were basically  , they were more or less trading computation for performance and we were , trading latency for performance . And they were dealing with noise explicitly and we weren't , and so I think of it as complementary , that if we can put the phd a: Think of it as what ? professor c: Complementary . professor c: I think the best systems so , everything that we did in in a way it was it was just adamantly insisting on going in with a brain damaged system , which is something actually , we 've done a lot over the last thirteen years . So , w th w this was a test that largely had additive noise and we did we adde did absolutely nothing explicitly to handle ad additive noise . And , we did this , RASTA - like filtering which was done in the log domain and was tending to handle convolutional noise . So , the , spectral sub subtraction schemes a couple places did seem to seem to do a nice job . And so , we 're talking about putting putting some of that in while still keeping some of our stuff . I think you should be able to end up with a system that 's better than both but clearly the way that we 're operating for this other stuff does involved some latency to to get rid of most of that latency . To get down to forty or fifty milliseconds we 'd have to throw out most of what we 're doing . And and , I don't think there 's any good reason for it in the application actually .  , you 're you 're you 're speaking to a recognizer on a remote server and , having a a a quarter second for some processing to clean it up . professor c: These aren't large vocabulary things so the decoder shouldn't take a really long time , and . phd a: And I don't think anybody 's gonna notice the difference between a quarter of a second of latency and thirty milliseconds of latency . What what does wa was your experience when you were doing this stuff with , the the the surgical , microscopes and so forth .  , how long was it from when somebody , finished an utterance to when , something started happening ? phd a: we had a silence detector , so we would look for the end of an utterance based on the silence detector . phd a: And I I can't remember now off the top of my head how many frames of silence we had to detect before we would declare it to be the end of an utterance . phd a: but it was , I would say it was probably around the order of two hundred and fifty milliseconds . phd a: So professor c: Yeah , so you you so you had a phd a: this w professor c: so you had a a quarter second delay before , plus some little processing time , phd a: Right . professor c: And there 's physical inertia there , so probably the the motion itself was all phd a: And it felt to , the users that it was instantaneous . professor c: Yeah , so you would think as long as it 's under half a second or something . A person I don't think a person can tell the difference between , you know , a quarter of a second and a hundred milliseconds , and I 'm not even sure if we can tell the difference between a quarter of a second and half a second .  , basically if you yeah , if you said , " what 's the ,  what 's the shortest route to the opera ? " and it took half a second to get back to you , phd a: Yeah . phd a: because when we talk to each other we tend to step on each other 's utterances . So like if I 'm asking you a question , you may start answering before I 'm even done . Well , anyway , I think we could cut we know what else , we could cut down on the neural net time by by , playing around a little bit , going more into the past , or something like that . phd a: So is the latency from the neural net caused by how far ahead you 're looking ? professor c:   professor c: And there 's also well , there 's the neural net and there 's also this , multi - frame , KLT . phd a: Wasn't there Was it in the , recurrent neural nets where they weren't looking ahead at all ? professor c: They weren't looking ahead much . And and then But you also could just , we haven't experimented with this but I imagine you could , predict a , a label , from more in the past than in than than in the future . phd a: but I don't think professor c: Yeah , but we 've but we played a little bit with with asymmetric , guys . So , that 's what that 's what you 're busy with , s messing around with this , phd b: yeah . And , phd d: Also we were thinking to to , apply the eh , spectral subtraction from Ericsson phd b: Yeah . professor c: phd a: What is the advantage of that ? phd d:  phd b: Well , it 's that by the for the moment we have , something that 's discriminant and nonlinear . Well , it 's it 's a linear transformation , that  professor c: So at least just to understand maybe what the difference was between how much you were getting from just putting the frames together and how much you 're getting from the discriminative , what the nonlinearity does for you or doesn't do for you . Actually what we want to do , perhaps it 's to replace to to have something that 's discriminant but linear , also . And to see if it if it improves ov over over the non - discriminant linear transformation . professor c: Yeah , well , that 's what I meant , is to see whether whether it having the neural net really buys you anything . professor c: But maybe it 's just the discrimination and and maybe yeah , maybe the nonlinear discrimination isn't necessary . But the other part you were saying was the spectral subtraction , so you just kind of ,  phd b: Yeah . professor c: At what stage do you do that ? Do you you 're doing that ,  ? phd b: So it would be on the  on on the mel frequency bands , phd d: We was think phd b: so . professor c: OK , phd d: Yeah , professor c: so just do that on the mel f phd d: we no nnn We we was thinking to do before after VAD or phd b: Yeah , phd d: Oh , we don't know exactly when it 's better . phd b:  phd d: Before after VAD or professor c: So so you know that that that the way that they 're phd d: and then phd b:  . professor c: one thing that would be no good to find out about from this conference call is that what they were talking about , what they 're proposing doing , was having a third party , run a good VAD , and and determine boundaries . professor c: The reason for that was that ,  if some one p one group put in the VAD and another didn't , or one had a better VAD than the other since that they 're not viewing that as being part of the the task , and that any any manufacturer would put a bunch of effort into having some s kind of good speech - silence detection . It still wouldn't be perfect but  , e the argument was " let 's not have that be part of this test . " And so , I guess they argued about that yesterday and , yeah , I 'm sorry , I don't don't know the answer but we should find out . So ,  Yeah , so there 's the question of the VAD but otherwise it 's it 's on the the ,  the mel fil filter bank , energies I guess ? phd d:   professor c: And you 're you 're subtracting in the in the in the I guess it 's power power domain , or or magnitude domain . phd b: and professor c: if you look at the theory , it 's it should be in the power domain but but , I 've seen implementations where people do it in the magnitude domain phd b: Yeah . professor c: I have asked people why and they shrug their shoulders and say , " oh , it works . professor c: and there 's this I guess there 's this mysterious  people who do this a lot I guess have developed little tricks of the trade .  , there 's there 's this ,  you don't just subtract the the estimate of the noise spectrum . phd b: so you have the estimation of the power spectra of the noise , and you multiply this by a factor which is depend dependent on the SNR . phd a:  ! phd b: When the speech lev when the signal level is more important , compared to this noise level , the coefficient is small , and around one . But when the power le the s signal level is  small compared to the noise level , the coefficient is more important . And this reduce actually the music musical noise , phd a: Oh ! phd b:  which is more important during silence portions , phd a:  - huh . professor c: Well , that 's  , that 's what differs from different different tasks and different s  , spectral subtraction methods . phd a:  ! professor c: if if you have , fair assurance that , the noise is is quite stationary , then the smartest thing to do is use as much data as possible to estimate the noise , get a much better estimate , and subtract it off . professor c: But if it 's varying at all , which is gonna be the case for almost any real situation , you have to do it on - line , with some forgetting factor or something . phd a: So do you is there some long window that extends into the past over which you calculate the average ? professor c: Well , there 's a lot of different ways of computing the noise spectrum . So one of the things that , Hans - Guenter Hirsch did ,  and pas and other people actually , he 's he wasn't the only one I guess , was to , take some period of of of speech and in each band , develop a histogram . And , in fact I think the NIST standard method of determining signal - to - noise ratio is based on this . phd a: A couple seconds ? professor c: So No , no , it 's based on this kind of method , phd a:  . Now , if you have signal and you have noise , you basically have these two bumps in the histogram , which you could approximate as two Gaussians . professor c: So so basically now you have this mixture of two Gaussians , you you n know what they are , and , sorry , you estimate what they are , and , so this gives you what the signal is and what the noise e energy is in that band in the spectrum . And the other thing to do is which is sort of more trivial and obvious is to , determine through magical means that that , there 's no speech in some period , and then see what the spectrum is .  , a variant on that for just determining signal - to - noise ratio is to just ,  you can do a w a  an iterative thing , EM - like thing , to determine means only . professor c: And then you just use those mean values as being the the ,   signal - to - noise ratio in that band . phd a: But what is the it seems like this kind of thing could add to the latency . Cuz if you don't look into the future , right ? phd a: OK , well that I guess that was my question , professor c: if you just yeah phd a: yeah . professor c: if you just if you you ,  a at the beginning you have some phd a: Guess . professor c: esti some guess and and ,  phd b: Yeah , but it professor c: It 's an interesting question . I wonder how they did do it ? phd b: Actually , it 's a mmm If - if you want to have a good estimation on non - stationary noise you have to look in the in the future .  , if you take your window and build your histogram in this window , what you can expect is to have an estimation of th of the noise in in the middle of the window , not at the end . So professor c: Well , yeah , phd b: the but but people professor c: but what does what what what does Alcatel do ? phd d:   grad e: Pretty stationary , phd b: but ,  professor c: Well , the thing , e e e e grad e: yeah . professor c: Yeah , y  , you 're talking about non - stationary noise but I think that spectral subtraction is rarely is is not gonna work really well for for non - stationary noise , phd b: Well , if y if you have a good estimation of the noise , professor c: you know ? phd b: yeah , because well it it has to work . professor c: But it 's hard to phd b: i professor c: but that 's hard to do . So so I think that that what what is wh what 's more common is that you 're going to be helped with r slowly varying or stationary noise . professor c: If it varies a lot , to get a If if to get a good estimate you need a few seconds of speech , even if it 's centered , right ? phd b:   professor c: if you need a few seconds to get a decent estimate but it 's changed a lot in a few seconds , then it , you know , i it 's kind of a problem . professor c: imagine e five hertz is the middle of the of the speech modulation spectrum , phd b: Mmm . So , phd b: So in this case , yeah , sure , you cannot professor c: Yeah . phd b: But I think y  , Hirsch does experiment with windows of like between five hundred milliseconds and one second .  and he worked on non - stationary noises , like noise modulated with well , wi with amplitude modulations and things like that , phd a: Were his , windows centered around the phd b: and But  , yeah . professor c: No , I understand it 's better to do but I just think that that , for real noises wh what what 's most likely to happen is that there 'll be some things that are relatively stationary phd b: Mmm . professor c: and other things where it 's not so stationary and  , you can always pick something that that falls between your methods , phd b:  . professor c: but I don't know if , you know , if sinusoidally , modul amplitude modulated noise is is sort of a big problem in in in practice . professor c: I think that it 's  phd a: We could probably get a really good estimate of the noise if we just went to the noise files , and built the averages from them . phd b: What What do you mean ? professor c: Just cheat You 're saying , cheat . phd b: But if the if the noise is stationary perhaps you don't even need some kind of noise estimation algorithm . phd b: We just take th th th the beginning of the utterance and professor c: Oh , yeah , sure . phd b: Well , everybody seems to use some kind of adaptive , well , scheme professor c: But but phd d: Yeah . phd b: is it very useful professor c: you know , stationary phd a: Very slow adaptation . phd b: and is the c phd a: th professor c: Right , the word " stationary " is has a very precise statistical meaning . But , you know , in in signal - processing really what we 're talking about I think is things that change slowly , compared with our our processing techniques . professor c: So if you 're driving along in a car I I would think that most of the time the nature of the noise is going to change relatively slowly . If you if you check it out , five minutes later you may be in a different part of the road phd b:   But it 's it 's i i i using the local characteristics in time , is probably going to work pretty well . professor c: But you could get hurt a lot if you just took some something from the beginning of all the speech , of , you know , an hour of speech and then later phd b: Yeah . professor c: so they may be you know , may be overly , complicated for for this test but but but , I don't know .  , if possible you shouldn't you should you should make it , the center of the center of the window .  , phd a: If they 're going to provide a , voice activity detector that will tell you the boundaries of the speech , then , couldn't you just go outside those boundaries and do your estimate there ? professor c: Oh , yeah . So I I imagine that 's what they 're doing , right ? Is they 're they 're probably looking in nonspeech sections and getting some ,  phd b: Yeah , they have some kind of threshold on on the previous estimate , and So . Yeah , so , they h they have an estimate of the noise level and they put a threshold like six or ten DB above , and what 's under this threshold is used to update the estimate . It 's like saying what 's under the threshold is silence , professor c: Does France Telecom do this phd b: and grad e:  . professor c: Does France Telecom do th do the same thing ? More or less ? phd b: I d I Y you know , perhaps ? phd d: No .  , maybe we can talk about a couple other things briefly , just , things that that we 've been chatting about but haven't made it into these meetings yet . So you 're coming up with your quals proposal , and ,  Wanna just give a two three minute summary of what you 're planning on doing ? grad e: Oh , two , three , it can be shorter than that .  , but I 'm , looking into extending the work done by Larry Saul and John Allen and  Mazin Rahim .  , they they have a system that 's , a multi - band , system but their multi - band is is a little different than the way that we 've been doing multi - band in the past , where  Where we 've been @ @  taking  sub - band features and i training up these neural nets and on on phonetic targets , and then combining them some somehow down the line , they 're they 're taking sub - band features and , training up a detector that detects for , these phonetic features for example , he presents  , a detector to detect sonorance . And so what what it basically is is ,  it 's there 's at the lowest level , there it 's it 's an OR ga  , it 's an AND gate . So , on each sub - band you have several independent tests , to test whether  , there 's the existence of sonorance in a sub - band . And at the at the higher level , for every if ,  The higher level there 's a soft OR gate .  , so if if this detector detects  , the presence of of sonorance in any of the sub - bands , then the detect  , the OR gate at the top says , " OK , well this frame has evidence of sonorance . " phd a: What are what are some of the low level detectors that they use ? grad e: And these are all Oh , OK .  , and the ,  professor c: So that , by the way , basically is a is one of the units in our in our our neural network . Yeah , so he uses , an EM algorithm to to  train up these  parameters for the logistic regression . professor c: Well , actually , yeah , grad e: The professor c: so I was using EM to get the targets . So so you have this this this AND gate what we were calling an AND gate , but it 's a product product rule thing at the output . And then he uses , i u and then feeding into that are I 'm sorry , there 's it 's an OR at the output , isn't it ? Yeah , grad e:   phd a: And so are each of these , low level detectors are they ,  are these something that you decide ahead of time , like " I 'm going to look for this particular feature or I 'm going to look at this frequency , " or What what what are they looking at ? grad e:  phd a: What are their inputs ? grad e:  Right , so the OK , so at each for each sub - band there are basically , several measures of SNR and and correlation .  , and for for every s every sub - band , e you you just pick ahead of time , " I 'm going to have like five i independent logistic tests . grad e: And you initialize these parameters , in some some way and use EM to come up with your training targets for a for the the low - level detectors . grad e: And then , once you get that done , you you you train the whole whole thing on maximum likelihood .  , and h he shows that using this this method to detect sonorance is it 's very robust compared to ,  to typical , full - band Gaussian mixtures  estimations of of sonorance . You get enough of these detectors together , then you have enough information to do , higher level discrimination , for example , discriminating between phones phd a:   grad e: and then you keep working your way up until you you build a full recognizer . grad e: So , that 's that 's the direction which I 'm I 'm thinking about going in my quals .  , one is the going towards , using narrow band information for , ph phonetic features of some sort rather than just , immediately going for the the typical sound units . professor c: Another thing I like about it is that you t this thing is going to be trained explicitly trained for a product of errors rule , which is what , Allen keeps pointing out that Fletcher observed in the twenties , phd a:   And then , the third thing I like about it is , and we 've played around with this in a different kind of way a little bit but it hasn't been our dominant way of of operating anything , this issue of where the targets come from . So in our case when we 've been training it multi - band things , the way we get the targets for the individual bands is , that we get the phonetic label for the sound there phd a:   professor c: and we say , " OK , we train every " What this is saying is , OK , that 's maybe what our ultimate goal is or not ultimate but penultimate goal is getting these these small sound units . But but , along the way how much should we , what should we be training these intermediate things for ?  , because , we don't know  , that this is a particularly good feature .  , there 's no way ,  someone in the audience yesterday was asking , " well couldn't you have people go through and mark the individual bands and say where the where it was sonorant or not ? " phd a:   professor c: But , you know , I think having a bunch of people listening to critical band wide , chunks of speech trying to determine whether I think it 'd be impossible . professor c: It 's all gonna sound like like sine waves to you , more or less . professor c:  Well not  , it 's g all g narrow band  , i I m I think it 's very hard for someone to to a person to make that determination . It could sh be that you should , not be paying that much attention to , certain bands for certain sounds , in order to get the best result . professor c: So , what we have been doing there , just sort of mixing it all together , is certainly much much cruder than that . Now we have I guess done experiments you 've probably done stuff where you have , done separate , Viterbis on the different grad e: Yeah . Did did that help at all ? grad e: it helps for one or t one iteration but  , anything after that it doesn't help . professor c: So so that may or may t it that aspect of what he 's doing may or may not be helpful because in a sense that 's the same sort of thing . You 're taking global information and determining what you how you should But this is this is , I th I think a little more direct . phd a: How did they measure the performance of their detector ? professor c: And Well , he 's look he 's just actually looking at , the confusions between sonorant and non - sonorant . professor c: So he hasn't applied it to recognition or if he did he didn't talk about it . It 's it 's just And one of the concerns in the audience , actually , was that that , the ,  he he did a comparison to , you know , our old foil , the the nasty old standard recognizer with mel mel filter bank at the front , and H M Ms , and and so forth . But the one of the good questions in the audience was , well , yeah , but that wasn't trained for that .  , this use of a very smooth , spectral envelope is something that , you know , has evolved as being generally a good thing for speech recognition but if you knew that what you were gonna do is detect sonorants or not So sonorants and non - sonorants is is is almost like voiced - unvoiced , except I guess that the voiced stops are are also called " obstruents " .  , so it 's it 's  , but with the exception of the stops I guess it 's pretty much the same as voiced - unvoiced , phd a:   So , if you knew you were doing that , if you were doing something say for a a ,  a a Vocoder , you wouldn't use the same kind of features . Nonetheless , it was one that was interesting because , this is what we are actually using for speech recognition , these smooth envelopes . And this says that perhaps even , you know , trying to use them in the best way that we can , that that that we ordinarily do , with , you know , Gaussian mixtures and H M Ms and so forth , you you don't , actually do that well on determining whether something is sonorant or not . phd a: Didn't they professor c: Which means you 're gonna make errors between similar sounds that are son sonorant or obstruent . phd a: Didn't they also do some kind of an oracle experiment where they said " if we could detect the sonorants perfectly and then show how it would improve speech recognition ? I thought I remember hearing about an experiment like that . professor c: That would that 's you 're right , that 's exactly the question to follow up this discussion , is suppose you did that , got that right . phd b: What could be the other low level detectors , for Other kind of features , or ? in addition to detecting sonorants or ? Th - that 's what you want to to to go for also grad e:  phd b: or ? grad e: What t Oh , build other other detectors on different phonetic features ? phd b: Other low level detectors ? Yeah . e  , w easiest thing would be to go go do some voicing stuff but that 's very similar to sonorance . grad e: phd a: When we when we talked with John Ohala the other day we made a list of some of the things that w grad e: Yeah . professor c: Yeah , so there 's a half dozen like that that are grad e: Yeah , nasality . professor c: Now this was coming at it from a different angle but maybe it 's a good way to start .  , these are things which , John felt that a a ,  a human annotator would be able to reliably mark . So the sort of things he felt would be difficult for a human annotator to reliably mark would be tongue position kinds of things . professor c: But stress doesn't , fit in this thing of coming up with features that will distinguish words from one another , grad e:   professor c: right ? It 's a it 's a good thing to mark and will probably help us ultimate with recognition phd a: Yeah , there 's a few cases where it can like permit and permit . professor c: Well , yeah , but i either case you 'd write PERMIT , right ? So you 'd get the word right . phd a: No , I 'm saying , i i e I thought you were saying that stress doesn't help you distinguish between words . As long as you get The sequence , professor c: We 're g if we 're doing if we 're talking about transcription as opposed to something else phd a: right ? Yeah . Yeah , so that 's yeah , that 's , you know , a neat neat thing and and ,  So . grad e: S so , Ohala 's going to help do these ,  transcriptions of the meeting data ? phd a: well I don't know .  , we just talked about some possible features that could be marked by humans and , grad e:  . phd a: because of having maybe some extra transcriber time we thought we could go through and mark some portion of the data for that . professor c: that 's not an immediate problem , that we don't immediately have a lot of extra transcriber time . professor c: But but , in the long term I guess Chuck is gonna continue the dialogue with John and and , and , we 'll we 'll end up doing some I think .  , so , y you want to talk maybe a c two or three minutes about what we 've been talking about today and other days ? grad f: Ri Yeah , OK , so , we 're interested in , methods for far mike speech recognition , mainly , methods that deal with the reverberation in the far mike signal . So , one approach would be , say MSG and PLP , like was used in Aurora one and , there are other approaches which actually attempt to remove the reverberation , instead of being robust to it like MSG . And so we 're interested in , comparing the performance of  , a robust approach like MSG with these , speech enhancement or de - reverber de - reverberation approaches . grad f: And , it looks like we 're gonna use the Meeting Recorder digits data for that . phd b: And the de - reverberation algorithm , do you have can you give some more details on this or ? Does it use one microphone ? grad f: o o phd b: Several microphones ? Does it ? grad f: OK , well , there was something that was done by , a guy named Carlos , I forget his last name , who worked with Hynek , who , professor c: Avendano . grad f: it was like RASTA in the sense that of it was , de - convolution by filtering  , except he used a longer time window , phd b:   And the reason for that is RASTA 's time window is too short to ,  include the whole , reverberation  , I don't know what you call it the reverberation response . The reverberation filter from my mouth to that mike is like it 's t got it 's too long in the in the time domain for the  for the RASTA filtering to take care of it . And , then there are a couple of other speech enhancement approaches which haven't been tried for speech recognition yet but have just been tried for enhancement , which , have the assumption that  , you can do LPC  analysis of th of the signal you get at the far microphone and the , all pole filter that you get out of that should be good . It 's just the , excitation signal that is going to be distorted by the reverberation and so you can try and reconstruct a better excitation signal and , feed that through the i  , all pole filter and get enhanced speech with reverberation reduced . professor c: There 's also this , echo cancellation stuff that we 've sort of been chasing , so ,  we have ,  and when we 're saying these digits now we do have a close microphone signal and then there 's the distant microphone signal . And you could as a kind of baseline say , " OK , given that we have both of these , we should be able to do , a cancellation . " So that , we we , essentially identify the system in between the linear time invariant system between the microphones and and and and re and invert it , or or cancel it out to to some some reasonable approximation phd b: that 's not a practical thing , if you have a distant mike , you don't have a close mike ordinarily , but we thought that might make also might make a good baseline .  , but And then there are s  , there are single microphone methods that I think people have done for ,  for this kind of de - reverberation . Do y do you know any references to any ? Cuz I I w I was w w I I lead him down a a bad path on that . phd b: I g I guess I guess when people are working with single microphones , they are more trying to do professor c: But . phd b: well , not not very Well , there is the Avendano work , professor c: Right . phd b: but also trying to mmm ,  trying to f t find the de - convolution filter but in the  not in the time domain but in the  the stream of features  I guess . Well , @ @ there there 's someone working on this on i in Mons professor c: Yeah , OK . phd b: So perhaps , yeah , we should try t to He 's working on this , on trying to professor c: Yeah . phd b: on re reverberation ,  professor c: The first paper on this is gonna have great references , I can tell already . professor c: It 's always good to have references , especially when reviewers read it or or one of the authors and , feel they 'll " You 're OK , you 've r You cited me . Well , he did echo cancellation and he did some fancier things like , training different network on different reverberation conditions and then trying to find the best one , but . professor c: The oth the other thing , that Dave was talking about earlier was , multiple mike things , where they 're all distant . So , there 's there 's all this work on arrays , but the other thing is , what can we do that 's cleverer that can take some advantage of only two mikes , particularly if there 's an obstruction between them , as we as we have over there . It 's part of why you have such good directionality with , with two ears phd b:   professor c: So that Yeah , the the head , in the way , is really that 's what it 's for . It 's basically , phd a: That 's what the head 's for ? To separate the ears ? professor c: Yeah , it 's to separate the ears . professor c: Yeah ? phd a: I think th that may be due to the fact that Adam ran out of digits , and didn't have time to regenerate any . professor c: Oh ! Oh ! I guess it 's Well there 's no real reason to write our names on here then , phd a: Yeah , if you want to put your credit card numbers and ,  professor c: is there ? grad e: Oh , no ? professor c: Or do did any do we need the names for the other stuff , phd a: yeah , I do need your names and and the time , and all that , professor c: or ? Oh , OK 
The professor expressed satisfaction with the improved model, attributing it to lower latencies. He believed that the effort put into developing a good Voice Activity Detection (VAD) was worthwhile. In order to determine the latency added by the VAD, the professor suggested running the process in parallel. Additionally, the professor informed the team that the maximum latency was still unknown. PhD C provided an explanation of the delays caused by various factors, including a 100ms delay from silence probabilities, a 40ms delay from delta at the input, and a 10ms delay from LDA filters. The professor criticized the insistence on using a flawed system and emphasized the need for straightforward solutions. He emphasized the importance of keeping the lag short for the task at hand.
What were the professor's thoughts on latencies, VAD latency, and the breakdown of latency?