grad e: Hello ? professor a: So everybody everybody 's on ? phd d: Today 's professor a: Yeah . professor a: eh e and  I guess Chuck you weren't there either , so the  phd b: I was there . professor a: Oh you were there ? phd b: With Hynek ? professor a: Yeah . professor a: What was the w what was the downsampling problem again ? phd c: So we had professor a: I forget . phd c: So the fact that there there is no  low - pass filtering before the downsampling . phd c: There is because there is LDA filtering but that 's perhaps not  the best w m professor a: Depends what it 's frequency characteristic is , yeah . So we discussed about this , about the  professor a: Was there any conclusion about that ? phd c:  " try it " . So again this is th this is the downsampling  of the  the feature vector stream phd c:  . professor a: and  Yeah I guess the the  LDA filters they were doing do have   let 's see , so the the the feature vectors are calculated every ten milliseconds so  the question is how far down they are at fifty fifty hertz . Does anybody know what the frequency characteristic is ? phd c: We don't have yet professor a: Oh OK . And the third point  was the  , yeah , the on - line normalization where , well , the recursion f recursion for the mean estimation is a filter with some kind of delay professor a: Yeah . For this , the conclusion of Hynek was , well , " we can try it but " professor a:  - huh . professor a: Try try what ? phd c: So try to   take into account the delay of the recursion for the mean estimation . And so while discussing about these these LDA filters , some i issues appeared , like well , the fact that if we look at the frequency response of these filters it 's  , well , we don't know really what 's the important part in the frequency response and there is the fact that in the very low frequency , these filters don't don't really remove a lot .  and that 's probably a reason why , yeah , on - line normalization helps because it it , professor a: Right . Yeah , but perhaps everything could should be could be in the filter ,  the the mean normalization and Yeah . phd c: And this was this LDA  tuning perhaps and Hynek proposed again to his  TRAPS , so . phd c: Yeah , professor a:  I g I guess the key thing for me is is figuring out how to better coordinate between the two sides phd c:  . professor a:  I was talking with Hynek about it later and the the sort of had the sense sort of that that neither group of people wanted to to bother the other group too much . And and I don't think anybody is , you know , closed in in their thinking or are unwilling to talk about things but I think that you were sort of waiting for them to tell you that they had something for you and and that and expected that they would do certain things and they were sor they didn't wanna bother you phd c:   professor a: and they were sort of waiting for you and and and  we ended up with this thing where they they were filling up all of the possible latency themselves , and they just had hadn't thought of that .  it 's true that maybe maybe no one really thought about that that this latency thing would be such a a strict issue phd c: Yeah . Well professor a: in in  the other phd c: Yeah I don't know what happened really , but professor a: Yeah . Because , well , we discussed about that about this problem and they told us " well , we will do all that 's possible to have enough space for a network " but then , yeah , perhaps they were too short with the time and professor a: Then they couldn't . Well maybe we should just   you 're you 're bus other than that you folks are busy doing all the all the things that you 're trying that we talked about before right ? And this machines are busy and you 're busy phd c: Yeah . professor a: Let 's let 's , I think that as as we said before that one of the things that we 're imagining is that  there there will be  in the system we end up with there 'll be something to explicitly   do something about noise phd c:   professor a: in addition to the  other things that we 're talking about and that 's probably the best thing to do . And there was that one email that said that it sounded like   things looked very promising up there in terms of  I think they were using Ericsson 's approach or something and in addition to They 're doing some noise removal thing , right ? phd c: Yeah , yeah . phd c: And phd d: Yeah , I modified it well , modifying I studied Barry 's sim code , more or less . and we have some the feature for Italian database and we will try with this feature with the filter to find the result . phd d: and maybe try another type of spectral subtraction , I don't professor a: When you say you don't have a result yet you mean it 's it 's just that it 's in process or that you it finished and it didn't get a good result ? phd d: No . No , no n we have n we have do the experiment only have the feature the feature but the experiment have phd c: Yeah . So  I suggest actually now we we we sorta move on and and hear what 's what 's what 's happening in in other areas like what 's what 's happening with your investigations about echos and so on . grad f: Oh  Well  I haven't started writing the test yet , I 'm meeting with Adam today professor a:   grad f:  and he 's going t show me the scripts he has for  running recognition on mee Meeting Recorder digits . grad f:  I also  haven't got the code yet , I haven't asked Hynek for for the for his code yet . Cuz I looked at  Avendano 's thesis and I don't really understand what he 's doing yet but it it it sounded like  the channel normalization part  of his thesis  was done in a a bit of I don't know what the word is , a a bit of a rough way  it sounded like he  he he it it wasn't really fleshed out and maybe he did something that was interesting for the test situation but I I 'm not sure if it 's what I 'd wanna use so I have to I have to read it more , I don't really understand what he 's doing yet . Yeah I haven't read it in a while so I 'm not gonna be too much help unless I read it again , phd d: It 's my phd c: Oh yeah ? phd d: I know this is mine here . The  so you , and then you 're also gonna be doing this echo cancelling between the the close mounted and the and the the the what we 're calling a cheating experiment  of sorts between the distant grad f:  I I 'm ho Right . grad f: I I think he 's at least planning to do it for the cl close - mike cross - talk and so maybe I can just take whatever setup he has and use it . Yeah actually  he should  I wonder who else is I think maybe it 's Dan Ellis is going to be doing  a different cancellation . One of the things that people working in the meeting task wanna get at is they would like to have cleaner close - miked recordings . So  this is especially true for the lapel but even for the close close - miked  cases  we 'd like to be able to have  other sounds from other people and so forth removed from So when someone isn't speaking you 'd like the part where they 're not speaking to actually be So what they 're talking about doing is using ec  echo cancellation - like techniques . It 's not really echo but  just   taking the input from other mikes and using   a  an adaptive filtering approach to remove the effect of that  other speech .  what was it , there was there was some some some point where eh  Eric or somebody was was speaking and he had lots of silence in his channel and I was saying something to somebody else  which was in the background and it was not it was recognizing my words , which were the background speech on the close close mike . phd b: Yeah that was actually my I was wearing the I was wearing the lapel and you were sitting next to me , professor a: Oh you it was you I was Yeah . phd b: and I only said one thing but you were talking and it was picking up all your words . So I think I think Dan Ellis or somebody who was working with him was going to  work on that . And  I don't know if we 've talked lately about the the plans you 're developing that we talked about this morning  I don't remember if we talked about that last week or not , but maybe just a quick reprise of of what we were saying this morning . So continuing to  extend phd b: What about the stuff that  Mirjam has been doing ? And and S Shawn , yeah . professor a: But that 's   all that 's is a a certainly relevant  study and , you know , what are the features that they 're finding . professor a:  what are the variables , what we 're calling this one , what are the variables that they 're found finding useful phd c:  . professor a:  for phd b: And their their targets are based on canonical mappings of phones to acoustic f features . And that 's certainly one thing to do and we 're gonna try and do something more f more fine than that but   so  So I guess you know what , I was trying to remember some of the things we were saying , do you ha still have that ? Yeah . professor a: There 's those that  yeah , some of some of the issues we were talking about was in j just getting a good handle on on  what " good features " are and phd b: What does what did  Larry Saul use for it was the sonorant  detector , right ? How did he H how did he do that ? Wh - what was his detector ?   professor a: And the other thing you were talking about is is is where we get the targets from . So  , there 's these issues of what are the what are the variables that you use and do you combine them using the soft " AND - OR " or you do something , you know , more complicated  and then the other thing was so where do you get the targets from ? The initial thing is just the obvious that we 're discussing is starting up with phone labels from somewhere and then  doing the transformation . But then the other thing is to do something better and eh w why don't you tell us again about this this database ? This is the phd b:  ! professor a: And then tell them to talk naturally ? Yeah , yeah . professor a: Maybe you could go to these parlors and and you could , you know you know have have , you know , reduced rates if you if you can do the measurements . You could what you could do is you could sell little rings and stuff with embedded you know , transmitters in them and things professor a: Yeah . phd b:  ! There 's a bunch of data that l around , that people have done studies like that w way way back right ?  I can't remember where  Wisconsin or someplace that used to have a big database of Yeah . I remember there was this guy at A T - andT , Randolph ? or r What was his name ? Do you remember that guy ?  , researcher at A T - andT a while back that was studying , trying to do speech recognition from these kinds of features . professor a: Do you mean eh but you  Mar phd c: Well he was the guy the guy that was using professor a: you mean when was was Mark Randolph there , or ? phd b: Mark Randolph . phd c: Is it the guy that was using the pattern of pressure on the tongue or ? phd b: I can't remember exactly what he was using , now . But I know I just remember it had to do with you know  positional parameters phd c: What Yeah . So the only the only  hesitation I had about it since ,  I haven't see the data is it sounds like it 's it 's continuous variables and a bunch of them . professor a: I don't know how complicated it is to go from there What you really want are these binary labels , and just a few of them . And maybe there 's a trivial mapping if you wanna do it and it 's e but it I I I worry a little bit that this is a research project in itself , whereas  if you did something instead that like  having some manual annotation by  you know , linguistics students , this would there 'd be a limited s set of things that you could do a as per our discussions with with John before phd b:   professor a: but the things that you could do , like nasality and voicing and a couple other things you probably could do reasonably well .  the other thing you could do is boot trying to to  get those binary variables and take the continuous variables from  the   the data itself there , but I I 'm not sure phd b: Could you cluster the just do some kind of clustering ? professor a: Guess you could , yeah . So anyway that 's that 's  that 's another whole direction that cou could be looked at .  in general it 's gonna be for new data that you look at , it 's gonna be hidden variable because we 're not gonna get everybody sitting in these meetings to wear the pellets and  . phd b: So you 're talking about using that data to get  instead of using canonical mappings of phones . phd b: So you 'd use that data to give you sort of what the the true mappings are for each phone ? grad e:   So wh yeah , where this fits into the rest in in my mind , I guess , is that  we 're looking at different ways that we can combine  different kinds of of rep front - end representations  in order to get robustness under difficult or even , you know , typical conditions . And part of it , this robustness , seems to come from  multi - stream or multi - band sorts of things and Saul seems to have a reasonable way of looking at it , at least for one one  articulatory feature . The question is is can we learn from that to change some of the other methods we have , since  , one of the things that 's nice about what he had I thought was that that it it  the decision about how strongly to train the different pieces is based on  a a reasonable criterion with hidden variables rather than  just assuming that you should train e e every detector  with equal strength towards  it being this phone or that phone . It 's a soft " AND " , I guess but in in principle you you wanna get a strong concurrence of all the different things that indicate something and then he " OR 's " across the different soft " OR 's " across the different  multi - band channels . And  the weight yeah , the target for the training of the " AND " " AND ' ed " things is something that 's kept  as a hidden variable , and is learned with EM . Whereas what we were doing is is  taking the phone target and then just back propagating from that phd b: So he doesn't have professor a: which means that it 's it 's  i It could be for instance that for a particular point in the data you don't want to   train a particular band train the detectors for a particular band . You you wanna ignore that band , cuz that 's a Ban - band is a noisy noisy measure . professor a: And we don't We 're we 're still gonna try to train it up . In our scheme we 're gonna try to train it up to do as well well as it can at predicting . phd b: So he doesn't have to have truth marks or Ho grad e: F right , and  he doesn't have to have hard labels . professor a: Well at the at the tail end , yeah , he has to know what 's where it 's sonorant . But he 's but what he 's - but what he 's not training up  what he doesn't depend on as truth is grad e: Right . professor a:  I guess one way of describing would be if if a sound is sonorant is it sonorant in this band ? Is it sonorant in that band ? grad e: Right . professor a: Is it sonorant in that band ? i It 's hard to even answer that what you really mean is that the whole sound is sonorant . professor a: then it comes down to , you know , to what extent should you make use of information from particular band towards making your decision . And   we 're making in a sense sort of this hard decision that you should you should use everything  with with  equal strength . professor a: And  because in the ideal case we would be going for posterior probabilities , if we had  enough data to really get posterior probabilities and if the if we also had enough data so that it was representative of the test data then we would in fact be doing the right thing to train everything as hard as we can . But  this is something that 's more built up along an idea of robustness from from the beginning and so you don't necessarily want to train everything up towards the phd b: So where did he get his  his tar his  high - level targets about what 's sonorant and what 's not ? grad e: From  canonical mappings  at first phd b: OK . grad e: and then it 's unclear  eh phd b: Using TIMIT ? or using grad e: using TIMIT phd b:  - huh .  we ha we have a kind of iterative training because we do this embedded Viterbi ,  so there is some something that 's suggested , based on the data but it 's it 's not I think it s doesn't seem like it 's quite the same , cuz of this cuz then whatever that alignment is , it 's that for all all bands . professor a: Well no , that 's not quite right , we did actually do them separate tried to do them separately so that would be a little more like what he did . But it 's still not quite the same because then it 's it 's  setting targets based on where you would say the sound begins in a particular band . Might be closer I guess if we did a soft soft target   embedded neural net training like we 've done a few times  f the forward  do the forward calculations to get the gammas and train on those .  what 's next ? phd b: I could say a little bit about w stuff I 've been playing with . You 're playing ? phd b: I  Huh ? professor a: You 're playing ? phd b: Yes , I 'm playing .  so I wanted to do this experiment to see   what happens if we try to  improve the performance of the back - end recognizer for the Aurora task and see how that affects things . And so I had this  I think I sent around last week a this plan I had for an experiment , this matrix where I would take the  the original  the original system . So there 's the original system trained on the mel cepstral features and then com and then  optimize the b HTK system and run that again . So look at the difference there and then  do the same thing for the ICSI - OGI front - end . professor a: What which test set was this ? phd b: This is that I looked at ? professor a:   phd b: So as far as I 've gotten is I 've  been able to go through from beginning to end the  full HTK system for the Italian data and got the same results that  that  Stephane had . So  I started looking to and now I 'm I 'm sort of lookin at the point where I wanna know what should I change in the HTK back - end in order to try to  to improve it . One of the first things I thought of was the fact that they use the same number of states for all of the models professor a:   phd b: and so I went on - line and I  found a pronunciation dictionary for Italian digits professor a:   phd b: and just looked at , you know , the number of phones in each one of the digits .  you know , sort of the canonical way of setting up a an  system is that you use  three states per phone and  so then the the total number of states for a word would just be , you know , the number of phones times three . And so when I did that for the Italian digits , I got a number of states , ranging on the low end from nine to the high end , eighteen . Now you have to really add two to that because in HTK there 's an initial null and a final null so when they use  models that have eighteen states , there 're really sixteen states . And so  their guess of eighteen states seems to be pretty well matched to the two longest words of the Italian digits , the four and five which  , according to my , you know , sort of off the cuff calculation , should have eighteen states each . So my guess  And then if you I I printed out a confusion matrix   for the well - matched case , and it turns out that the longest words are actually the ones that do the best . So my guess about what 's happening is that you know , if you assume a fixed the same amount of training data for each of these digits and a fixed length model for all of them but the actual words for some of them are half as long you really  have , you know , half as much training data for those models . Because if you have a long word and you 're training it to eighteen states ,  you 've got you know , you 've got the same number of Gaussians , you 've gotta train in each case , professor a:   phd b: but for the shorter words , you know , the total number of frames is actually half as many . phd b: So it could be that , you know , for the short words there 's because you have so many states , you just don't have enough data to train all those Gaussians . So  I 'm going to try to  create more word - specific   prototype H M Ms to start training from . professor a: Yeah , it 's not at all uncommon you do worse on long word on short words than long words anyway just because you 're accumulating more evidence for the for the longer word , phd b:   phd b: Yeah so I 'll I 'll , the next experiment I 'm gonna try is to just  you know create  models that seem to be more w matched to my guess about how long they should be . phd b: And as part of that  I wanted to see sort of how the  how these models were coming out , you know , what w when we train up  th you know , the model for " one " , which wants to have nine states , you know , what is the  what do the transition probabilities look like in the self - loops , look like in in those models ? And so I talked to Andreas and he explained to me how you can calculate the expected duration of an  just by looking at the transition matrix professor a:   phd b: and so I wrote a little Matlab script that calculates that and so I 'm gonna sort of print those out for each of the words to see what 's happening , you know , how these models are training up , professor a:   I d I did quickly , I did the silence model and and  that 's coming out with about one point two seconds as its average duration and the silence model 's the one that 's used at the beginning and the end of each of the string of digits . And so the S P model , which is what they put in between digits , I I haven't calculated that for that one yet , but  . So they basically their their model for a whole digit string is silence digit , SP , digit , SP blah - blah - blah and then silence at the end . professor a: Are the SP 's optional ?  skip them ? phd b: I have to look at that , but I 'm not sure that they are . Now the one thing about the S P model is really it only has a single s emitting state to it . phd b: So if it 's not optional , you know , it 's it 's not gonna hurt a whole lot professor a: I see . phd b: and it 's tied to the center state of the silence model so it 's not its own  It doesn't require its own training data , professor a:   phd b: So it , it 's pretty good the way that they have it set up , but  i So I wanna play with that a little bit more . I 'm curious about looking at , you know how these models have trained and looking at the expected durations of the models and I wanna compare that in the the well - matched case f to the unmatched case , and see if you can get an idea of just from looking at the durations of these models , you know , what what 's happening . professor a: Yeah , I think that  , as much as you can , it 's good to d sort of not do anything really tricky . professor a: Not do anything that 's really finely tuned , but just sort of eh you know you t you i z phd b: Yeah . professor a: The premise is kind of you have a a good person look at this for a few weeks and what do you come up with ? phd b:   professor a: And  phd b: And Hynek , when I wa told him about this , he had an interesting point , and that was th  the the final models that they end up training up have I think probably something on the order of six Gaussians per state . And Hynek was saying that well , probably in a real application , you wouldn't have enough compute to handle models that are very big or complicated . But you know , it depends on what the actual application is and it 's really hard to know what your limits are in terms of how many Gaussians you can have . professor a: I I I what I thought you were gonna say i but which I was thinking was  where did six come from ? Probably came from the same place eighteen came from . professor a:  that 's another parameter , right ? that that maybe , you know ,  you really want three or nine or phd b: Yeah , yeah . Well one thing  , if I if if I start  reducing the number of states for some of these shorter models that 's gonna reduce the total number of Gaussians . professor a: how much better can you make it ? And  since they 're only simple things there 's nothing that you 're gonna do that is going to blow up the amount of computation phd b:   professor a: if you found that nine was better than six that would be O K , I think , actually . I really wasn't even gonna play with that part of the system yet , professor a:  -  , OK . phd b: I was just gonna change the the t professor a: Yeah , just work with the models , yeah . So  what 's  I guess your plan for You you you guys ' plan for the next next week is just continue on these these same things we 've been talking about for Aurora and phd c: Yeah , I guess we can try to have some kind of new baseline for next week perhaps . And then do other things , play with the spectral subtraction , and retry the MSG and things like that . I think that after all of this   confusion settles down in another some point a little later next year there will be some sort of standard and it 'll get out there and hopefully it 'll have some effect from something that that has  been done by our group of people but  e even if it doesn't there 's there 's go there 'll be standards after that . phd b: Does anybody know how to  run Matlab sort of in batch mode like you c send it s a bunch of commands to run and it gives you the output . Is it possible to do that ? grad e: I I think  Mike tried it phd b: Yeah ? grad e: and he says it 's impossible so he went to Octave . phd c: What is Octave so ? It 's a free software ? grad e: What 's that ?  , Octave ? phd c: Yeah . phd c: And it does the same syntax and everything eh like Matlab , or ? grad e:  i it 's a little behind , it 's the same syntax but it 's a little behind in that Matlab went to these like  you can have cells and you can you can  implement object - oriented type things with Matlab .  Octave doesn't do that yet , so I think you , Octave is kinda like Matlab  four point something or . phd b: If it 'll do like a lot of the basic matrix and vector stuff grad e: The basic stuff , right <doc-sep>grad d:  grad e: What does your thing say on the back ? grad d: Testing . grad f: Yeah , well , I g guess it 's coming up then , or grad d: Cuz it 's That starts counting from zero and these start counting from one . grad d: Your mike number is what we 're t grad e: Look at the back . grad d: Ho ! grad b: So grad d: I 've bested you again , Nancy . Damn ! Foiled again ! grad d: So is Keith showing up ? He 's talking with George right now .  , is he gonna get a rip  rip himself away from from that ? grad b: He 'll probably come later . grad d: Oh , then it 's just gonna be the five of us ? professor c: Yeah . grad e: Well , he he was very affirmative in his way of saying he will be here at four . So , Eva just reported she 's really happy about the CBT 's being in the same order in the XML as in the  be Java declaration format grad f: Yeah . The e grad e: so you don't have to do too much in the style sheet transversion . grad e: The  , Java the embedded Bayes wants to take input  , a Bayes - net in in some Java notation and Eva is using the Xalan style sheet processor to convert the XML that 's output by the Java Bayes for the into the , E Bayes input . grad f: Actually , maybe I could try , like , emailing the guy and see if he has any something already . grad f: That 'd be weird , that he has both the Java Bayes and the embedded Bayes in grad d: But that 's some sort of conversion program ? grad f: Yeah . grad e: And , well pretty mu pretty much on t on the top of my list , I would have asked Keith how the " where is X ? " hand parse is standing . grad d: So the trees for the XML trees for the for the gene for the synthesizer are written . Just gonna be you know professor c: Oh ! You were gonna send me a note about hiring grad e: Yes . grad e: OK , so natural language generation produces not a just a surface string that is fed into a text - to - speech but , a surface string with a syntax tree that 's fed into a concept - to - speech . grad e: Now and this concept - to - speech module has certain rules on how if you get the following syntactic structure , how to map this onto prosodic rules . grad e: And Fey has foolheartedly agreed to rewrite  , the German concept  syntax - to - prosody rules grad b: I didn't know she spoke German . And  therefore the ,  if it 's OK that we give her a couple of more hours per week , then she 'll do that . grad d: What language is that written i Is that that Scheme thing that you showed me ? grad e: Yeah . grad d: She knows how to program in Scheme ? I hope ? grad e: No , I My guess is I I asked for a commented version of that file ? If we get that , then it 's doable , even without getting into it , even though the Scheme li  , stuff is really well documented in the Festival . grad d: Well , I guess if you 're not used to functional programming , Scheme can be completely incomprehensible . But anyway , send me the note and then I 'll - I 'll check with , Morgan on the money . Oh , so this was You know , on the generation thing ,  if sh y she 's really going to do that , then we should be able to get prosody as well . grad d: Are we gonna Can we change the voice of the of the thing , because right now the voice sounds like a murderer . grad b: Wh - Which one ? grad d: The the little Smarticus Smarticus sounds like a murderer . grad e: It is  , we have the choice between the , usual Festival voices , which I already told the SmartKom people we aren't gonna use because they 're really bad . grad b: Festival ? professor c: It 's the name of some program , grad b: Oh , oh . grad e: OGI has , crafted a couple of diphone type voices that are really nice and we 're going to use that . grad d: Does OGI stand for ? Original German Institute ? professor c: Orego grad b: So . grad b: Oregon Graduate Insti professor c: Oregon @ @ Graduate Institute grad d: Oh . professor c: It turns out there 's the long - standing links with these guys in the speech group . professor c: In fact , there 's this guy who 's basically got a joint appointment , Hynek Hermansky . And it 's probably also absolutely uninteresting for all of you to ,  learn that as of twenty minutes ago , David and I , per accident ,  managed to get the whole SmartKom system running on the  , ICSI Linux machines with the ICSI NT machines thereby increasing the number of running SmartKom systems in this house from one on my laptop to three . grad e: I suggested to try something that was really kind of even though against better knowledge shouldn't have worked , but it worked . grad b: Will it work again , grad e: Maybe maybe maybe a bit for the AI i intuition thing . It - it 's just like why why the generation ma the presentation manager is now working ? grad a:  ! This is something you ha you get used to as a programmer , right ? grad e: Which grad a: You know , and it 's cool , it works out that way . So , the the people at Saarbruecken and I decided not to touch it ever again . grad a: What what thing is this ? grad e: Where is X ? grad a: OK . grad e: Oh , but by  , we can ask , did you get to read all four hundred words ? professor c: I did . That professor c: You know , i Yeah , it grad d: Each paragraph is good , though . grad b: Yeah , I didn't know about it until Robert told me , like ,  professor c: Yeah , I I ra I ran across it in I don't even know where , you know some just some weird place . And , yeah , I I 'm surprised I didn't know about it grad b: Y yeah . I was like , why didn't Dan tell me ? professor c: since we know all the invited speakers , an grad a: Right . Before we get started on this st so I also had a nice email correspondence with Daphne Kohler , who said yes indeed she would love to work with us on the , you know , using these structured belief - nets and stuff but starting in August , that she 's also got a new student working on this and that we should get in touch with them again in August and then we 'll figure out a way for you  you to get seriously connected with ,  their group . So ,  And it looks to me like we 're now at a good point to do something start working on something really hard . grad a: Oh ! professor c: w Which is mental spaces and  and - or grad a:  ! grad b: It 's hard . But the other part of it is the way they connect to these , probabilistic relational models . So there 's all the problems that the linguists know about , about mental spaces , and the cognitive linguists know about , but then there 's this problem of the belief - net people have only done a moderately good job of dealing with temporal belief - nets . professor c: So there 's a term " dynamic belief - net " , doesn't mean that . But one of the things I w would like to do over the next , month , it may take more , is to st understand to what extent we can not only figure out the constructions for them for multiple worlds and  sort of what the formalism will look like and where the slots and fillers will be , but also what that would translate into in terms of belief - net and the inferences . So the story is that if you have these probabilistic relational models , they 're set up , in principle , so that you can make new instances and instances connect to each other , and all that sort of stuff , so it should be feasible to set them up in such a way that if you 've got the past tense and the present tense and each of those is a separate  , belief structure that they do their inferences with just the couplings that are appropriate . But that 's g that 's , as far as I can tell , it 's it 's putting together two real hard problems . One is the linguistic part of what are the couplings and and when you have a certain , construction , that implies certain couplings and other couplings , you know , between let 's say between the past and the present , or any other one of these things and then we have this inference problem of exactly technically how does the belief - net work if it 's got  , let 's say one in in , you know , different tenses or my beliefs and your beliefs , or any of these other ones of of multiple models . So  you know , in the long run we need to solve both of those and my suggestion is that we start digging into them both , in a way we that , you know , th hopefully turns out to be consistent , so that the  . But anyway , so that 's ,  grad a: Oh yeah , like  , I solved the the problem of  we were talking about how do you various issues of how come a plural noun gets to quote " count as a noun phrase " , you know , occur as an argument of a higher construction , but a bare singular stem doesn't get to act that way . grad a: and it would take a really long time to explain it now , but I 'm about to write it up this evening . I solved that at the same time as " how do we keep adjectives from floating to the left of determiners and how do we keep all of that from floating outside the noun phrase " to get something like " I the kicked dog " . No , I know , I th I I think that is gonna be sort of the key to this wh to th the big project of the summer of of getting the constructions right is that people do manage to do this so there probably are some , relatively clean rules , they 're just not context - free trees . professor c: And if we if the formalism is is good , then we should be able to have , you know , sort of moderate scale thing . And that by the way is is , Keith , what I encouraged George to be talking with you about . professor c: The p And Oh , another thing ,  there was this ,  thing that Nancy agreed to in a in a weak moment this morning that grad a:  ! grad b: I was really strong . professor c: Anyway , that we were that we 're gonna try to get a  , first cut at the revised formalism by the end of next week . professor c: just trying to write up essentially what what you guys have worked out so that everybody has something to look at . We 've talked about it , but only the innermost inner group currently , grad a:   grad a: There 's The group as a whole knows but no individual member kno professor c: Well that that yeah th there 's one of the advantages of a document , right ? , grad a: Yeah . Anyway , so , with a little luck  l let 's , let 's have that as a goal anyway . grad a: So , what was the date there ? professor c: And grad a: Monday or ? It 's a Friday . grad b: But , but but the two of us will probably talk to you at well before th grad a: I thought you said beginning of n Yeah . Anyway , w let 's talk separately about how t grad a: Yeah , I have a busy weekend but after that Yeah , gung - ho . grad a: Great , professor c: Now if it turns out that that effort leads us into some big hole that 's fine . There 's a really hard problem we haven't solved yet that , that 's just fine . grad a: But at at least sort of try and work out what the state of the art is right now . professor c: Right , t t if to the extent that we have it , let 's write it grad a: OK . professor c: and to the extent we don't , let 's find out what we need to do . professor c: So ,  grad e: Can we ? Is it worth thinking of an example out of our tourism thing domain , that involves a a a decent mental space shift or setting up professor c: I think it is , but  but I interrupted before Keith got to tell us what happened with " where is the Powder - Tower ? " or whatever grad b: Right .  , what was supposed to happen ? I 've sort of been actually caught up in some other ones , so , you know , I don't have a write - up of or I haven't elaborated on the ideas that we were already talking about which were grad e: yeah . I think I think we already came to the conclusion that we have two alternative paths that we two alternative ways of representing it . grad e:  grad a: The question of whether the polysemy is sort of like in the construction or pragmatic . grad a: so d ' you Is it clear what we 're talking about here ? grad b: I agree . grad a: The question is whether the construction is semantic or like ambiguous between asking for location and asking for path . grad e: It 's grad a:  or or whether the construction semantically , is clearly only asking for location grad e: Should we have a a a grad b:  grad a: but pragmatically that 's construed as meaning " tell me how to get there " . grad e: So assume these are two , nodes we can observe in the Bayes - net . grad e: So these are either true or false and it 's also just true or false . If we encounter a phrase such as " where is X ? " , should that set this to true and this to true , and the Bayes - net figures out which under the c situation in general is more likely ?  , or should it just activate this , have this be false , and the Bayes - net figures out whether this actually now means ? professor c:  w that 's a s grad b: Slightly different . professor c: So I a I I th I agree with you that , it 's a disaster to try to make separate constructions for every  , pragmatic reading , grad a:   professor c: there there 's some that grad b: Or have every construction list all the possible pragmatic implications of the same one . But , you know , c  almost certainly " can you pass the salt " is a construction worth noting that there is this th this this this  grad a: Yeah . Is it is it like that or is it just sort of obvious from world knowledge that no one you wouldn't want to know the location without wanting to know how to get there or whatever . grad e: One Or in some cases , it 's it 's quite definitely professor c: Yeah . Well the question is basically , is this conventional or conversational implicature ? professor c: Exactly . professor c: And I guess , see , the more important thing at this stage is that we should be able to know how we would handle it in ei f in the short run it 's more important to know how we would treat technically what we would do if we decided A and what we would do if we decided B , than it is t to decide A or B r right now . professor c: So I guess In the short run , let 's let 's be real clear on h what the two alternatives would be . grad e: And then the we had another idea floating around  , which we wanted to , get your input on , and that concerns the But the nice thing is w we would have a person that would like to work on it , and that 's Ir - Irina Gurevich from EML who is going to be visiting us , the week before , August and a little bit into August .  , and one of the reas one of the those ideas was , so , back to the old Johno observation that if y if you have a dialogue history and it said the word " admission fee " was  , mentioned  , it 's more likely that the person actually wants to enter than just take a picture of it from the outside . Now what could imagine to , you know , have a list for each construction of things that one should look up in the discourse history , yeah ? That 's the really stupid way . Then there is the really clever way that was suggested by Keith and then there is the , middle way that I 'm suggesting and that is you you get X , which is whatever , the castle . The ontology will tell us that castles have opening hours , that they have admission fees , they have whatever . And then , this is We go via a thesaurus and look up certain linguistic surface structures that are related to these concepts and feed those through the dialogue history and check dynamically for each e entity . We look it up check whether any of these were mentioned and then activate the corresponding nodes on the discourse side . But Keith suggested that a a much cleaner way would be is , you know , to keep track of the discourse in such a way that you if you know that something like that ha has been mentioned before , this just a continues to add up , you know , in th in a grad a: So if someone mentions admission f fees , that activates an Enter schema which sticks around for a little while in your rep in the representation of what 's being talked about . And then when someone asks " where is X ? " you 've already got the the Enter schema activated grad b: Kind of a priming professor c:   professor c:  grad d: Well , is it doesn't it seem like if you just managed the dialogue history with a a thread , that you know , kept track of ho of the activity of  , cuz it would the the thread would know what nodes like , needed to be activated , so it could just keep track of how long it 's been since something 's been mentioned , and automatically load it in . But here 's here 's a way in th in the bl Bayes - net you could you could think about it this way , that if  at the time " admissions fee " was mentioned you could increase the probability that someone wanted to enter . grad d: We - yeah th th that 's what I wa I wasn't I was I wasn't thinking in terms of Enter schemas . I was just professor c: Fair enough , OK , but , but , in terms of the c c the current implementation right ? so that  grad b: It would already be higher in the context . professor c: th that th the the the conditional probability that someone So at the time you mentioned it This is this is essentially the Bayes - net equivalent of the spreading activation . professor c: It 's In some ways it 's not as good but it 's the implementation we got . Now Now my guess is that it 's not a question of time but it is a question of whether another intervening object has been mentioned . professor c: we could look at dialo this is Of course the other thing we ha we do is , is we have this data coming grad a: Yeah . professor c: but but skipping that so so but my guess is what what 'll probably will happen , Here 's a here 's a proposed design . is that there 're certain constructions which , for our purposes do change the probabilities of EVA decisions and various other kinds and th that the , standard way that that the these contexts work is sort of stack - like or whatever , but that 's sort of the most recent thing . And so it could be that when another  , en tourist entity gets mentioned , you grad b: Renew professor c: re re essentially re - initiali you know , re - i essentially re - initialize the state . professor c: And of course i if we had a fancier one with multiple worlds you could have  , you could keep track of what someone was  saying about this and that . professor c: You know , " I wanna go in the morning grad a: " Here 's my plan for today . " professor c: Yeah , or Yeah , in the morning morning I I 'm planning t to go shopping , grad a: hypothetically . professor c: tal so I 'm talking about shopping and then you say , you know , well ,  " What 's it cost ? " or something . professor c: But I do th think that the It 'll turn out that it 's gonna be depend pretty much on whether there 's been an override . grad e: Yeah , if if you ask " how much does a train ride and and cinema around the vineyards cost ? " and then somebody tells you it 's sixty dollars and then you say " OK How much is ,  I would like to visit the " whatever , something completely different , " then I go to , you know , Point Reyes " , professor c: Yeah . grad e: it it 's not more likely that you want to enter anything , but it 's , as a matter of fact , a complete rejection of entering by doing that . grad b: So when you admit have admission fee and it changes something , it 's only for that particular It 's relational , right ? It 's only for that particular object . Well , and and and the simple idea is that it 's on it 's only for m for the current  , tourist e entity of instre interest . But that 's  this this function , so , has the current object been mentioned in in with a question about concerning its professor c: No , no . Is When th When the this is mentioned , the  probability of of , let 's say , entering changes grad b: Of that object . grad d: You could just hav  , just basically , ob it It observes an er , it sets the a node for " entered " or " true " or something , professor c: Yeah . Now ,  But I think Ro - Robert 's right , that to determine that , OK ? you may well want to go through a th thesaurus grad d: " discourse enter " . professor c: and and So , if the issue is , if so now th this construction has been matched and you say " OK . Does this actually have any implications for our decisions ? " Then there 's another piece of code that presumably does that computation . professor c: But but what 's Robert 's saying is is , and I think he 's right , is you don't want to try to build into the construction itself all the synonyms and all you know , all the wo  maybe . Well , it 's just another , sort of , construction side is how to get at the possible inferences we can draw from the discourse history or changing of the probabilities , and - or grad b: Guess it 's like I g The other thing is , whether you have a m m user model that has , you know , whatever , a current plan , whatever , plans that had been discussed , and I don't know ,  grad d: What  , what 's the argument for putting it in the construction ? Is it just that the s synonym selection is better , or ? professor c: Oh , wel Well , the ar the The argument is that you 're gonna have the If you 've recognized the word , you 've recognized the word , which means you have a lexical construction for it , so you could just as well tag the lexical construction with the fact that it 's a  , you know , thirty percent increase in probability of entering . You So you could you could you could invert invert the whole thing , so you s you tag that information on to the lexicon grad d: Mmm . at at Yeah , and this is grad e: Even though  the lexical construction itself out out of context , won't do it . grad e: " But I but I 'm not interested in the opening times " is sort of a more a V type . But , we 'll   , we have time to This is a s just a sidetrack , but  I think it 's also something that people have not done before , is  , sort of abuse an ontology for these kinds of , inferences , on whether anything relevant to the current something has been  , has crept up in the dialogue history already , or not . And ,  I have the ,  If we wanted to have that function in the dialogue hi dialogue module of SmartKom , I have the written consent of Jan to put it in there . professor c: So the point is , it 's very likely that Robert 's thesis is going to be along these lines , grad b: Oh , s professor c: and the local rules are if it 's your thesis , you get to decide how it 's done . So if , you know if this is seriously , if this becomes part of your thesis , you can say , hey we 're gonna do it this way , that 's the way it 's done . So h he 's got a th he 's got a meet meeting in Germany with his thesis advisor .  , this is this is , speaking of hard problems , this is a very good time  , to start trying to make explicit where construal comes in and you know , where c where the construction per - se ends and where construal comes in , grad a:   But that 's part of what the f grad b: We have many jobs for you , Ro - Robert . professor c: is is because th it is gonna have implicit in it grad e: Was I ? In the room ? grad b: No , you weren't there on purpose . professor c: Yeah , that That 's the point , is is th grad a: Yeah . grad d: Yeah , but it he the decisions I made wer had to do with my thesis . So consequently don't I get to decide then that it 's Robert 's job ? professor c: No . grad b: Well , I 'll just pick a piece of the problem and then just push the hard stuff into the center and say it 's Robert 's . grad e: I 've always been completely in favor of consensus decisions , grad b: I can professor c: Right . professor c: not grad e: It it might even be interesting then to say that I should be forced to  , sort of pull some of the ideas that have been floating in my head out of the ,  out of the top hat professor c: Yes . grad e: Yes , and , it 's Ha - None of that is basically still around , grad b: I didn't get professor c: And a another draft OK . grad e: but it 's professor c: D i grad a: That 's normal . professor c: I i grad b: Oh , I guess it 's good I didn't read it . professor c: and grad e: Yeah , and I would like to d discuss it and , you know , get you guys 's input professor c: Right . professor c:  So that , so th thi this  , so this is the point , is we we 're going to have to cycle through this , grad a: Yeah . professor c: but th the draft of the p proposal on the constructions is is going to tell us a lot about what we think needs to be done by construal . grad b:  grad e: Meeting regular meeting time for the summer , we really haven't found one . It 's - it 's a coincidence that he can't do couldn't do it today here . professor c: And the NTL meeting moved to Wednesday , grad e: So professor c: cuz of of ,  grad e: Yeah , it was just an exception . professor c: Yeah , you weren't here , but but but s  , And so , if that 's OK with you , grad a: It 's i Is it staying basically at the Wednesday noon ? professor c: you would grad a: OK . How do we feel about doing it Wednesdays ? Because it seems to me that this is sort of a time where when we have things to discuss with other people , there they seem to be s tons of people around . professor c: The only disadvantage is that it may interfere with other grad e: Or subgroup meetings professor c: s you know , other other No , you  , people in this group connecting with with grad b: Those people who happen to be around . I I  you know I have no fixed grad a: To tell you the truth , I 'd rath I 'd , I 'd would like to avoid more than one ICSI meeting per day , if possible .  that grad e: The I 'd like to have them all in one day , grad a: Yeah , I can understand that . professor c: Well p grad e: so package them up and then professor c: people people differ in their tastes in this matter . I 'm always here anyway , grad e: It 's OK , that grad b: so It doesn't matter . grad e: Well , if one sort of thing is , this room is taken at after three - thirty pr pretty much every day by the data collection . grad e: So we just knew i grad b: So did you just say that Ami can't make one o ' grad e: No , he can . About the c the th grad b: Could we do Thursday at one - thirty ? Would that that be horrible ? grad e: No . grad b: Oh really ? grad e: Because , this room is again taken at two - thirty by Morgan . grad e: And the s meeting recorder meeting meeting meeting recording on meeting meetings grad b: OK , OK , OK . professor c: I I could grad b:  we usually meet Tuesday or l like , linguists  , at two . Do you want to meet again here bef grad d: And the s Is the Speech - Gen meeting still at on Tuesdays ? grad e:  w Well , actually we w we we did scrap our Monday time just because Bhaskara couldn't come Monday . grad a: That doesn't apply to a grad d: Although I thought you wanted to go camping on Monday er , take off Mondays a lot so you could go camping .  ! grad b: Well , the linguists ' meeting i happens to be at two , but I think that 's  . grad a: And , you know , of course Nancy and I are just sort of always talking anyway and sometimes we do it in that room . grad e: Monday ? professor c: OK , whate  What I think Robert 's saying is that grad a: Earlier in the week professor c: earlier we At least for next week , there 's a lot of stuff we want to get done , grad a:   grad b: At o o o o one , two , three ? grad e: One , two , three ? Three 's too late . professor c: Oh , I i Yeah , I actually Two is the earliest I can meet on Monday . professor c: Here I 'm blissfully agreeing to things and realizing that I actually do have some stuff scheduled on Monday . grad b: You guys will still remind me , right ? grad d: No way ! grad b: Y you 'll come and take all the the headph the good headphones first and then remind me . grad e: And grad b: Why do I have this unless I 'm gonna write ? grad e: do I get to see th  , your formalism before that ? grad b: Fine . grad e: I wo I would like I would sort of get a get a notion of what what you guys have in store for me . professor c: Well m @ @ you know , w maybe Mond - Maybe we can put This is part of what we can do Monday , if we want . grad b: Yeah , so there was like , you know , m m in my head the goal to have like an intermediate version , like , everything I know . grad b: And then , w I would talk to you and figure out everything you know , that you know , see if they 're consistent . Why don't w Maybe you and I should meet sort of more or less first thing Monday morning and then we can work on this . I might I might  , grad e: You - y grad b: s You said you 're busy over th until the weekend , right ? grad a: Yeah , sort of through the weekend because Kate has a photography show . So , maybe I 'll send you some grad a: if you have time after this I 'll show you the noun phrase thing . OK , and we 'll You wanna m grad e: So the idea is on Monday at two we 'll we 'll see an intermediate version of the formalism for the constructions , grad a: Yeah . grad b: So that 's OK for you grad e: and do an on - line merging with my construal ideas . grad e: So it won't be , like , a for semi - formal presentation of my proposal . grad b: Cuz then you 'll find out more of what we 're making you do . Can you also write it up ? grad b: It 's like , " this is what we 're doing . " grad e: I 'll I 'll send you I 'll I 'll send you a style file , right ? grad b: OK . grad e: You just grad b: I already sent you my fi my bib file . grad a: Someday we also have to we should probably talk about the other side of the " where is X " construction , which is the issue of , how do you simulate questions ? What does the simspec look like for a question ? grad e: Yeah . professor c: Yeah , now , we we w grad a: We had to we had an idea for this which seemed like it would probably work . I professor c: OK ? So let 's think of a name for for whatever the this intermediate structure is . Oh , we talked about semspec , for " semantic spec specification " grad a: Mmm . grad a: It 's more general professor c: You know , so it 's a m minimal change . professor c: Right , a little substi substi You know , that 's what text substitution  macros are for . professor c: Anyway , so let 's let 's for the moment call it that until we think of something better . professor c: And , yeah , we absolutely need to find Part of what was missing were markings of all sorts that weren't in there , incl including the questions grad a:   professor c: We didn't we never did figure out how we were gonna do emphasis in in  , the semspec . grad b: Yeah , we 've talked a little bit about that , too , which  , it 's hard for me to figure out with sort of our general linguistic issues , how they map onto this particular one , grad a: Yeah . professor c: But that 's part of the formalism is got to be  , how things like that get marked . grad b: W do you have data , like the the You have preliminary data ? Cuz I know , you know , we 've been using this one easy sentence and I 'm sure you guys have  , maybe you are the one who 've been looking at the rest of it grad a: I grad b: it 'd it 'd be useful for me , if we want to have it a little bit more data oriented . grad a: To tell you the truth , what I 've been looking at has not been the data so far , grad b: Yeah . grad a: I just sort of said " alright let 's see if I can get noun phrases and , major verb co  , constructions out of the way first . grad a: So , I have not really approached a lot of the data , but  obviously like these the the question one , since we have this idea about the indefinite pronoun thing and all that , you know , I ca can try and ,  run with that , you know , try and do some of the sentence constructions now . grad a: Oh yeah , the basic idea is that  ,  you know  , let 's see if I can formulate this . grad e: So you perform the mental sum and then , you know , " who fixed the car with a wrench ? " You basically are told , to to do this In the in analogously to the way you would do " someone fixed the car with a wrench " . And then you hand it back to your hippocampus and find out what that , you know , grad a: Means . grad a: The WH question has this as sort of extra thing which says " and when you 're done , tell me who fills that slot " or w you know . And , you know , this is sort of a nice way to do it , the idea of sort of saying that you treat from the simulation point of view or whatever you treat , WH constructions similarly to  , indefinite pronouns like " someone fixed the car " because lots of languages , have WH questions with an indefinite pronoun in situ or whatever , grad b: Use actually the same one . So it makes sense professor c: Alright , which is grad a:  professor c: Skolemization . professor c: In in logic , it 's it 's @ @ it 's actual Huh ? grad b: Right . professor c: That - that 's not that 's not saying it 's bad , grad a: Right . Anyway , but just that kind of thing and we 'll figure out exactly how to write that up and so on , but professor c: Good . We sort of just dropped that cuz it was too weird and we didn't even know , like , what we were talking about exactly , what the object of study was . Well , if if  , i part of of what the exercise is , t by the end of next week , is to say what are the things that we just don't have answers for yet . grad e: Well , if you if you do wanna discuss focus background and then get me into that because  , I wo I w scientifically worked on that for for almost two years . grad b: Yeah , you should definitely ,  be on on that maybe maybe by after Monday we 'll y you can see what things we are and aren't grad a: Yeah . grad b: I would say that tha that those discussions have been primarily , Keith and Keith and me , but  like in th the meeting  , he sort of I thin like the last meeting we had , I think we were all very much part of it grad a: Yeah . grad b: but  grad a: Sometimes Hans has been sort of coming in there as sort of like a devil 's advocate type role or something , grad b: but different perspec Yeah . grad a: like " This make you know , I 'm going to pretend I 'm a linguist who has nothing to do with this . " And he 'll just go off on parts of it which definitely need fixing grad b: Right . grad a: but aren't where we 're at right now , so it 's grad b: Like like what you call certain things , grad a: Yeah . grad b: But in a sense , it 's good to know that he of all people professor c: OK . grad b: you know , like maybe a lot of people would have m much stronger reactions , so , you know , he 's like a relatively friendly linguist grad a: Yeah . professor c: OK , which was that o one of the things I would like to see happen is a paper that was tentatively called " Towards a formal cognitive semantics " which was addressed to these linguists  who haven't been following this stuff at all . professor c: So it could be that he 's actually , at some level , thinking about how am I going to communicate this story grad a: Yeah . professor c: But if he g if he turns is is really gonna turn around and help t to write this version that does connect with as many as possible of the other linguists in the world  then then it becomes important to use terminology that doesn't make it hard grad a:   professor c: it 's gonna be plenty hard for for people to understand it as it is , grad a: Yeah . I just wanted t to I have to catch up with him , and I wanted t to get a feeling for that . grad a: So I don't know what his take will be on these meetings exactly , you know . grad a: Cuz sometimes he sort of sounds like we 're talking a bunch of goobledy - gook from his point of view . grad b: I think it 's good when we 're when we 're into data and looking at the some specific linguistic phenomenon in in English or in German , in particular , whatever , that 's great , professor c: Yeah . grad b: and Ben and and Hans are , if if anything , more you know , they have more to say than , let 's say , I would about some of these things . grad b: But when it 's like , well , w how do we capture these things , you know , I think it 's definitely been Keith and I who have d you know , who have worried more about the grad a:   That 's I I I think that should be the the core group grad b: s Which is fine . professor c: and  that 's , you know , I think very close to the maximum number of people working together that can get something done . And then we 're gon we 're gonna because of this other big thing we haven't talked about is actually implementing this stuff ? So that I guess the three of us are gonna connect tomorrow about that . I was just gonna say , though , that , for instance , there was you know , out of a meeting with Johno came the suggestion that " oh , could it be that the meaning constraints really aren't used for selection ? " which has sort of been implicit in the parsing strategy we talked about . grad b: In which case we w we can just say that they 're the effects or the bindings . Which  , so far , in terms of like putting up all the constraints as , you know , pushing them into type constraints , the when I 've , you know , propo then proposed it to linguists who haven't yet given me you know , we haven't yet thought of a reason that that wouldn't work . professor c: Yeah , it has to in the sense that you 're gonna use them eventu it 's you know , it 's sort of a , generate and test kind of thing , grad b: if there are some constraints that you hold back and don't use  , in your initial matching then you 'll match some things grad b:   professor c: I I d I don't think there 's any way that it could completely fail . It it could be that  , you wind up  The original bad idea of purely context - free grammars died because there were just vastly too many parses . And so th the concern might be that not that it would totally fail , but that grad b:   Right ? So by just having semantic even bringing semantics in for matching just in the form of j semantic types , right ? professor c: it would still genera grad b: Like " conceptually these have to be construed as this , this , and this " might still give us quite a few possibilities professor c: Yeah .  , cuz I think i I think it 's As you know , I think it 's real hard and if w if we Right . grad e: So , you your dance card is completely filled now ? grad a: Shoot . grad e: Why don't grad b: No , that 's not really true , grad a: Bummer . grad b: but like grad d: What about What about DDR ? grad b: It 's almost true . grad f:  grad b: Oh , I don't have it this weekend , so , tsk don't have to worry about that . professor c: DDR , he asked ? grad b: Speaking of dance , Dance Dance Revolution I can't believe I 'm It 's a it 's like a game , but it 's for , like , dancing . Hard to It 's like karaoke , but for dancing , and they tell you what It 's amazing . Well , y you know of it ? I i i it 's one of your hobbies ? It 's great exercise , I must say . They have , like , places instead of like Yeah , instead of karaoke bars now that have , like , DDR , like Yeah , yeah , I didn't until I started hanging out with this friend , who 's like " Oh , well , I can bring over the DDR if you want <doc-sep>OK , so , you 've got some , Xerox things to pass out ? phd a: Yeah , professor d: That are phd a:  .  , lowering the word hour rate is fine , but having big font ! phd a: Next time we will put colors or something . phd a: OK , s so there is kind of summary of what has been done professor d:  Go ahead . So since last week we 've started to fill the column with   features w with nets trained on PLP with on - line normalization but with delta also , because the column was not completely professor d:    phd a: well , it 's still not completely filled , professor d:  phd a: but we have more results to compare with network using without PLP and finally , hhh ,  ehhh PL -  delta seems very important . If you take  , let 's say , anyway Aurora - two - B , so , the next t the second , part of the table , professor d:   phd a:  when we use the large training set using French , Spanish , and English , you have one hundred and six without delta and eighty - nine with the delta . professor d: a And again all of these numbers are with a hundred percent being , the baseline performance , phd a: Yeah , on the baseline , yeah . So professor d: but with a mel cepstra system going straight into the HTK ? phd a: Yeah . So now we see that the gap between the different training set is much   much smaller professor d: Yes . phd a: But , actually , for English training on TIMIT is still better than the other languages . If you take the second set of experiment for Italian , so , the mismatched condition , professor d:   phd a:  when we use the training on TIMIT so , it 's multi - English , we have a ninety - one number , professor d:   phd a: And , yeah , and here the gap is still more important between using delta and not using delta . If y if I take the training s the large training set , it 's we have one hundred and seventy - two , professor d: Yes . Yeah , so the second point is that we have no single cross - language experiments , that we did not have last week .  , so this is training the net on French only , or on English only , and testing on Italian . phd a: And training the net on French only and Spanish only and testing on ,  TI - digits . What we see is that these nets are not as good , except for the multi - English , which is always one of the best . Yeah , then we started to work on a large dat database containing , sentences from the French , from the Spanish , from the TIMIT , from SPINE ,  from  English digits , and from Italian digits . phd a: and  , actually we did this before knowing the result of all the data , so we have to to redo the  the experiment training the net with ,  PLP , but with delta . Well , it 's it 's better than the net using French , Spanish , and English only . The first one , yeah , is combining , two feature streams ,  using and each feature stream has its own MPL . And the third one is to u use a single KLT trans transform features as well as MLP outputs . You know you can you can comment these results , phd b: Yes , I can s I would like to say that , for example , mmm , if we doesn't use the delta - delta ,  we have an improve when we use s some combination . But when phd a: Yeah , we ju just to be clear , the numbers here are  recognition accuracy . phd b: w Yeah , this Yeah , this number recognition acc phd a: So it 's not the Again we switch to another phd b: Yes , and the baseline the baseline have i is eighty - two . phd b: Yeah phd a: So it 's experiment only on the Italian mismatched for the moment for this . phd b: And first in the experiment - one I I do I I use different MLP , professor d:   And I try to combine different type of feature , but the result is that the MSG - three feature doesn't work for the Italian database because never help to increase the accuracy . phd a: Yeah , eh , actually , if w we look at the table , the huge table , we see that for TI - digits MSG perform as well as the PLP , professor d:   phd a: but this is not the case for Italian what where the error rate is c is almost  twice the error rate of PLP . phd a: So , well , I don't think this is a bug but this this is something in probably in the MSG  process that  I don't know what exactly . Perhaps the fact that the the there 's no low - pass filter , well , or no pre - emp pre - emphasis filter and that there is some DC offset in the Italian , or , well , something simple like that . But that we need to sort out if want to  get improvement by combining PLP and MSG professor d:   phd a: And as Carmen said , if we combine the two , we have the result , basically , of PLP . professor d: I  , the  , baseline system when you said the baseline system was  ,  eighty - two percent , that was trained on what and tested on what ? That was ,  Italian mismatched d  , digits , is the testing , phd b: Yeah . professor d: So the " mismatch " just refers to the noise and and ,  microphone and so forth , phd a: Yeah . professor d: right ? So ,  did we have So would that then correspond to the first line here of where the training is is the  Italian digits ? phd b: The train the training of the HTK ? professor d: The phd b: Yes . So ,  So what that says is that in a matched condition , we end up with a fair amount worse putting in the  PLP . Now w would do we have a number , I suppose for the matched I I don't mean matched , but  use of Italian training in Italian digits for PLP only ? phd b:  yes ? phd a:  yeah , so this is basically this is in the table . phd a: Fift - So No , it 's it 's the phd b: No . professor d: No , fifty - two percent of eighty - two ? phd a: Of of of  eighteen phd b: Eighty . So  professor d: Oh this is accuracy !  phd a: so we have nine nine let 's say ninety percent . professor d: OK , so even just PLP , it is not , in the matched condition  I wonder if it 's a difference between PLP and mel cepstra , or whether it 's that the net half , for some reason , is not helping . It 's not Do you have this result with PLP alone , j fee feeding HTK ? professor d: So , s phd a: That That 's what you mean ? phd b: Yeah , phd a: Just PLP at the input of HTK . professor d: so adding MSG phd a:  professor d:  Well , but that 's yeah , that 's without the neural net , phd a: Yeah , that 's without the neural net professor d: right ? phd a: and that 's the result basically that OGI has also with the MFCC with on - line normalization . phd b:  phd a: Eighty - two is the it 's the Aurora baseline , so MFCC . Then we can use well , OGI , they use MFCC th the baseline MFCC plus on - line normalization professor d: Oh , I 'm sorry , I k I keep getting confused because this is accuracy . phd a: So what happ what happens is that when we apply on - line normalization we jump to almost ninety percent . If we use n neural network , even if the features are not correctly normalized , we jump to ninety percent . So professor d: So we go from eighty - si eighty - eight point six to to ninety , or something . phd a: Well , ninety No , I  ninety It 's around eighty - nine , ninety , eighty - eight . So ,  So actually , the answer for experiments with one is that adding MSG , if you  does not help in that case . professor d: The other ones , we 'd have to look at it , but And the multi - English , does  So if we think of this in error rates , we start off with ,  eighteen percent error rate , roughly . professor d:  and we  almost ,  cut that in half by  putting in the on - line normalization and the neural net . About ,  sixteen percent or something of the error , if we use multi - English instead of the matching condition . OK ? So then you 're assuming multi - English is closer to the kind of thing that you could use since you 're not gonna have matching , data for the  for the new for the other languages and so forth .  , one qu thing is that ,  I think I asked you this before , but I wanna double check . When you say " ME " in these other tests , that 's the multi - English , phd a: That 's it 's a part it 's professor d: but it is not all of the multi - English , right ? It is some piece of part of it . professor d: And the multi - English is how much ? phd b: You have here the information . professor d: Oh , so you used almost all You used two thirds of it , phd a: Yeah . So , it it 's still it hurts you seems to hurt you a fair amount to add in this French and Spanish . grad c: Well Stephane was saying that they weren't hand - labeled , phd a: Yeah , it 's phd b: Yeah . Mmm , with the experiment type - two , I first I tried to to combine , nnn , some feature from the MLP and other feature another feature . phd b: And we s we can first the feature are without delta and delta - delta , and we can see that in the situation , the MSG - three , the same help nothing . phd b: And then I do the same but with the delta and delta - delta PLP delta and delta - delta . And they all p but they all put off the MLP is it without delta and delta - delta . And we have a l little bit less result than the the the baseline PLP with delta and delta - delta . phd b: Maybe if when we have the new the new neural network trained with PLP delta and delta - delta , maybe the final result must be better . phd a: Actually , just to be some more phd b:  phd a: Do This number , this eighty - seven point one number , has to be compared with the professor d: Yes , yeah ,  it can't be compared with the other phd a: Which number ? professor d: cuz this is ,  with multi - English , training . professor d: So you have to compare it with the one over that you 've got in a box , which is that ,  the eighty - four point six . professor d: So phd a: Yeah , but  in this case for the eighty - seven point one we used MLP outputs for the PLP net professor d: Yeah . phd b: No , but they they feature @ @ without phd a: So we use feature out  , net outputs together with features . So yeah , this is not perhaps not clear here but in this table , the first column is for MLP and the second for the features . So you 're saying w so asking the question , " What what has adding the MLP done to improve over the , phd a: So , just Yeah so , actually it it it decreased the the accuracy . phd a: And even the MLP alone What gives the MLP alone ? Multi - English PLP . So we have our eighty - three point six and now eighty - eighty point six , phd b: But phd a: that gives eighty - seven point one . Eighty - s I thought it was eighty Oh , OK , eighty - three point six and eighty eighty - eight point six . But I don't know but maybe if we have the neural network trained with the PLP delta and delta - delta , maybe tha this can help . professor d: Well , that 's that 's one thing , but see the other thing is that ,  it 's good to take the difficult case , but let 's let 's consider what that means . What what we 're saying is that one o one of the things that  my interpretation of your your s original suggestion is something like this , as motivation . When we train on data that is in one sense or another , similar to the testing data , then we get a win by having discriminant training . professor d: When we train on something that 's quite different , we have a potential to have some problems . professor d: And , if we get something that helps us when it 's somewhat similar , and doesn't hurt us too much when it when it 's quite different , that 's maybe not so bad . professor d: So the question is , if you took the same combination , and you tried it out on ,  on say digits , phd a: On TI - digits ? OK .  , then does that , eh you know maybe with similar noise conditions and so forth , does it does it then look much better ? phd a:   professor d: And so what is the range over these different kinds of  of tests ? So , an anyway . phd b: And , with this type of configuration which I do on experiment using the new neural net with name broad klatt s twenty - seven , d I have found more or less the same result . phd a: So , it 's slightly better , phd b: Little bit better ? phd a: yeah . professor d: And and you know again maybe if you use the , delta there , you would bring it up to where it was ,  you know at least about the same for a difficult case . phd a: It 's either less information from the neural network if we use only the silence output . And then w with the first configuration , I f I am found that work , doesn't work professor d: Yeah . Because I for the del Engli - PLP delta and delta - delta , here I have eighty - five point three accuracy , and with the second configuration I have eighty - seven point one . professor d: by the way , there is a another , suggestion that would apply , to the second configuration , which , was made , by , Hari . And that was that , if you have  feed two streams into HTK , and you , change the ,  variances if you scale the variances associated with ,  these streams  , you can effectively scale the streams . Right ? So , you know , without changing the scripts for HTK , which is the rule here , you can still change the variances phd a:   professor d: And , so , if you do that , for instance it may be the case that , the MLP should not be considered as strongly , for instance . professor d: And , so this is just setting them to be , excuse me , of equal equal weight . professor d: Right ? You know , I I 'm sorry to say that gives more experiments if we wanted to look at that , but but , you know on the other hand it 's just experiments at the level of the HTK recognition . Yeah , you have to change the No , you can just do it in as once you 've done the training grad c: And then you can vary it . professor d: Yeah , the training is just coming up with the variances so I guess you could you could just scale them all . But Is it i th  the HTK models are diagonal covariances , so I d Is it professor d: That 's  , exactly the point , I think , that if you change  , change what they are phd a:  . professor d: It 's diagonal covariance matrices , but you say what those variances are . professor d: So , that you know , it 's diagonal , but the diagonal means th that then you 're gonna it 's gonna it 's gonna internally multiply it and and  , i it im  implicitly exponentiated to get probabilities , and so it 's it 's gonna it 's it 's going to affect the range of things if you change the change the variances of some of the features . phd b: do ? professor d: So , i it 's precisely given that model you can very simply affect , the s the strength that you apply the features . So it could just be that h treating them equally , tea treating two streams equally is just just not the right thing to do . Of course it 's potentially opening a can of worms because , you know , maybe it should be a different number for for each kind of test set , or something , phd a:   professor d: So I guess the other thing is to take you know if one were to take , you know , a couple of the most successful of these , phd a: Yeah , and test across everything . phd a: So , the next point , yeah , we 've had some discussion with Steve and Shawn , about their  , articulatory stuff ,  . phd a: discussion with Hynek , Sunil and Pratibha for trying to plug in their our our networks with their within their block diagram , where to plug in the the network , after the the feature , before as  a as a plugin or as a anoth another path , discussion about multi - band and TRAPS , actually Hynek would like to see , perhaps if you remember the block diagram there is , temporal LDA followed b by a spectral LDA for each  critical band . And he would like to replace these by a network which would , make the system look like a TRAP . Basically , this is a TRAP system kind of TRAP system , but where the neural network are replaced by LDA .  , yeah , and about multi - band , I started multi - band MLP trainings ,  mmh Actually , I w I w hhh prefer to do exactly what I did when I was in Belgium . So I take exactly the same configurations , seven bands with nine frames of context , and we just train on TIMIT , and on the large database , so , with SPINE and everything . So , this would would be something between TRAPS and multi - band because we still have quite large bands , and but with a lot of context also . So  Yeah , we still have to work on Finnish , basically , to make a decision on which MLP can be the best across the different languages .  , well , the next part of the document is , well , basically , a kind of summary of what everything that has been done . We have seventy - nine M L Ps trained on one , two , three , four , three , four , five , six , seven ten on ten different databases . phd a: the number of frames is bad also , so we have one million and a half for some , three million for other , and six million for the last one .  , yeah ! As we mentioned , TIMIT is the only that 's hand - labeled , and perhaps this is what makes the difference . First ,  with respect to the on - line normalization , there are that use bad on - line normalization , and other good on - line normalization . With respect to the features , with respect to the use of delta or no ,  with respect to the hidden layer size and to the targets . s What 's this ? We only have two hundred eighty six different tests And no not two thousand . phd b: I say this morning that @ @ thought it was the professor d: Alright , now I 'm just slightly impressed , OK . but when the M - MLP is trained on the  is not trained on the target task , it increased the error rate compared to using straight features . Except if the features are bad  , actually except if the features are not correctly on - line normalized . In this case the tandem is still better even if it 's trained on not on the target digits . phd a: so the fourth point is , yeah , the TIMIT plus noise seems to be the training set that gives better the best network . professor d: So , on the MSG  problem  , I think that in in the  , in the short time solution  , that is , trying to figure out what we can proceed forward with to make the greatest progress , phd a:   professor d: much as I said with JRASTA , even though I really like JRASTA and I really like MSG , phd a:   professor d: I think it 's kind of in category that it 's , it it may be complicated . professor d: And  it might be if someone 's interested in it , certainly encourage anybody to look into it in the longer term , once we get out of this particular rush  for results . professor d: But in the short term , unless you have some some s strong idea of what 's wrong , phd a: I don't know at all but I 've perhaps I have the feeling that it 's something that 's quite quite simple or just like nnn , no high - pass filter professor d: Yeah , probably . professor d: There 's supposed to well MSG is supposed to have a an on - line normalization though , right ? phd a: It 's There is , yeah , an AGC - kind of AGC . professor d: Yeah , but also there 's an on - line norm besides the AGC , there 's an on - line normalization that 's supposed to be  , yeah , phd a: Mmm . professor d: In fac in fact the on - line normalization that we 're using came from the MSG design , phd a:  . Are your results are still with the bad the bad phd b: Maybe , may No ? With the better phd a: With the O - OLN - two ? phd b: No ? phd a: Ah yeah , you have you have OLN - two , phd b: Oh ! Yeah , yeah , yeah ! With " two " , with " on - line - two " . professor d: " Two " is good ? phd a: And professor d: No , " two " is bad . It 's probably something simple  , i if if  someone , you know , wants to play with it for a little bit . professor d: but but my my guess would be that it 's something that is a simple thing that could take a while to find . That 's that what we were concerned about is that if it 's not on the target task If it 's on the target task then it it it helps to have the MLP transforming it . professor d: If it  if it 's not on the target task , then , depending on how different it is ,  you can get  , a reduction in performance . professor d: And the question is now how to how to get one and not the other ? Or how to how to ameliorate the the problems . professor d: because it it certainly does is nice to have in there , when it when there is something like the training data . So , the the reason Yeah , the reason is that the perhaps the target the the task dependency the language dependency , and the noise dependency professor d: So that 's what you say th there . phd a: Well , the e e But this is still not clear because , I I I don't think we have enough result to talk about the the language dependency . Well , the TIMIT network is still the best but there is also an the other difference , the fact that it 's it 's hand - labeled .  , I d I don't think we want to mess with the microphones but it 's  Just  , have a seat . s Summary of the first  ,  forty - five minutes is that some stuff work and works , and some stuff doesn't OK , phd a: We still have  this One of these perhaps ? phd b: Yeah .  professor d: Yeah , I guess we can do a little better than that but I think if you if you start off with the other one , actually , that sort of has it in words and then th that has it the associated results . So you 're saying that  , although from what we see , yes there 's what you would expect in terms of a language dependency and a noise dependency . That is , when the neural net is trained on one of those and tested on something different , we don't do as well as in the target thing . But you 're saying that  , it is Although that general thing is observable so far , there 's something you 're not completely convinced about . What what do you mean ? phd a: mmm , that the the fact that s Well , for for TI - digits the TIMIT net is the best ,  which is the English net . But you have two two effects , the effect of changing language and the effect of training on something that 's Viterbi - aligned instead of hand hand - labeled . professor d: Do you think the alignments are bad ?  , have you looked at the alignments at all ? What the Viterbi alignment 's doing ? phd a: Mmm . Because , that is just looking but  ,  It 's not clear to me you necessarily would do so badly from a Viterbi alignment . But , perhaps it 's not really the the alignment that 's bad but the just the ph phoneme string that 's used for the alignment professor d: Aha ! phd a: Mmm .  professor d: The pronunciation models and so forth phd a:  for We It 's single pronunciation ,  professor d: Aha . phd a: so we asked people to listen to the  the sentence and we gave the phoneme string and they kind of correct them .  , the third The third  issue is the noise dependency perhaps but , well , this is not clear yet because all our nets are trained on the same noises and professor d: I thought some of the nets were trained with SPINE and so forth . professor d: OK , yeah , just don't just need more more results there with that @ @ .  , with respect to the network size , there 's one experiment that 's still running and we should have the result today , comparing network with five hundred and one thousand units . We can , we can tell which training set gives the best result , but we don't know exactly why . professor d: " Multi - multi - English " just means " TIMIT " , phd a: Yeah . professor d: It 's sort of , yes it 's mul it 's multi -  - purpose . So Yeah , the training targets actually , the two of the main issues perhaps are still the language dependency and the noise dependency . And perhaps to try to reduce the language dependency , we should focus on finding some other kind of training targets . For moment you use we use phonetic targets but we could also use articulatory targets , soft targets , and perhaps even ,  use networks that doesn't do classification but just regression so  , train to have neural networks that  , professor d:   phd a: does a regression and well , basically com com compute features and noit not , nnn , features without noise . phd a:  professor d: Yeah , that seems like a good thing to do , probably ,  not  again a short - term sort of thing . professor d:  one of the things about that is that  it 's e u the ri I guess the major risk you have there of being is being dependent on very dependent on the kind of noise and and so forth . phd a: So , this is w w i wa wa this is one thing , this this could be could help could help perhaps to reduce language dependency and for the noise part  we could combine this with other approaches , like , well , the Kleinschmidt approach . I think Kleinschmidt was using more than fifty different noises to train his network , phd b: Yeah . phd a: and So this is one approach and the other is multi - band  , that I think is more robust to the noisy changes . phd a: So perhaps , I think something like multi - band trained on a lot of noises with  , features - based targets could could could help . professor d: Yeah , if you i i It 's interesting thought maybe if you just trained up  w yeah , one one fantasy would be you have something like articulatory targets and you have  some reasonable database ,  but then which is  copied over many times with a range of different noises , phd a:   professor d: And  If Cuz what you 're trying to do is come up with a a core , reasonable feature set which is then gonna be used  , by the the   system . The future work is , well , try to connect to the to make to plug in the system to the OGI system . professor d: And I guess , you know , the the the real open question , e u there 's lots of open questions , but one of the core quote " open questions " for that is  , if we take the  you know , the best ones here , maybe not just the best one , but the best few or something You want the most promising group from these other experiments .  , how well do they do over a range of these different tests , not just the Italian ? phd a: Mmm , professor d:  . professor d: y Right ? And then  then see , again , how We know that there 's a mis there 's a  a a loss in performance when the neural net is trained on conditions that are different than than ,  we 're gonna test on , but well , if you look over a range of these different tests  , how well do these different ways of combining the straight features with the MLP features ,  stand up over that range ? phd b:   And  , take let 's say , just take  multi - English cause that works pretty well for the training . How does that How does that compare between the phd a: So all the all the test sets you mean , yeah . professor d: All the different test sets , phd a: And professor d: and for and for the couple different ways that you have of of of combining them . And perhaps doing this for cha changing the variance of the streams and so on getting different scaling phd b:   Yeah , so thi this sh would be more working on the MLP as an additional path instead of an insert to the to their diagram . Perhaps the insert idea is kind of strange because nnn , they they make LDA and then we will again add a network does discriminate anal nnn , that discriminates , professor d: Yeah . It 's a little strange phd a: or ? Mmm ? professor d: but on the other hand they did it before . And because also perhaps we know that the when we have very good features the MLP doesn't help .  , we we wanna get their path running here , right ? If so , we can add this other stuff . professor d: as an additional path right ? phd a: Yeah , the the way we want to do professor d: Cuz they 're doing LDA RASTA . phd a: The d What ? professor d: They 're doing LDA RASTA , phd a: Yeah , the way we want to do it perhaps is to just to get the VAD labels and the final features . professor d: yeah ? phd a: So they will send us the Well , provide us with the feature files , professor d: I see . phd a: and with VAD  , binary labels so that we can  , get our MLP features and filter them with the VAD and then combine them with their f feature stream . First thing of course we 'd wanna do there is to make sure that when we get those labels of final features is that we get the same results as them . You mean Oh , yeah ! Just re re retraining r retraining the HTK ? professor d: Yeah just th w i i Just to make sure that we have we understand properly what things are , our very first thing to do is to is to double check that we get the exact same results as them on HTK . But ,  just for the testing , jus just make sure that we get the same results so we can duplicate it before we add in another phd a: Mmm . Yeah , so fff , LogRASTA , I don't know if we want to We can try networks with LogRASTA filtered features . But professor d: Oh ! You know , the other thing is when you say comb I 'm I 'm sorry , I 'm interrupting . that u  , when you 're talking about combining multiple features ,  Suppose we said , " OK , we 've got these different features and so forth , but PLP seems pretty good . We have different languages , we have different different noises ,  If we have some drastically different conditions and we just train up different M L Ps with them . What what What Mike found , for the reverberation case at least , who knows if it 'll work for these other ones . That is , that yes , if you knew what the reverberation condition was gonna be and you trained for that , then you got the best results . But if you had , say , a heavily - reverberation ca heavy - reverberation case and a no - reverberation case , and then you fed the thing ,  something that was a modest amount of reverberation then you 'd get some result in between the two . So you you think it 's perhaps better to have several M L Yeah but professor d: It works better if what ? phd a: Yea professor d: I see . Well , see , i oc You were doing some something that was So maybe the analogy isn't quite right . Here the problem seems to be is that we don't have a hug a really huge net with a really huge amount of training data . But we have s f for this kind of task , I would think , sort of a modest amount . We have a modest amount of of  training data from a couple different conditions , and then  in yeah , that and the real situation is that there 's enormous variability that we anticipate in the test set in terms of language , and noise type  , and  , channel characteristic , sort of all over the map . And so , I 'm just concerned that we don't really have  , the data to train up  one of the things that we were seeing is that when we added in we still don't have a good explanation for this , but we are seeing that we 're adding in  , a fe few different databases and  the performance is getting worse and  , when we just take one of those databases that 's a pretty good one , it actually is is is is is better . And  that says to me , yes , that , you know , there might be some problems with the pronunciation models that some of the databases we 're adding in or something like that . But one way or another we don't have  , seemingly , the ability to represent , in the neural net of the size that we have , all of the variability that we 're gonna be covering . So that I 'm I 'm I 'm hoping that  , this is another take on the efficiency argument you 're making , which is I 'm hoping that with moderate size neural nets , that  if we if they look at more constrained conditions they they 'll have enough parameters to really represent them .  i i e The  I think it 's true that the OGI folk found that using LDA RASTA , which is a kind of LogRASTA , it 's just that they have the  it 's done in the log domain , as I recall , and it 's it  it 's just that they d it 's trained up , right ? That that  benefitted from on - line normalization . So will it be in our case , where we 're using the neural net ?  they they were not not using the neural net . We 've been sort of ignoring that , haven't we ? phd a: Yeah , so I don't know . professor d: But phd a: But we have to address the problem of CPU and memory we professor d: Yeah , but I li Well , I think My impression You you folks have been looking at this more than me . But my impression was that  , there was a a a a strict constraint on the delay , phd b: Yeah . professor d: but beyond that it was kind of that  using less memory was better , and using less CPU was better . We have to get some reference point to where we Well , what 's a reasonable number ? Perhaps be because if it 's if it 's too large or large or @ @ professor d: well I don't think we 're  completely off the wall .  I think that if we if we have  ,  the ultimate fall back that we could do If we find   we may find that we we 're not really gonna worry about the M L You know , if the MLP ultimately , after all is said and done , doesn't really help then we won't have it in . professor d: If the MLP does , we find , help us enough in some conditions , we might even have more than one MLP . professor d: And it 's  We do the other manipulations that we 're doing before that .  , what what are they What are they gonna be working Do we know what they 're gonna be working on while we take their features , phd a: They 're They 're starting to wor work on some kind of multi - band . Sunil , what was he doing , do you remember ? phd b: Sunil ? phd a: Yeah . He was doing something new or ? phd b: I I don't re I didn't remember . phd a: I think they were also mainly , well , working a little bit of new things , like networks and multi - band , but mainly trying to tune their their system as it is now to just trying to get the best from this this architecture . So I guess the way it would work is that you 'd get There 'd be some point where you say , " OK , this is their version - one " or whatever , and we get these VAD labels and features and so forth for all these test sets from them , phd a:   We have a certain level we try to improve it with this other path and then  , when it gets to be  , January some point  , we say , " OK we we have shown that we can improve this , in this way . So now   what 's your newest version ? " And then maybe they 'll have something that 's better and then we we 'd combine it .  I I I used to work with  folks who were trying to improve a good  ,  system with  with a neural net system and  , it was a common problem that you 'd Oh , and this Actually , this is true not just for neural nets but just for in general if people were working with  , rescoring  , N - best lists or lattices that come came from  , a mainstream recognizer .  , You get something from the the other site at one point and you work really hard on making it better with rescoring . So by the time you have  , improved their score , they have also improved their score phd a: Mmm . professor d: So , I guess at some point we 'll have to phd a: So it 's professor d: I I don't know . I think we 're we 're integrated a little more tightly than happens in a lot of those cases . I think at the moment they they say that they have a better thing we can we e e phd a: Mmm . professor d: What takes all the time here is that th we 're trying so many things , presumably  , in a in a day we could turn around  , taking a new set of things from them and and rescoring it , phd a: Mmm . I think that the most wide open thing is the issues about the  , you know , different trainings . So we we can for we c we can forget combining multiple features and MLG perhaps , professor d: That 's sort of wide open . phd a: or focus more on the targets and on the training data and ? professor d: Yeah , I think for right now  , I th I I really liked MSG . And I think that , you know , one of the things I liked about it is has such different temporal properties . And  , I think that there is ultimately a really good  , potential for , you know , bringing in things with different temporal properties .  , but  , we only have limited time and there 's a lot of other things we have to look at . professor d: And it seems like much more core questions are issues about the training set and the training targets , and fitting in  what we 're doing with what they 're doing , and , you know , with limited time . And then , you know , once we  , having gone through this process and trying many different things , I would imagine that certain things  , come up that you are curious about  , that you 'd not getting to and so when the dust settles from the evaluation  , I think that would time to go back and take whatever intrigued you most , you know , got you most interested  and  and and work with it , you know , for the next round .  , as you can tell from these numbers  , nothing that any of us is gonna do is actually gonna completely solve the problem . professor d: Well I figured that , but That what what what were you involved in in this primarily ? grad c: helping out  , preparing Well , they 've been kind of running all the experiments and stuff and I 've been  ,  w doing some work on the on the preparing all all the data for them to to  , train and to test on . Right now , I 'm I 'm focusing mainly on this final project I 'm working on in Jordan 's class . What 's what 's that ? grad c: I 'm trying to  So there was a paper in ICSLP about  this this multi - band  , belief - net structure . grad c:  basically it was two H M Ms with with a with a dependency arrow between the two H M professor d:  - huh . grad c: And so I wanna try try coupling them instead of t having an arrow that that flows from one sub - band to another sub - band . And  , I 'm just gonna see if if that that better models  ,  asynchrony in any way or  Yeah . OK , so speaking of which , if we don't have anything else that we need You happy with where we are ? phd a: Mmm . professor d: Know know wher know where we 're going ?  phd a: I think so , yeah . grad e: Al - actually I should mention So if  , about the Linux machine " Swede . grad e: And  Dan Ellis I believe knows something about using that machine so phd a: Mmm . grad e: If people are interested in in getting jobs running on that maybe I could help with that . phd a: Yeah , but I don't know if we really need now a lot of machines . professor d: Right ?  there 's there 's some different things that we 're trying to get at now . Yeah , as far as you can tell , you 're actually OK on C - on CPU  , for training and so on ? Yeah . Well , more is always better , but mmm , I don't think we have to train a lot of networks , now that we know We just select what works fine professor d: OK . to work professor d: And we 're OK on And we 're OK on disk ? phd a: and It 's OK , yeah . Alright , so  , since  , we didn't ha get a channel on for you , you don't have to read any digits but the rest of us will . We didn't  I think I won't touch anything cuz I 'm afraid of making the driver crash which it seems to do , pretty easily . OK , so we 'll  I 'll start off the   connect the phd a: My battery is low 
Summary:
The professor expressed surprise at the better-than-expected results and concluded that MSG had no impact. However, he observed that the inclusion of French and Spanish in the multilingual English model had a negative effect. The professor was not enthusiastic about clustering and suggested giving more weight to the most promising bands in the model. Currently, the model treated all bands equally. In terms of the SmartKom generation module, the syntax-to-prosody rules for English will be rewritten, and OGI will provide a variety of synthesizer voices to choose from.
What were the Professor's thoughts on the results and clustering in language generation?