phd e: What am I ? professor f: Little low ? phd e: Channel four ? professor f: Channel five . phd g: The gai the gain 's up at it what it usually is , professor f: Is it ? phd g: but if you think it 's Yeah . phd d: I think phd g: She can just walk in , I guess , or phd d: Yeah . I was gonna ask Adam to , say if he thought anymore about the demo stuff because it occurred to me that this is late May and the DARPA meeting is in mid July .  , but I don't remember w what we I know that we were gonna do something with the transcriber interface is one thing , but I thought there was a second thing . Anybody remember ? phd g: Well , we were gonna do a mock - up , like , question answering or something , I thought , that was totally separate from the interface . Do you remember ? Remember , like , asking questions and retrieving , but in a pre - stored fashion . phd g: That was the thing we talked about , I think , before the transcriber professor f: Yeah . So anyway , you have to sort out that out and get somebody going on it cuz we 're got a got a month left basically . phd g: I was writing the digits and then I realized I could xerox them , professor f: Oh , oh . professor f: So , the choice is , which which do we want more , the the the comparison , of everybody saying them at the same time or the comparison of people saying the same digits at different times that ? phd g: It 's just cuz I didn't have any more digit sheets .  , we could use them for normalizing or something , but it of course goes more quickly doing them in unison . phd e: But anyway , they won't be identical as somebody is saying zero in some sometimes , you know , saying O , and so , it 's not i not identical .  ju what what might those be ? phd d: IBM stuff and , just getting  , meeting information organized . phd c: Are you implying that it 's currently disorganized ? phd d: In my mind . professor f: Is there stuff that 's happened about , the SRI recognizer et cetera , tho those things that were happening before with ? phd c: Well . professor f: Y y you guys were doing a bunch of experiments with different front - ends and then with Is is that still sort of where it was , the other day ? phd c: We 're improving . phd d: Now the the You saw the note that the PLP now is getting basically the same as the MFCC .  , it seems It looks l I haven't The It 's The experiment is still not complete , but , it looks like the vocal tract length normalization is working beautifully , actually , w using the warp factors that we computed for the SRI system and just applying them to the ICSI front - end . phd c: Just had to take the reciprocal of the number because they have different meanings in the two systems .  phd c: But one issue actually that just came up in discussion with Liz and and Don was , as far as meeting recognition is concerned , we would really like to , move , to , doing the recognition on automatic segmentations . phd c: Because in all our previous experiments , we had the  , you know , we were essentially cheating by having the , you know , the h the hand - segmentations as the basis of the recognition . phd c: And so now with Thilo 's segmenter working so well , I think we should consider doing a phd e: Mmm . phd g: And even The good thing is that since you , have high recall , even if you have low precision cuz you 're over - generating , that 's good because we could train noise models in the recognizer for these kinds of , transients and things that come from the microphones , phd c: Right . phd g: but I know that if we run recognition unconstrained on a whole waveform , we do very poorly because we 're we 're getting insertions in places what that you may well be cutting out . phd c: We should we should consider doing some extra things , like , you know , retraining or adapting the the models for background noise to the to this environment , for instance . phd g: And , yeah , using Thilo 's , you know , posteriors or some kind of or phd c: So . So , talked with Brian and gave him the alternatives to the single beep at the end of each utterance that we had generated before . And so he talked it over with the transcriber and the transcriber thought that the easiest thing for them would be if there was a beep and then the nu a number , a digit , and then a beep , at the beginning of each one professor f: Yeah . And , so Adam wrote a little script to generate those style , beeps phd c: Where 'd you get the digits from ? phd d: and so we 're I came up here and just recorded the numbers one through ten . phd c: And do you splice them into the waveform ? Or ? phd d: Yeah . He then he d I recorded Actually , I recorded one through ten three times at three different speeds and then he picked . phd d: He liked the fastest one , so he just cut those out and spliced them in between , two beeps . phd d: Does it ? phd e: It will be funny when you 're really reading digits , and then there are the chunks with with your digits in ? phd d: Yeah . postdoc a: Now actually , phd d: That 'll throw them , postdoc a: we 're Are we handling ? phd d: huh ? professor f: maybe we should have you record A , B , C for those or something . And she said it wasn't gonna the transcriber said it wouldn't be a problem cuz they can actually make a template , that has beep , number , beep . phd d: So , we We 're gonna send them one more sample meeting , and Thilo has run his segmentation . And when we get that back we 'll see if that sort of fixes the problem we had with , too many beeps in the last transcription . Do w do what Do you have any idea of the turn - around on on those steps you just said ? phd g: Great . professor f: e e u u The reason I 'm asking is because , Jane and I have just been talking , and she 's just been doing . professor f: And so we don't sort of really know exactly what they 'll be doing , how long they 'll be doing it , and so forth , because right now she has no choice but to operate in the mode that we already have working . professor f: And , so it 'd be It 'd be good to sort of get that resolved , soon as we could , phd d: Yeah . I Yeah , I I hope @ @ we can get a better estimate from this one that we send them .   in particular I would I would really hope that when we do this DARPA meeting in July that we sort of have we 're we 're into production mode , somehow phd d:   professor f: You know , that we we actually have a stream going and we know how how well it does and how and how it operates . Maybe before we do the meeting info organize thing , maybe you could say relevant stuff about where we are in transcriptions . So , we  , the transcribers have continued to work past what I 'm calling " set one " , which was the s the set that I 've been ,  OK , talking about up to this point , but , they 've gotten five meetings done in that set . I 'm thinking of hiring another one , which will because we 've had a lot of attrition . And that will bring our total to professor f: They die off after they do this for a while .  , you know , one of them really w wasn't planning phd c: Oh , that was an unfor unforeseen side effect of postdoc a: Eh , one of them , had never planned to work past January .  , it 's th all these various things , cuz we , you know , we presented it as possibly a month project back in January and and and and  , so it makes sense .  , through attrition we we 've we 're down to to two , but they 're really solid . We had a good core phd g: Well , they won't hear this since they 're going .  , I d it 's just a matter of we w we 're we 've got , professor f: No backs . postdoc a: two of the ones who who , ha had been putting in a lot of hours up to this point and they 're continuing to put in a a lot of hours , which is wonderful , and excellent work . And so , then , in addition , I hired two more today and I 'm planning to h hire a third one with this within this coming week , but but the plan is just as , Morgan was saying we discussed this , and the plan right now is to keep the staff on the on the leaner side , you know , rather than hiring , like , eight to ten right now , professor f:   postdoc a: because if the IBM thing comes through really quickly , then , we wouldn't wanna have to , you know , lay people off and stuff . And this way it 'll  , I got really a lot of response for for my notice and I think I could hire additional people if I wish to . An - and the other thing is , in the unlikely event and since we 're so far from this , it 's a little hard to plan this way in the unlikely event that we actually find that we have , transcribers on staff who are twiddling their thumbs because , you know , there 's , you know , all the stuff that that was sitting there has been transcribed and they 're and they 're faster the the pipeline is faster than  , than the generation , eh , i in in the day e event that that day actually dawns , I I bet we could find some other stuff for them to do . professor f: So I I think that , eh , eh , a as we were talking , if we if we hire twelve , then we could , you know , run into a problem later . But but ,  for all sorts of reasons but if we hire f you know , f we have five on staff five or six on staff at any given time , then it 's a small enough number so we can be flexible either way . phd g: It 'd be great , too , if , we can we might need some help again getting the tighter boundaries or some hand to experiment with ,  you know , to have a ground truth for this segmentation work , which I guess you have some already that was really helpful , and we could probably use more . That was a thing I I planned working on , is , to use the the transcriptions which are done by now , and to to use them as ,  phd g: Oh . To to create some speech - nonspeech labels out of them , and Yeah , but that that 's a thing w was w what I 'm just looking into . postdoc a: The the the pre - segmentations are so much are s so extremely helpful . Now there was , I g guess So , a couple weeks ago I needed some new ones and it happened to be during the time that he was on vacation f for just very few days you were away . postdoc a: so I so I started them on the non - pre - segmented and then switched them over to yours and , they ,  you know , they always appreciate that when they have that available . postdoc a: And and and she was And so , I asked her  , They 're very perceptive . I haven't done it yet , but I wanna do that and she 's out of town , for a couple of weeks , but I wanna do that when she returns .  , cuz she was saying , you know , in a in a span of very short period we asked It seems like the ones that need to be adjusted are these these these things , and she was saying the short utterances , the ,  phd g:  . But but actually i it 's so correct for so much of the time , that it 's an enormous time saver phd e: Yeah . phd g: Is there actually a record of where they change ?  , you can compare , do a diff on the just so that we knew postdoc a: You could do it . It 's it 's complicated in that  , hhh , i hhh , i phd e: Yeah . Actually , when when they create new yeah , new segments or something , it will be , not that easy but  . phd g: just so that if we run it we know whether we 're which ones were cheating phd e: Yeah . phd g: and postdoc a: There is a there is one problem with that and that is when they start part way through then what I do is I merge what they 've done with the pre - segmented version . Wha - what you 'd really like is that they started with pre - segmented and were pre - segmented all the way through . postdoc a: And ,  @ @ I ,  the it wasn't possible for about four of the recent ones . As long as we have a record , I guess , of the original automatic one , we can always find out how well we would do fr from the recognition side by using those boundaries . phd g: Also if you need someone to record this meeting , I 'm happy to for the transcribers I could do it , or Chuck or Adam . So , Jane and Adam and I had a meeting where we talked about the reorganization of the directory structure for all of the meeting professor f: Did you record it ? phd d: No . And then , Jane also s prepared a  , started getting all of the the meetings organized , so she prepared a a spreadsheet , which I spent the last couple of days adding to . So I went through all of the data that we have collected so far , and have been putting it into , a spreadsheet with start time , the date , the old meeting name , the new meeting name , the number of speakers , the duration of the meeting , comments , you know , what its transcription status is , all that kind of stuff . And so , the idea is that we can take this and then export it as HTML and put it on the Meeting Recorder web page so we can keep people updated about what 's going on . phd d: I 've gotta get some more information from Jane cuz I have some some gaps here that I need to get her to fill in , but so far , as of Monday , the fourteenth , we 've had a total number of meeting sixty - two hours of meetings that we have collected .  , and I 'm gonna have on here the total amount that 's been transcribed so far , but I 've got a bunch of  , that 's what I have to talk to Jane about , figuring out exactly which ones have have been completed and so forth . But , this 'll be a nice thing that we can put up on the the web site and people can be informed of the status of various different ones . And it 'll also list , like under the status , if it 's at IBM or if it 's at ICSI , or if it 's completed or which ones we 're excluding and and there 's a place for comments , so we can , say why we 're excluding things and so forth . professor f: Now would the ones that , are already transcribed we h we have enough there that c you know , we 've already done some studies and so forth and  , shouldn't we go through and do the business - es u of of having the , participants approve it , for approve the transcriptions for distribution and so forth ? postdoc a: interesting idea . In principle , I I would say yes , although I still am doing some the final - pass editing , trying to convert it over to the master file as the being the channelized version and it 's Yeah , it seems like I get into that a certain way and then something else intervenes and I have to stop . Cleaning up the things like the , places where the transcriber was uncertain , and and doing spot - checking here and there . So , I guess it would make sense to wait until th that 's done , but but professor f: Well , le let me put in another sort of a milestone kind of as as I did with the ,  the the pipeline . professor f: we are gonna have this DARPA meeting in the middle of July , postdoc a: Yes . professor f: and I think it w it 'd be given that we 've been we 've given a couple public talks about it already , spaced by months and months , I think it 'd be pretty bad if we continued to say none of this is available . So we can s we we wanna be able to say " here is a subset that is available right now " postdoc a:   phd c: And they don't have to approve , you know , th an edited version , they can just give their approval to whatever version postdoc a: Well , maybe professor f: Well , in principle , yes . But , i if if if somebody actually did get into some legal issue with it then we phd c: Bu Yeah . Presumably if if s errors are found , they will be fixed , but they won't change the the content of the meetings . phd g: Well , i if Jane is clarifying question question , then , you know , how can they agree to it before they know her final version ? postdoc a: The other thing , too , is there can be subtleties where a person uses this word instead of that word , which @ @ could 've been transcribed in the other way . phd g: Thing postdoc a: And no and they wouldn't have been slanderous if it had been this other word . You know ? professor f: I it you know , there there is a point at which I agree it becomes ridiculous because , you know , you could do this final thing and then a year from now somebody could say , you know , that should be a period and not a question mark . Right ? And you don't you there 's no way that we 're gonna go back and ask everybody " do you approve this , you know this document now ? " So So I think what it is is that the the the the thing that they sign I I haven't looked at it in a while , but it has to be open enough that it sort of says " OK , from now on you know , now that I 've read this , you can use do anything you want with these data . professor f: And ,  But , i I think we wanna So , assuming that it 's in that kind of wording , which I don't remember , I think i we just wanna have enough confidence ourselves that it 's so close to the final form it 's gonna be in , a year from now that they 're postdoc a:   It 's just , a question of , if if the person is using the transcript as the way of them judging what they said and whether it was slanderous , then it seems like it 's it 's i it needs to be more correct than if we could count on them re - listening to the meeting . postdoc a: Because it becomes , eh , in a way a a f  , a legal document i if they 've agreed to that . I forget how we ended up on this , but I remember my taking the position of not making it so so easy for everybody to observe everything and Adam was taking the position of of having it be really straightforward for people to check every aspect of it including the audio . And I don't remember who won , Adam or me , but postdoc a: Well , if it 's only the transcript , though  , th this this is my point , that that professor f: the  , that that 's why I 'm bringing this up again , because I can't remember how we ended up . postdoc a: then it becomes professor f: That it was the transcrip He wanted to do a web interface that would make it postdoc a: Well , if it 's just the audio Well . phd g: with the web interface it 's interesting , because you could allow the person who signs to be informed when their transcript changes , or something like that . Like , I don't wanna know , but some people might be really interested and then y In other words , they would be informed if there was some significant change other than typos and things like that . professor f: You decided you were whispering Satanic incantations under your breath when you were phd g: Well , I don't know what happened to the small heads thing , but I j  , I 'm just saying that , like , you know , you can sort of say that any things that are deemed professor f: They disappeared from view .  , I agree that at some point people probably won't care about typos but they would care about significant meaning changes and then they could be asked for their consent , I guess , if if those change . Cuz assumi assuming we we don't really distribute things that have any significant changes from what they sign anyway . phd c: Tha That 's How about having them approve the audio and not the transcripts ? phd g: Oh , my God . grad b: That 's phd c: We just have to give them a chance to listen to it , and if they don't , that 's their problem . phd g: You you d That 's like postdoc a: Unfortunately , in in the sign thing that they signed , it says " transcripts " . postdoc a: " You 'll be you 'll be provided the transcripts when they 're available . phd g: that 's a lot to ask for people that have been in a lot of meetings . professor f: W anyway , haven't we we 've gone down this path a number of times . I know this can lead to extended conversations and and not really get anywhere , so let let me just suggest that  , off - line that , the people involved figure it out and take care of it before it 's July . So so that in July we can tell people " yes , we have this and you can use it " . phd g: it 's I guess one thing we 're learning is that the amount We have eight meetings there because we couldn't use the non - native all non - native meetings and it 's , well , probably below threshold on enough data for us for the things we 're looking at because the prosodic features are very noisy and so you you need a lot of data in order to model them .  , so we 're starting to see some patterns and we 're hoping that maybe with , I don't know , double or triple the data with twenty meetings or so , that we would start to get better results . But we did find that some of the features that , I gue Jane would know about , that are expressing sort of the distance of , boundaries from peaks in the utterance and some local , range pitch range effects , like how close people are to their floor , are showing up in these classifiers , which are also being given some word features that are cheating , cuz they 're true words . Right ? phd g: spurts is not cheating except that of course you know the real words , grad b: Right . phd c: No phd g: Not exactly , but i grad b: But ra somewhat ? professor f: On the average . Well , we don't know and actually that 's one of the things we 're interested in doing , is a sort of professor f:  - huh . phd c: Have you tried using just time , as opposed to number of words ? phd g: So . grad b: I think ti  Just p time position , like when the word starts ? phd c: Yeah . grad b: I don't know if that was in the phd c: Well , no ,  t time time position relative to the beginning of the spurt . phd g: we didn't try it , but it 's s grad b: Like , there 's a lot of different features you could just pull out . professor f: How about time position normalized by speak phd g: And it depends on speaking rate professor f: Yeah . phd g: But we one of the interesting things was I guess you reported on some te punctuation type grad b: Yeah . phd g: finding sentence boundaries , finding disfluency boundaries , and then I had done some work on finding from the foreground speech whether or not someone was likely to interrupt , so where you know , if I 'm talking now and someone and and Andreas is about to interrupt me , is he gonna choose a certain place in my speech , either prosodically or word - based . And there the prosodic features actually showed up and a neat thing even though the word features were available . And a neat thing there too is I tried some putting the speaker So , I gave everybody a short version of their name . So that means that overall , it wasn't just modeling Morgan , or it wasn't just modeling a single person , professor f:   phd g: but was sort of trying to , get a general idea the model the tree classifier was trying to find general locations that were applicable to different speakers , even though there are huge speaker effects . The but the main limitation now is I because we 're only looking at things that happen every ten words or every twenty words , we need more more data and more data per speaker . It 'd also be interesting to look at the EDU meetings because we did include meeting type as a feature , so whether you were in a r Meeting Recorder meeting or a Robustness meeting did matter to interrupts because there are just fewer interrupts in the Robustness meetings . phd g: And so the classifier learns more about Morgan than it does about sort of the average person , professor f:   phd g: So it 's And I think Don ,  Well , we have a long list of things he 's starting to look at now over the summer , where we can And he 'll be able to report on more things in the future . But it was great that we could at least go from the you know , Jane 's transcripts and the , recognizer output and get it to this point . And I think it 's something Mari can probably use in her preliminary report like , " yeah , we 're at the point where we 're training these classifiers and we 're just reporting very preliminary but suggestive results that some features , both word and pro prosodic , work . " The other thing that was interesting to me is that the pitch features are better than in Switchboard . And I think that really is from the close - talking mikes , cuz the pitch processing that was done has much cleaner behavior than than the Switchboard telephone bandwidth . Well , first of all , the pitch tracks are m have less , halvings and doublings than than Switchboard and there 's a lot less dropout , so if you ask how many regions where you would normally expect some vowels to be occurring are completely devoid of pitch information , professor f:    phd g: in other words the pitch tracker just didn't get a high enough probability of voicing for words for for , you know , five word professor f:  . So the missing We had a big missing data problem in Switchboard and , so the features weren't as reliable cuz they were often just not available . phd d: Could it have to do with the the lower frequency cut - off on the Switchboard ? phd g: So that 's actually good .  , the tele we had telephone bandwidth for Switchboard and we had the an annoying sort of telephone handset movement problem that I think may also affect it . phd g: Anyway , Don 's been doing a great job and we hope to continue with , Andreas 's help and also some of Thilo 's help on this , professor f: Great . phd e: Y phd g: to to try to get a non - cheating version of how all this would work . professor f: Has has ,  ? We just I think , just talked about this the other day , but h has has anybody had a chance to try changing , insertion penalty sort of things with the with the , using the tandem system input for the ? phd c: Oh , yeah . phd c: There were a little the relative number of I think there were a higher number of deletions , actually . phd c: So , you ,  So , actually it it preferred to have a positive er , negative insertion penalty , phd g: Deletions ? phd c: which means that ,  professor f:  - huh . phd c: But , you know , it didn't change th the by adjusting that the ,  professor f: OK . But , you know , given that that word error rate is so high , that 's not a professor f: OK . phd c: But , we s just ,  you know , Chuck and I talked and the @ @ next thing to do is probably to tune the  , the size of the Gaussian system , @ @ to to this to this feature vector , which we haven't done at all . phd c: And , for instance , Dan @ @ Dan just sent me a message saying that CMU used , something like ten Gaussians per cluster You know , each each mixture has ten Gaussians phd d:   We 're using sixty - four , phd c: and and we 're using sixty - four , phd d: right ? phd c: so that 's obviously a big difference phd d: Yeah . phd c: and it might be way off and give very poorly trained , you know , Gaussians that way , professor f:  . So so , we have The turn - around time on the training when we train only the a male system with , you know , our small training set , is less than twenty - four hours , so we can run lots of  , basically just brute force , try a whole bunch of different  , settings . professor f: huh ? phd c: But the PLP features work  , you know , continue to improve the , professor f: OK . phd c:  As I said before , the  using Dan 's , vocal tract normalization option works very well . So , @ @ I ran one experiment where we 're just did the vocal tract le normalization only in the test data , professor f:   phd c: so I didn't bother to retrain the models at all , and it improved by one percent , which is about what we get with  , with , you know , just @ @ actually doing both training and test normalization , with , the , with the standard system . So , in a few hours we 'll have the numbers for the for retraining everything with vocal tract length normalization and So , that might even improve it further . phd c: So , it looks like the P L - fea P features do very well now with after having figured out all these little tricks to to get it to work . So you mean you improve one percent over a system that doesn't have any V T L in it already ? phd c: Exactly . And and what that suggests also is of course that the current Switchboard MLP isn't trained on very good features . phd c: because it was trained on whatever , you know , was used , last time you did Hub - five stuff , which didn't have any of the professor f: Right . professor f: Right ?  , y the phd c: Well , but if you add them all up you have , almost five percent difference now . phd c: actually , and it 's , What 's actually qu interesting is that with  , well , you m prob maybe another half percent if you do the VTL in training , and then interestingly , if you optimize you get more of a win out of rescoring the , the N best lists , and optimizing the weights ,  than phd d: Than you do with the standard ? phd c: Yeah . But the part that 's actually adjustment of the front - end per se as opposed to doing putting VTLN in or something is it was a couple percent . professor f: Right ? It was it was there was there was one thing that was one and a half percent and one that was point eight . One of them was , the change to ,  because it did it all at once , to  , from bark scale to mel scale , phd c:   professor f: which I really feel like saying in quotes , because @ @ they 're essentially the same scale but the but but but any i individual particular implementation of those things puts things in a particular place . professor f: So that 's why I wanted to look I still haven't looked at it yet . professor f: and it it 's probably something like there 's one fewer or one more filter in the sub one kilohertz band phd c:   professor f: it could be there 's something more fundamental but it you know , I I don't know it yet . And the other and the other that was like one and a half or something , and then there was point eight percent , which was what was the other thing ? phd d: Well , that was combined with the triangular . professor f: We d weren't able to separate them out cuz it was just done in one thing . So that was that was ,  that one I can claim credit for , i in terms of screwing it up in the first place . So that someone e until someone else fixed it , which is that , I never put when I u We had some problems before with offsets . professor f: So we we had ,  ea everybody else who was doing Wall Street Journal knew that there were big DC offsets in th in these data in those data and and and nobody happened to mention it to us , phd c:  . professor f: and we were getting these , like , really terrible results , like two , three times the error everybody else was getting . And then in casual conversation someone ment mentioned "  , well , I guess , you know , of course you 're taking care of the offsets . professor f: And at that point , you know , we were pretty new to the data and we 'd never really , like , looked at it on a screen and then when we just put it on the screen and wroop ! phd c:   So , in PLP phd g: There was a like a hum or some or when they recorded it ? professor f: No . It 's just , it it 's it 's not at all uncommon for for recorded electronics to have different , DC offsets . It 's you know , you could have ten , twenty , maybe thirty millivolts , whatever , and it 's consistently in there . The thing is , most people 's front - ends have pre - emphasis with it , with zero at zero frequency , so that it 's irrelevant . We had we had the equivalent of pre - emphasis in a a , Fletcher - Munson style weighting that occurs in the middle of P L but it doesn't actually have a zero at zero frequency , phd g:  . So at that point I reali " oh sh we better have a have a high - pass filter " just , you know just take care of the problem . So I put in a high - pass filter at , I think ninety ninety hertz or so  , for a sixteen kilohertz sampling rate . And so well , so , you know , the code doesn't know anything about that and so this is all at eight kilohertz and so it was at forty - five hertz instead of at instead of at ninety . professor f: So , I don't know if Dan fixed it or or , what he phd c: Well , he made it a parameter . Yeah , I guess if he did it right , he did fix it and then and then it 's taking care of sampling rate , which is great . phd d: What what is the parameter ? professor f: He had a phd d: Is it , just the f lower cut - off that you want ? phd c: It 's called , H - HPF . phd c: u And but HPF , you know , when you put a number after it , uses that as the hertz value of the cut - off . professor f: frankly , we never did that with the RASTA filter either , phd c:   professor f: so the RASTA filter is actually doing a different thing in the modulation spectral domain depending on what sampling rate you 're doing , which is another old old bug of mine . So that that was the problem there was th we we we had always intended to cut off below a hundred hertz phd c:   Well , but ,  Well , again , after completing the current experiments , we 'll we can add up all the  differences professor f: Oh , yeah . phd c: and and an professor f: But but , I guess my my point was that that , the hybrid system thing that we did was , primitive in many ways . professor f: And I think I agree with you that if we fixed lots of different things and they would all add up , we would probably have a a a competitive system . professor f: unless you call well , if you call VTL the front - en front - end , that 's , a little more . phd d: One experiment we should we 'll probably need to do though when  , at some point , is , since we 're using that same the net that was trained on PLP without all these things in it , for the tandem system , we may wanna go back and retrain , professor f: Right ? But . phd c: eh At this point I 'm as  , you know e I 'm wondering is it Can we expect , a tandem system to do better than a properly trained you know , a Gaussian system trained directly on the features with , you know , the right ch choice of parameters ? professor f: Well , that 's what we 're seeing in other areas . Right ? So , it 's so ,  phd d: So , we But but we may not . professor f: the reason to think it should is because you 're putting in the same information and you 're transforming it to be more discriminative . Now the thing is , in some databases I wouldn't expect it to necessarily give you much and and part of what I view as the real power of it is that it gives you a transformational capability for taking all sorts of different wild things that we do , not just th the standard front - end , but other things , like with multiple streams and so forth , phd c:   professor f: If it 's , you know , like way way worse then , you know phd c: Right . phd d: So , Morgan , an another thing that Andreas and I were talking about was , so @ @ in the first experiment that he did we just took the whole fifty - six , outputs and that 's , basically compared to a thirty - nine input feature vector from either MFCC or PLP . phd d: But one thing we could do is professor f: Let let me let me just ask you something . When you say take the fifty - six outputs , these are the pre final nonlinearity outputs phd d: Yeah . And so so then you u Do you use all fifty - six of the KLT phd d: That 's what we did . professor f: or ? phd d: Right ? So one thing we were wondering is , if we did principal components and , say , took out just thirteen , and then did deltas and double - deltas on that professor f: OK . professor f:  these were all different databases and different you know , in HTK and all that , phd d: Yeah . But my recollection of it was that it didn't make it better but it didn't make it worse . professor f: But , again , given all these differences , maybe it 's more important in your case that you not take a lot of these low - variance , components . phd d: Cuz in a sense , the net 's already got quite a bit of context in those features , professor f: Yeah . phd d: so if we did deltas and double - deltas on top of those , we 're getting sort of even more . phd c: But there the main point is that , you know , it took us a while but we have the procedure for coupling the two systems debugged now and  , there 's still conceivably some bug somewhere in the way we 're feeding the tandem features  , either generating them or feeding them to this to the SRI system , phd d:   phd c: but it 's professor f: There might be , cuz that 's a pretty big difference . phd c: I 'm actually f quite sure that the feeding the features into the system and training it up , professor f: What if ? phd c: that that I think that 's this that 's essentially the same as we use with the ce with the P L P fe features . phd c: There we could the another degree of freedom is how do you generate the K L T transform ? phd d:   professor f: These nets are trained with particular normalization and when that gets screwed up it it can really hurt it . phd d: I 'm doing what Eric E Eric coached me through then that part of it , so I 'm pretty confident in that . phd d: the only slight difference is that I use normalization values that , Andreas calculated from the original PLP , phd c: Right . So , I u I do Oh , we actually don't do that normalization for the PLP , do we ? For the st just the straight PLP features ? phd c: No . Well , you might e e phd c: So , there 's there is there is room for bugs that we might not have discovered , phd d: So that 's that 's another Yeah . I I would actually double check with Stephane at this point , phd c: but professor f: cuz he 's probably the one here  , he and Dan are the ones who are at this point most experienced with the tandem phd d:   professor f: thing and there may there may be some little bit here and there that is not not being handled right .  , you can't just , like , print the the values out in ASCII and , you know , look at them , see if they 're professor f: Not unless you had a lot of time phd g: Well professor f: and phd g: eh , and also they 're not  , as I understand it , you you don't have a way to optimize the features for the final word error . phd g: these are just discriminative , but they 're not , optimized for the final phd c: They 're optimized for phone discrimination , not for phd g: Right . So it there 's always this question of whether you might do better with those features if there was a way to train it for the word error metric that you 're actually that you 're actually professor f: That 's right . It 's indirect , so you don't know professor f: So wha w what an and you may not be in this case , come to think of it , because , you 're just taking something that 's trained up elsewhere . So in fact you the the net is trained on , a , Viterbi alignment of the training data that comes from your full system . And so that 's where the feedback comes all around , so that it is actually discriminant . You can prove that it 's it 's a ,  If you believe in the Viterbi assumption that , getting the best path , is almost equivalent to getting the best , total probability , then you actually do improve that by ,  by training up on local local ,  local frames . But , we aren't actually doing that here , because we did we did that for a hybrid system , and now we 're plugging it into another system and so it isn't i i i it wouldn't quite apply here . phd c: Do y phd d: So another huge experiment we could do would be to take the tandem features , do SRI forced alignments using those features , and then re - do the net with those . Another thing is since you 're not using the net for recognition per se but just for this transformation , it 's probably bigger than it needs to be . So , you 're using a l a long a larger phone set than what phd d: Mmm . professor f: The other thing , just to mention that Stephane this was an innovation of Stephane 's , which was a pretty neat one , and might particularly apply here , given all these things we 're mentioning . Even the local ones , given , you know , these potential outer loops which , you know , you can convince yourself turn into the global ones .  , when something about the test set is different enough from the training set that that , the discrimination that you 're learning is is is not a good one . professor f: So , his idea was to take as the input feature vector to the , Gaussian mixture system , a concatenation of the neural net outputs and the regular features . No , but we we when when we when I first started corresponding with Dan about how to go about this , I think that was one of the things that we definitely went there .  , I 'm sure that Stephane wasn't the first to think of it , phd c: Yeah . So that 's that that 's our current best best system in the , in the Aurora thing . phd c: And do you do a KLT transform on the con on the combined feature vector ? professor f: Yeah . phd c: Do you d you do a KLT transform on the combined feature vector ? professor f: Yeah . professor f: Well , actually , I ,  you should check with him , because he tried several different combinations . phd c: Because you end up with this huge feature vector , so that might be a problem , a unless you do some form of dimensionality reduction . So he did one where he put o put e the whole thing into one KLT , and another one , since the the PLP things are already orthogonalized , he left them alone and and just did a KLT on the on the on the net outputs phd c:   phd d: Did he did he try to ? So he always ended up with a feature vector that was twice as long as either one of the ? professor f: No . phd g: We need to close up cuz I need to save the data and , get a call . phd g: or , do them together ? professor f: I I g I think , given that we 're in a hurry for snacks , maybe we should do them together . professor f: Well , it 's it 's it 's not You know , it 's not gonna work out phd g: Adam 's not here , so he 's not here to tell me no . professor f: but we could we could just , see if we find a rhythm , you know , what phd g: Sure . professor f: O 's or zeroes , we wanna agree on that ? phd g: Maybe just whatever people would naturally do ? I don't know . professor f: Why don't we do zer i Anyone have a problem with saying zero ? Is zero OK ? phd g: No <doc-sep>grad g: So I think maybe what 's causing it to crash is I keep starting it and then stopping it to see if it 's working . postdoc b: And it looks like you 've found a way of  mapping the location to the without having people have to give their names each time ? phd a: Sounds like an initialization thing . postdoc b:  it 's like you have the So you know that grad g: No . postdoc b: are you going to write down that I sat here ? grad g: I 'm gonna collect the digit forms and write it down . u And I should say that  , you just pau you just read each line an and then pause briefly . professor e: So  , you see , Don , the unbridled excitement of the work that we have on this project . grad g: And I 'm surprised I sort of I 'm surprised I forgot that , professor e: Yeah , I I 'd I think it 's some grad g: but  I think that would be a good thing to add . professor e: Yeah , well , that 's  , so I I do have a a an agenda suggestion .  , we I think the things that we talk about in this meeting  tend to be a mixture of  procedural  mundane things and  research points and  I was thinking I think it was a meeting a couple of weeks ago that we we spent much of the time talking about the mundane stuff cuz that 's easier to get out of the way and then we sort of drifted into the research and maybe five minutes into that Andreas had to leave . So  I 'm suggesting we turn it around and and  sort of we have anybody has some mundane points that we could send an email later ,  hold them for a bit , and let 's talk about the the research - y kind of things .  , so  the one th one thing I know that we have on that is  we had talked a a couple weeks before   about the  the stuff you were doing with with    l l attempting to locate events , we had a little go around trying to figure out what you meant by " events " but I think , you know , what we had meant by " events " I guess was  points of overlap between speakers . But I th I gather from our discussion a little earlier today that you also mean  interruptions with something else phd d: Yeah . phd d: To professor e: So at any rate you were you 've you 've done some work on that phd d: right . professor e: and  then the other thing would be it might be nice to have a preliminary discussion of some of the other  research  areas that  we 're thinking about doing .  , I think especially since you you haven't been in in these meetings for a little bit , maybe you have some discussion of some of the p the plausible things to look at now that we 're starting to get data ,  and one of the things I know that also came up  is some discussions that that  that  Jane had with Lokendra  about some some some   work about I I I d I I don't want to try to say cuz I I 'll say it wrong , but anyway some some potential collaboration there about about the about the working with these data . Well , I don't know if we if this is sort of like everybody has something to contribute sort of thing , I think there 's just just a couple a couple people primarily  but   , wh why don't Actually I think that that last one I just said we could do fairly quickly so why don't you you start with that . postdoc b: so , he was interested in the question of you know , relating to his to the research he presented recently ,  of inference structures , and  , the need to build in , this this sort of  mechanism for understanding of language . And he gave the example in his talk about how  , e a I 'm remembering it just off the top of my head right now , but it 's something about how  , i " Joe slipped " you know , " John had washed the floor " or something like that . And I don't have it quite right , but that kind of thing , where you have to draw the inference that , OK , there 's this time sequence , but also the the the causal aspects of the  floor and and how it might have been the cause of the fall and that  it was the other person who fell than the one who cleaned it and it These sorts of things . So , I looked through the transcript that we have so far , and  , fou identified a couple different types of things of that type and  , one of them was something like  , during the course of the transcript , w we had gone through the part where everyone said which channel they were on and which device they were on , and  , the question was raised " Well , should we restart the recording at this point ? " And and Dan Ellis said , " Well , we 're just so far ahead of the game right now we really don't need to " . Now , how would you interpret that without a lot of inference ? So , the inferences that are involved are things like , OK , so , how do you interpret " ahead of the game " ? You know . So it 's the it 's i What you what you int what you draw you know , the conclusions that you need to draw are that space is involved in recording , grad g: metaphorically . postdoc b: that  , i that i we have enough space , and he continues , like " we 're so ahead of the game cuz now we have built - in downsampling " . So you have to sort of get the idea that  , " ahead of the game " is sp speaking with respect to space limitations , that  that in fact downsampling is gaining us enough space , and that therefore we can keep the recording we 've done so far . grad g: So , do you think his interest is in using this as a data source , or training material , or what ? professor e: Well , I I should maybe interject to say this started off with a discussion that I had with him , so  we were trying to think of ways that his interests could interact with ours grad g:   professor e: and   I thought that if we were going to project into the future when we had a lot of data ,  and  such things might be useful for that in or before we invested too much  effort into that he should  , with Jane 's help , look into some of the data that we 're already have and see , is there anything to this at all ? grad g:   professor e: Is there any point which you think that , you know , you could gain some advantage and some potential use for it . Cuz it could be that you 'd look through it and you say " well , this is just the wrong task for for him to pursue his " grad g: Wrong , yeah . professor e: And and  I got the impression from your mail that in fact there was enough things like this just in the little sample that that you looked at that that it 's plausible at least .  , he was he he you know We met and he was gonna go and  you know , y look through them more systematically professor e: Yeah . postdoc b: So it 's , you know , not a matter of a professor e: Yeah . professor e: So anyway , that 's that 's e a quite different thing from anything we 've talked about that , you know , might might might come out from some of this .  , he 's talking about just using text postdoc b: That 's his major I mentioned several that w had to do with implications drawn from intonational contours phd c: pretty much , or ? postdoc b: and that wasn't as directly relevant to what he 's doing . postdoc b: inferences that you draw i from professor e: he certainly could use text , but we were in fact looking to see if there is there is there something in common between our interest in meetings and his interest in in in this stuff . grad g: And I imagine that transcripts of speech  text that is speech probably has more of those than sort of prepared writing . postdoc b: Yeah , I don't think I would make that leap , because i in narratives , you know  , if you spell out everything in a narrative , it can be really tedious , grad g:   grad g: Yeah , I 'm just thinking , you know , when you 're when you 're face to face , you have a lot of backchannel and And postdoc b: Oh . And so I think it 's just easier to do that sort of broad inference jumping if it 's face to face .  , so , if I just read that Dan was saying " we 're ahead of the game " in that in that context , postdoc b: Well Yeah . grad g: I might not realize that he was talking about disk space as opposed to anything else . postdoc b: I you know , I I had several that had to do with backchannels and this wasn't one of them . postdoc b: This this one really does  m make you leap from So he said , you know , " we 're ahead of the game , w we have built - in downsampling " . postdoc b: And the inference , i if you had it written down , would be grad g: I guess it would be the same . But there are others that have backchannelling , it 's just he was less interested in those .  , I f f f I 've @ @ d A minute  , several minutes ago , I , like , briefly was was not listening and So who is " he " in this context ? phd c: Yeah , there 's a lot of pronoun phd f: OK . So I was just realizing we 've You guys have been talking about " he "  for at least  , I don't know , three three four minutes without ever mentioning the person 's name again . Actually to make it worse , Morgan uses " you " and " you " phd f: So this is this is this is gonna be a big , big problem if you want to later do  , you know , indexing , or speech understanding of any sort . phd f: You just wrote this ? phd c: and he never said Li - He looked grad g: Well , I think he 's doing that intentionally , phd c: Right , so it 's great . grad g: aren't you ? phd c: So this is really great phd f: Right . phd c: because the thing is , because he 's looking at the per even for addressees in the conversation , phd d: Yeah . phd c: Yeah , if you have the P Z Ms you should be able to pick up what a person is looking at from their voice . postdoc b: Oh , but you 'd have the phd c: Put Morgan always like this postdoc b: You 'd have fainter phd c: and postdoc b: Wouldn't you get fainter reception out here ? professor e: Well , these grad g: Sure , but I think if I 'm talking like this ? Right now I 'm looking at Jane and talking , now I 'm looking at Chuck and talking , I don't think the microphones would pick up that difference . grad g: So if I 'm talking at you , or I 'm talking at you . professor e: I probably been affect No , I th I think I 've been affected by too many conversations where we were talking about lawyers and talking about and concerns about " oh gee is somebody going to say something bad ? " and so on . professor e: And so I so I 'm I 'm tending to stay away from people 's names even though  postdoc b: I am too . phd c: Even though you could pick up later on , just from the acoustics who you were t who you were looking at . grad g: or or is that just too sensitive ? professor e: No no , it isn't sensitive at all . postdoc b: Well professor e: I was just I was just I was overreacting just because we 've been talking about it . professor e: It 's OK to postdoc b: I I came up with something from the Human Subjects people that I wanted to mention .  , it fits into the m area of the mundane , but they did say You know , I asked her very specifically about this clause of how , you know , it says " no individuals will be identified  , " in any publication using the data . " OK , well , individuals being identified , let 's say you have a a snippet that says , " Joe s  thinks such - and - such about about this field , but I think he 's wrongheaded . " Now  , we 're we 're gonna be careful not to have the " wrongheaded " part in there , but but you know , let 's say we say , you know , " Joe used to think so - and - so about this area , in his publication he says that but I think he 's changed his mind . Then the issue of of being able to trace Joe , because we know he 's well - known in this field , and all this and and tie it to the speaker , whose name was just mentioned a moment ago , can be sensitive . professor e: b But I postdoc b: So I think it 's really really kind of adaptive and wise to not mention names any more than we have to because if there 's a slanderous aspect to it , then how much to we wanna be able to have to remove ? professor e: Yeah , well , there 's that . But I  I think also to some extent it 's just educating the Human Subjects people , in a way , because there 's If  You know , there 's court transcripts , there 's there 's transcripts of radio shows  people say people 's names all the time . It 's just that i  you 're right that there 's more poten If we never say anybody 's name , then there 's no chance of of of slandering anybody , phd c: But , then it won't  , if we if we professor e: but grad g: It 's not a meeting . postdoc b: Well , my feeling on it was that it wasn't really important who said it , you know . phd f: Well , if you ha since you have to  go over the transcripts later anyway , you could make it one of the jobs of the people who do that to mark grad g: Well , we t we t we talked about this during the anon anonymization . grad g: If we wanna go through and extract from the audio and the written every time someone says a name . professor e: So it 's postdoc b: So , we need to talk about this later . I sorta knew I was doing it but it was phd f: Well , I still don't know who " he " is . phd c: No , you have to say , you still don't know who " he " is , with that prosody .  , I I would like to move it into into  what Jose  has been doing postdoc b: Yeah . I I remind that me my first objective eh , in the project is to to study difference parameters to to find a a good solution to detect eh , the overlapping zone in eh speech recorded . But eh , tsk , ehhh In that way I I I begin to to study and to analyze the ehn the recorded speech eh the different session to to find and to locate and to mark eh the the different overlapping zone . And eh so eh I was eh I am transcribing the the first session and I I have found eh , eh one thousand acoustic events , eh besides the overlapping zones , eh I I  the eh breaths eh aspiration eh , eh , talk eh , eh , clap , eh I don't know what is the different names eh you use to to name the the n speech phd a: Nonspeech sounds ? phd d: Yeah . grad g: Oh , I don't think we 've been doing it at that level of detail . Eh , I I I do I don't need to to to mmm to m to label the the different acoustic , but I prefer because eh I would like to to study if eh , I I will find eh , eh , a good eh parameters eh to detect overlapping I would like to to to test these parameters eh with the another eh , eh acoustic events , to nnn to eh to find what is the ehm the false eh , the false eh hypothesis eh , nnn , which eh are produced when we use the the ehm this eh parameter eh  pitch eh , eh , difference eh , feature grad g:   phd a: You know I think some of these  that are the nonspeech overlapping events may be difficult even for humans to tell that there 's two there . phd a: if it 's a tapping sound , you wouldn't necessarily or , you know , something like that , it 'd be it might be hard to know that it was two separate events . grad g: Well You weren't talking about just overlaps phd d: Ye grad g: were you ? You were just talking about acoustic events . phd d: I I I I t I t I talk eh about eh acoustic events in general , grad g: Someone starts , someone stops Yeah . professor e: How many overlaps were there  in it ? No no , how many of them were the overlaps of speech , though ? phd d: How many ? Eh almost eh three hundred eh in one session grad g: Oh , God ! phd d: in five eh in forty - five minutes . postdoc b: Does this ? So if you had an overlap involving three people , how many times was that counted ? phd d: Yeah , three people , two people . Eh ,  I would like to consider eh one people with difference noise eh in the background , be professor e: No no , but I think what she 's asking is if at some particular for some particular stretch you had three people talking , instead of two , did you call that one event ? phd d: Oh . This th I I I con I consider I consider eh an acoustic event , the overlapping zone , the period where three speaker or eh are talking together . grad g: Well So let 's postdoc b: For grad g: So let 's say me and Jane are talking at the same time , and then Liz starts talking also over all of us . phd d: Yeah ? grad g: Is there an event right here ? phd d: Eh no . For me is the overlapping zone , because because you you have s you have more one eh , more one voice eh , eh produced in a in in a moment . If professor e: So then , in the region between since there there is some continuous region , in between regions where there is only one person speaking . professor e: Is it Are you calling the beginning or the end of it the event , phd d: Yeah . professor e: or are you calling the entire length of it the event ? phd d: I consider the the , nnn the nnn , nnn eh , the entirety eh , eh , all all the time there were the voice has overlapped .  , I 'm not considering eh the the ehm eh , the fact of eh , eh , for example , what did you say ? Eh at first eh , eh two talkers are  , eh speaking , and eh , eh a third person eh join to to that . For me , it 's eh it 's eh , all overlap zone , with eh several numbers of speakers is eh , eh the same acoustic event . Wi - but  , without any mark between the zone of the overlapping zone with two speakers eh speaking together , and the zone with the three speakers . grad g: Well , but But you could imagine that three people talking has a different spectral characteristic than two . phd d: I I don't know what eh will will happen with the grad g: Yep . grad g: That 's a lot of overlap , phd d: Yeah ? professor e: So again , that 's that 's three three hundred in forty - five minutes that are that are speakers , just speakers . postdoc b: But a a a th professor e: So that 's about eight per minute . postdoc b: But a thousand events in twelve minutes , that 's phd d: Yeah , but Yeah . postdoc b: Well , but a thousand taps in eight minutes is a l in twelve minutes is a lot . phd c: Actually phd d: I I con I consider I consider acoustic events eh , the silent too . grad g: Silence starting or silence ending phd d: Yeah , silent , ground to bec to detect eh because I consider acoustic event all the things are not eh speech .  silent , I I I I don't I I haven't the eh I I would like to to do a stylistic study professor e: Yeah . phd d: and give you eh with the report eh from eh the the study from the the the session one session . When eh eh I w I I was eh look at eh nnn , the difference speech file , for example , eh if eh we use the ehm the mixed file , to to transcribe , the the events and the words , I I saw that eh the eh speech signal , collected by the eh this kind of mike eh of this kind of mike , eh are different from the eh mixed signal eh , we eh collected by headphone . The the the I I I knew that eh the signal eh , eh would be different , but eh the the problem is eh , eh we eh detected eh difference events in the speech file eh collected by by that mike  qui compared with the mixed file . And so if when you transcribe eh only eh using the nnn the mixed file , it 's possible eh if you use the transcription to evaluate a different system , it 's possible you eh in the eh i and you use the eh speech file collected by the eh fet mike , to eh to nnn to do the experiments with the the system , professor e:   phd d: its possible to evaluate eh , eh or to consider eh acoustic events that which you marked eh in the mixed file , but eh they don't appear in the eh speech signal eh collected by the by the mike . The the reason that I generated the mixed file was for IBM to do word level transcription , not speech event transcription . grad g: So I agree that if someone wants to do speech event transcription , that the mixed signals here phd d: Yeah . grad g: if I 'm tapping on the table , you it 's not gonna show up on any of the mikes , but it 's gonna show up rather loudly in the PZM . So and I I I say eh that eh , eh , or this eh only because eh I c I I in my opinion , it 's necessary to eh to eh to put the transcription on the speech file , collected by the objective signal . phd d:  the the the signal collected by the eh , the real mike in the future , in the prototype to to eh correct the initial eh segmentation eh with the eh real speech professor e:   professor e: Yeah , well , just  , just in that that one s ten second , or whatever it was , example that Adam had that that we we passed on to others a few months ago , there was that business where I g I guess it was Adam and Jane were talking at the same time and and  , in the close - talking mikes you couldn't hear the overlap , and in the distant mike you could . So yeah , it 's clear that if you wanna study if you wanna find all the places where there were overlap , it 's probably better to use a distant mike . professor e: On the other hand , there 's other phenomena that are going on at the same time for which it might be useful to look at the close - talking mikes , phd d: Yeah . phd c: But why can't you use the combination of the close - talking mikes , time aligned ? professor e: so it 's grad g: If you use the combination of the close - talking mikes , you would hear Jane interrupting me , but you wouldn't hear the paper rustling . And so if you 're interested in phd c: I  if you 're interested in speakers overlapping other speakers and not the other kinds of nonspeech , that 's not a problem , professor e: Some of it 's masking masked . grad g: Although the other issue is that the mixed close - talking mikes  , I 'm doing weird normalizations and things like that . phd d: I I I think eh I saw the nnn the eh but eh I eh I have eh any results . I I I saw the the speech file collected by eh the fet mike , and eh eh signal eh to eh to noise eh relation is eh low . phd d: And I I found that nnn that eh , ehm , pr probably , grad g: Did Did you phd d: I 'm not sure eh by the moment , but it 's it 's probably that eh a lot of eh , eh for example , in the overlapping zone , on eh in in several eh parts of the files where you you can find eh , eh eh , smooth eh eh speech eh from eh one eh eh talker in the in the meeting , professor e:   phd d: it 's probably in in that eh in in those files you you can not find you can not process because eh it 's confused with with noise . But eh my idea is to to process only nnn , this eh nnn , this kind of s of eh speech . I 'm not sure it 's a good idea , but eh professor e: No i grad g: Well , it 's more realistic but it 'll it 'll be a lot harder . professor e: Well , it 'd be hard , but on the other hand as you point out , if your if i if if your concern is to get  the overlapping people people 's speech , you will you will get that somewhat better . professor e: Are you making any use  you were you were working with th the data that had already been transcribed . professor e: Now  did you make any use of that ? See I was wondering cuz we st we have these ten hours of other stuff that is not yet transcribed . professor e: Do you phd d: The the transcription by Jane , t eh i eh , I I I want to use to to nnn , eh to put i i it 's a reference for me . But eh the transcription eh for example , I I don't I I 'm not interested in the in the in the words , transcription words , eh transcribed eh eh in eh follow in the in the in the speech file , but eh eh Jane eh for example eh put a mark eh at the beginning eh of each eh talker , in the in the meeting ,  eh she she nnn includes information about the zone where eh there are eh there is an overlapping zone . But eh there isn't any any mark , time temporal mark , to to c eh to mmm e - heh , to label the beginning and the end of the of the professor e:   Right , so she is phd d: ta I 'm I I I think eh we need this information to professor e: Right . So the twelve you you it took you twelve hours of course this included maybe some some time where you were learning about what what you wanted to do , but but  , it took you something like twelve hours to mark the forty - five minutes , your grad g: Twelve minutes . professor e: I thought you did forty - five minutes of phd d: No , forty - five minutes is the is the session , all the session . phd d: Tw - twelve hours of work to to segment eh and label eh twelve minutes from a session of part of f professor e: Oh . I I consider all the all the session because eh I I count the nnn the nnn the overlappings marked by by Jane , professor e: Oh , OK . So it 's three hundred in forty - five minutes , but you have you have time  ,  marked twelve minute the the the  overlaps in twelve minutes of it . phd f: So , can I ask can I ask whether you found  , you know , how accurate  Jane 's   labels were as far as grad g: Well , not just the overlaps , everything . phd f: you know , did she miss some overlaps ? or did she n ? phd d: But , by by the moment , I I don't compare , my my temporal mark with eh Jane , but eh I I want to do it . Because eh eh i per perhaps I have eh errors in the in the marks , I and if I I compare with eh Jane , it 's probably I I I can correct and and and to get eh eh a more accurately eh eh transcription in the file . phd c: Well , not only a word level , but actually phd d: Well phd f: I 'm expect I 'm not expecting phd d: No , it 's phd c: you didn't need to show the exact point of interruption , you just were showing at the level of the phrase or the level of the speech spurt , or grad g: Right . postdoc b: And and sometimes , you know , it was like you could have an overlap where someone said something in the middle , phd d: Yeah . postdoc b: but , yeah , w it just wasn't important for our purposes to have it that i disrupt that unit in order to have , you know , a the words in the order in which they were spoken , it would have it would have been hard with the interface that we have . postdoc b: Now , my a Adam 's working on a of course , on a revised overlapping interface , phd d:  - huh . phd d: I I I think It 's it 's a good eh work , postdoc b: but phd d: but eh I think we need eh eh more information . phd f: I expect you to find more overlaps than than Jane grad g: Always need more for postdoc b: Yeah . I I have to go to phd f: because you 're looking at it at a much more detailed level . professor e: I have grad g: But if it takes sixty to one professor e: Well , I but I have a suggestion about that .  , obviously this is very , very time - consuming , and you 're finding lots of things which I 'm sure are gonna be very interesting , but in the interests of making progress ,  might I s how how would it affect your time if you only marked speaker overlaps ? phd d: Only . professor e: but only mark speaker Do you think that would speed it up quite a bit ? phd d: OK . I I I I w I I wanted to professor e: Do y do you think that would speed it up ?  , speed up your your your marking ? phd d: nnn , I don't understand very . professor e: Now , my suggestion was for the other thirty - three phd d: On - only to mark only to mark overlapping zone , but professor e: Yeah , and my question is , if you did that , if you followed my suggestion , would it take much less time ? phd d: Oh , yeah . professor e: Then I think it 's a good idea , because it phd d: Sure , because I I need a lot of time to to put the label or to do that . professor e: There 's there 's  continual noise  from fans and so forth , and there is  more impulsive noise from  taps and so forth phd d: Yeah . We know that all that 's there and it 's a g worthwhile thing to study , but obviously it takes a lot of time to mark all of these things . professor e: Whereas th i I would think that  you we can study more or less as a distinct phenomenon the overlapping of people talking . Then you can get the Cuz you need If it 's three hundred  i i it sounds like you probably only have fifty or sixty or seventy events right now that are really phd d: Yeah . professor e: And and you need to have a lot more than that to have any kind of  even visual sense of of what 's going on , much less any kind of reasonable statistics . phd c: Now , why do you need to mark speaker overlap by hand if you can infer it from the relative energy in the grad g: Well , that 's That 's what I was gonna bring up . phd c: you shouldn't need to do this p completely by hand , professor e: OK , yeah . professor e: So the idea was that what he was going to be doing was experimenting with different measures such as the increase in energy , such as the energy in the LPC residuals , such as  there 's a bunch of things  , increased energy is - is sort of an obvious one . professor e: and  , it 's not obvious , you could you could do the dumbest thing and get get it ninety percent of the time . But when you start going past that and trying to do better , it 's not obvious what combination of features is gonna give you the you know , the right detector . And so the i the idea of the manual marking was to say " OK this , i you know , it 's it 's really here " . phd a: But I think Liz is saying why not get it out of the transcripts ? phd c: What  is get it from the close - talking mikes . phd c: A or ge get a first pass from those , professor e: We t we t w we t we talked about that . phd c: and then go through sort of It 'd be a lot faster probably to phd f: And you can grad g: Yeah , that 's his ,  professor e: We we we talked about that . s But so it 's a bootstrapping thing and the thing is , phd c: Yeah , I just professor e: the idea was , i we i i we thought it would be useful for him to look at the data anyway , and and then whatever he could mark would be helpful , phd c: Right . You know , do you bootstrap from a simple measurement which is right most of the time and then you g do better , or do you bootstrap from some human being looking at it and then then do your simple measurements ,  from the close - talking mike . phd c: Well , that 's what I wonder , because  or how bad it is , professor e: Well phd c: be  , because that would be interesting grad g: I 'm working on a program to do that , and phd c: especially because the bottleneck is the transcription . phd c: so  it seems like one kind of project that 's not perfect , but  , that you can get the training data for pretty quickly is , you know , if you infer form the close - talking mikes where the on - off points are of speech , professor e: Right , we discussed that . phd c: you know , how can we detect that from a far - field ? grad g: And postdoc b: Oh . grad g: I 've I 've written a program to do that , phd c: OK , I 'm sorry I missed the grad g: and it ,  professor e: It 's OK . And , it seems to work , I 've I 'm sort of fiddling with the parameters , to get it to actually generate something , and I haven't I don't what I 'm working on was working on was getting it to a form where we can import it into the user interface that we have , into Transcriber . I 've worked on it for about half a day , grad h: I have to go . grad g: so give me another half day and I we 'll have something we can play with . professor e: See , this is where we really need the Meeting Recorder query stuff to be working , because we 've had these meetings and we 've had this discussion about this , and I 'm sort of remembering a little bit about what we decided , phd c: Right . phd c: It professor e: So , I think it was partly that , you know , give somebody a chance to actually look at the data and see what these are like , partly that we have e some ground truth to compare against , you know , when when he he gets his thing going , grad g: But professor e: and phd c: Well , it 's definitely good to have somebody look at it . I was just thinking as a way to speed up you know , the amount of postdoc b:   postdoc b: Was that  there m there was this already a script I believe  that Dan had written , that  handle bleedthrough ,  cuz you have this this close you have contamination from other people who speak loudly . So I I haven't tried that , but that If It it might be something it might be a good way of cleaning it up a little . postdoc b: So , some thought of maybe having Yeah , having that be a preprocessor and then run it through yours . professor e: But but that 's a refinement postdoc b: That 's what we were discussing . professor e: and I think we wanna see try the simple thing first , cuz you add this complex thing up  afterwards that does something good y y yo you sort of wanna see what the simple thing does first . professor e: But  , having having somebody have some experience , again , with with  with marking it from a human standpoint , we 're  , I don't expect Jose to to do it for  f fifty hours of of speech , but  we if  if he could speed up what he was doing by just getting the speaker overlaps so that we had it , say , for forty - five minutes , then at least we 'd have three hundred examples of it . professor e: And when when  Adam was doing his automatic thing he could then compare to that and see what it was different . phd a: You know , I did I did  something almost identical to this at one of my previous jobs , and it works pretty well .  , i almost exactly what you described , an energy detector with a median filter , you look for runs . phd a: And so I think doing that to generate these possibilities and then going through and saying yes or no on them would be a quick way to to do it . grad g: Yeah , do you have a patent on it ? phd a: It was when I was working for the government . postdoc b: Well , is this something that we could just co - opt , or is it ? phd a: Nah . I think I think it 's phd a: Yeah , he 's it it doesn't take a long time . I just thought if it was tried and true , then and he 's gone through additional levels of of development . Although if you if you have some parameters like what 's a good window size for the median filter phd a: Oh ! I have to remember . phd f: They grad g: I was doing pretty short , you know , tenth of a second , sorts of numbers . professor e: I don't know , it if if we want to  So , maybe we should move on to other other things in limited time . postdoc b: Can I ask one question about his statistics ? So so in the tw twelve minutes , if we took three hundred and divided it by four , which is about the length of twelve minutes , i  , I 'd expect like there should be seventy - five overlaps . postdoc b: Did you find  more than seventy - five overlaps in that period , or ? phd d: More than ? postdoc b: More than How many overlaps in your twelve minutes ? phd d: How many ? Eh , not @ @ I Onl - only I I transcribe eh only twelve minutes from the professor e: Yeah . phd d: but eh I I don't co eh I don't count eh the the overlap . phd d: I consider I I The the nnn The the three hundred is eh considered only you your transcription . grad g: I b I bet they 're more , because the beginning of the meeting had a lot more overlaps than than sort of the middle . grad g: Because i we 're we 're dealing with the  , in the early meetings , we 're recording while we 're saying who 's talking on what microphone , and things like that , phd d: Yeah . professor e: So so I was gonna ask , I guess about any any other things that that that either of you wanted to talk about , especially since Andreas is leaving in five minutes , that that you wanna go with . phd c: Can I just ask about the data , like very straightforward question is where we are on the amount of data and the amount of transcribed data , just cuz I 'm I wanted to get a feel for that to sort of be able to know what what can be done first and like how many meetings are we recording professor e: Right so there 's this this There 's this forty - five minute piece that Jane transcribed . phd c: and professor e: That piece was then  sent to IBM so they could transcribe so we have some comparison point . Then there 's s a larger piece that 's been recorded and  put on CD - ROM and sent  to IBM . phd c: How many meetings is that ? Like how many grad g: What 's that ? professor e: That was about ten hours , and there was about phd c: t ten It 's like ten meetings or something ?  - huh . And then then we phd a: Ten meetings that have been sent to IBM ? phd c: And professor e: Yeah . grad g: Well , I haven't sent them yet because I was having this problem with the missing files . phd a: H how many total have we recorded now , altogether ? professor e: We 're saying about twelve hours . And we 're recording only this meeting , like continuously we 're only recording this one now ? or ? professor e: No . No , so the the that 's the that 's the biggest one  , chunk so far , grad g: Nope . phd c: Do they meet every week , professor e: And then there phd c: or every professor e: they do . w w And we talked to them about recording some more and we 're going to , we 've started having a morning meeting , today  i starting a w a week or two ago , on the  front - end issues , and we 're recording those ,  there 's a network services and applications group here who 's agreed to have their meetings recorded , phd c: Great . So actually , we 're gonna h start having a a pretty significant chunk and so , you know , Adam 's sort of struggling with trying to get things to be less buggy , and come up quicker when they do crash and stuff things like that , now that  the things are starting to happen . So right now , yeah , I th I 'd say the data is predominantly meeting meetings , but there are scattered other meetings in it and that that amount is gonna grow  so that the meeting meetings will probably ultimately i if we 're if we collect fifty or sixty hours , the meeting meetings it will probably be , you know , twenty or thirty percent of it , not not not eighty or ninety . phd c: So there 's probably there 's three to four a week , grad g: That 's what we 're aiming for . professor e: Yeah and th the the other thing is I 'm not pos I 'm sort of thinking as we 've been through this a few times , that I really don't know maybe you wanna do it once for the novelty , but I don't know if in general we wanna have meetings that we record from outside this group do the digits .  in fact , the morning group is really motivated cuz they 're working on connected digits , so it 's grad g: Actually that 's something I wanted to ask , is I have a bunch of scripts to help with the transcription of the digits . grad g: We don't have to hand - transcribe the digits because we 're reading them and I have those . grad g: And so I have some scripts that let you very quickly extract the sections of each utterance .  , if I did that , is someone gonna be working on it ? professor e: yeah , I I think definitely s so Absolutely . grad g: is it something of interest ? professor e: Yeah , whoever we have working on the acoustics for the Meeting Recorder are gonna start with that .  , I I 'm I 'm interested in it , I just don't have time to do it now . phd f: I was these meetings I 'm sure someone thought of this , but these this  reading of the numbers would be extremely helpful to do  adaptation . phd c: Actually I have o grad g: I I would really like someone to do adaptation . grad g: So if we got someone interested in that , I think it would be great for Meeting Recorder . professor e: Well  , one of the things I wanted to do , that I I talked to to Don about , is one of the possible things he could do or m also , we could have someone else do it , is to do block echo cancellation , grad g: Since it 's the same people over and over . professor e: to try to get rid of some of the effects of the the the far - field effects .  ,  we have the party line has been that echo cancellation is not the right way to handle the situation phd f:   professor e: because people move around , and  , if if it 's if it 's  not a simple echo , like a cross - talk kind of echo , but it 's actually room acoustics , it 's it 's it 's you can't really do inversion , phd f:   professor e: and even echo cancellation is going to  be something It may you Someone may be moving enough that you are not able to adapt quickly and so the tack that we 've taken is more " lets come up with feature approaches and multi - stream approaches and so forth , that will be robust to it for the recognizer and not try to create a clean signal " . But it occurred to me a few months ago that  party lines are always , you know , sort of dangerous . And so we haven't had anybody try to do a good serious job on echo cancellation and we should know how well that can do . So that 's something I 'd like somebody to do at some point , just take these digits , take the far - field mike signal , and the close  mike signal , and apply really good echo cancellation .  , there was a have been some nice talks recently by by Lucent on on their b phd f:  . professor e: the block echo cancellation particularly appealed to me ,  you know , trying and change it sample by sample , but you have some reasonable sized blocks . And  , you know , th phd a: W what is the  the artifact you try to you 're trying to get rid of when you do that ? phd f: Ciao . professor e:  so it 's it you have a a direct   , what 's the difference in If you were trying to construct a linear filter , that would  phd f: I 'm signing off . that would subtract off the   parts of the signal that were the aspects of the signal that were different between the close - talk and the distant . You know , so so   I guess in most echo cancellation Yeah , so you Given that  Yeah , so you 're trying to So you 'd There 's a a distance between the close and the distant mikes so there 's a time delay there , and after the time delay , there 's these various reflections . And if you figure out well what 's the there 's a a least squares algorithm that adjusts itself adjusts the weight so that you try to subtract essentially to subtract off  different  different reflections . Right ? So let 's take the simple case where you just had you had some  some delay in a satellite connection or something and then there 's a there 's an echo . phd a: So that would mean like if you were listening to the data that was recorded on one of those .  , just the raw data , you would you might hear kind of an echo ? And and then this noise cancellation would get professor e: Well , I 'm I 'm I 'm saying That 's a simplified version of what 's really happening . What 's really happening is Well , when I 'm talking to you right now , you 're getting the direct sound from my speech , but you 're also getting , the indirect sound that 's bounced around the room a number of times . OK ? So now , if you  try to r you To completely remove the effect of that is sort of impractical for a number of technical reasons , but I but not to try to completely remove it , that is , invert the the room response , but just to try to   eliminate some of the the effect of some of the echos .  , a number of people have done this so that , say , if you 're talking to a speakerphone ,  it makes it more like it would be , if you were talking right up to it . You say I I I want to use this  this item but I want to subtract off various kinds of echos . So you construct a filter , and you have this this filtered version  of the speech  gets   gets subtracted off from the original speech . phd a: Kind of a clean up thing , that professor e: It 's a clean up thing . professor e: So , echo cancelling is is , you know , commonly done in telephony , and and and it 's sort of the obvious thing to do in this situation if you if , you know , you 're gonna be talking some distance from a mike . phd a: When  , I would have meetings with the folks in Cambridge when I was at BBN over the phone , they had a  some kind of a special speaker phone and when they would first connect me , it would come on and we 'd hear all this noise . And then it was  And then it would come on and it was very clear , professor e: Yeah . So it 's taking samples , it 's doing adaptation , it 's adjusting weights , and then it 's getting the sum . So  ,  anyway that 's that 's kind of a reasonable thing that I 'd like to have somebody try somebody look And and the digits would be a reasonable thing to do that with . I think that 'd be enough data plenty of data to do that with , and i for that sort of task you wouldn't care whether it was  large vocabulary speech or anything .  postdoc b: Is Brian Kingsbury 's work related to that , or is it a different type of reverberation ? professor e: Brian 's Kingsbury 's work is an example of what we did f f from the opposite dogma . Right ? Which is what I was calling the " party line " , which is that  doing that sort of thing is not really what we want . We want something more flexible ,  i i where people might change their position , and there might be , you know There 's also  oh yeah , noise . It 's if you have a clean situation but you just have some delays , Then we 'll figure out the right the right set of weights for your taps for your filter in order to produce the effect of those those echos . But  if there 's noise , then the very signal that it 's looking at is corrupted so that it 's decision about what the right you know , right right  delays are is ,  is right delayed signal is is is  is incorrect . And so , in a noisy situation , also in a in a situation that 's very reverberant with long reverberation times and really long delays , it 's it 's sort of typically impractical . So for those kind of reasons , and also a a c a complete inversion , if you actually I mentioned that it 's kind of hard to really do the inversion of the room acoustics .  , that 's difficult because  often times the the  the system transfer function is such that when it 's inverted you get something that 's unstable , and so , if you you do your estimate of what the system is , and then you try to invert it , you get a filter that actually  , you know , rings , and and  goes to infinity . So it 's so there 's there 's there 's that sort of technical reason , and the fact that things move , and there 's air currents  there 's all sorts of all sorts of reasons why it 's not really practical . So for all those kinds of reasons ,  we we we sort of  , concluded we didn't want to in do inversion , and we 're even pretty skeptical of echo cancellation , which isn't really inversion , and  we decided to do this approach of taking  , just picking  features , which were  will give you more something that was more stable , in the presence of , or absence of , room reverberation , and that 's what Brian was trying to do . I guess you you actually already said this thing about the  about the consent forms , which was that we now don't have to So this was the human subjects folks who said this , or that that ? postdoc b: The a apparently  , we 're gonna do a revised form , of course .  but once a person has signed it once , then that 's valid for a certain number of meetings . So I think from a s practical standpoint , maybe we could have them do it once every ten meetings , or something . It won't be that many people who do it that often , but  just , you know , so long as they don't forget that they 've done it , I guess .  , back on the data thing , so there 's this sort of one hour , ten hour , a hundred hour sort of thing that that we have . We have we have an hour  that that is transcribed , we have we have twelve hours that 's recorded but not transcribed , and at the rate we 're going ,  by the end of the semester we 'll have , I don't know , forty or fifty or something , if we if this really  Well , do we have that much ? phd c: Not really . professor e: Let 's see , we have phd c: So that 's what You know , that professor e:  eight weeks ,  is phd c: So that 's not a lot of hours . professor e: Eight weeks times three hours is twenty - four , so that 's Yeah , so like thirty thirty hours ? phd a: Three Three hours .  I was starting to think of some projects where you would use well , similar to what we talked about with  energy detection on the close - talking mikes . There are a number of interesting questions that you can ask about how interactions happen in a meeting , that don't require any transcription . So what are the patterns , the energy patterns over the meeting ? And I 'm really interested in this but we don't have a whole lot of data . So I was thinking , you know , we 've got the room set up and you can always think of , also for political reasons , if ICSI collected you know , two hundred hours , that looks different than forty hours , even if we don't transcribe it ourselves , professor e: But I don't think we 're gonna stop at the end of this semester . phd c: so professor e: Right ? So , I th I think that if we are able to keep that up for a few months , we are gonna have more like a hundred hours . phd c: is there Are there any other meetings here that we can record , especially meetings that have some kind of conflict in them or some kind of deci  , that are less well I don't  , that have some more emotional aspects to them , or strong grad g: We had some good ones earlier . phd c: There 's laughter ,  I 'm talking more about strong differences of opinion meetings , maybe with manager types , or grad g: I think it 's hard to record those . phd c: To be allowed to record them ? postdoc b: It 's also likely that people will cancel out afterwards . professor e: Yeah , people will get postdoc b: But I but I wanted to raise the KPFA idea . So I I  , I I 'd mentioned to Adam , and that was another thing I was gonna talk  , mention to them before that  there 's  It it oc it occurred to me that we might be able to get some additional data by talking to  acquaintances in local broadcast media . Because , you know , we had talked before about the problem about using found data , that that  it 's just set up however they have it set up and we don't have any say about it and it 's typically one microphone , in a ,  or and and so it doesn't really give us the the the  characteristics we want . But  , it did occur to me that we could go to friends in broadcast media and say " hey you have this panel show , or this you know , this discussion show , and  can you record multi - channel ? " And  they may be willing to record it  with phd c: With lapel mikes or something ? professor e: Well , they probably already use lapel , but they might be able to have it it wouldn't be that weird for them to have another mike that was somewhat distant . professor e: It wouldn't be exactly this setup , but it would be that sort of thing , and what we were gonna get from UW , you know , assuming they they they start recording , isn't als also is not going to be this exact setup . the other thing that occurred to me after we had that discussion , in fact , is that it 's even possible , since of course , many radio shows are not live ,  that we could invite them to have like some of their record some of their shows here . postdoc b: Wow ! phd c: Well Or The thing is , they 're not as averse to wearing one of these head - mount  , they 're on the radio , grad g: Right , as we are . phd c: Th - that 's an a side of style a style that we 're not collecting here , so it 'd be great . professor e: And and the  , the other side to it was the what which is where we were coming from I 'll I 'll talk to you more about it later is that is that there 's there 's  the radio stations and television stations already have stuff worked out presumably ,  related to , you know , legal issues and and permissions and all that . So I think it 's something we should look into , you know , we 'll collect what we collect here hopefully they will collect more at UW also and  and maybe we have this other source . But yeah I think that it 's not unreasonable to aim at getting , you know , significantly in excess of a hundred hours . The thing was , I was hoping that we could @ @ in the under this controlled situation we could at least collect , you know , thirty to fifty hours . And at the rate we 're going we 'll get pretty close to that I think this semester . Yeah I was mostly trying to think , " OK , if you start a project , within say a month , you know , how much data do you have to work with . And you you wanna s you wanna sort of fr freeze your your data for awhile so  right now and we don't have the transcripts back yet from IBM right ? Do Oh , do we now ? professor e: Well , we don't even have it for this f you know , forty - five minutes , that was phd c: So  , not complaining , I was just trying to think , you know , what kinds of projects can you do now versus  six months from now professor e: Yeah . So I was thinking right now it 's sort of this exploratory stuff where you you look at the data , you use some primitive measures and get a feeling for what the scatter plots look like , grad g: Right . professor e: and and and  and meanwhile we collect , and it 's more like yeah , three months from now , or six months from now you can you can do a lot of other things . phd c: Cuz I 'm not actually sure , just logistically that I can spend you know , I don't wanna charge the time that I have on the project too early , before there 's enough data to make good use of the time . phd c:  anyway , I shouldn't say too much , but  if someone came that was great and wanted to do some real work and they have to end by the end of this school year in the spring , how much data will I have to work with , with that person . And so it 's professor e: i Yeah , so I would think , exploratory things now .  , three months from now  ,  the transcriptions I think are a bit of an unknown cuz we haven't gotten those back yet as far as the timing , but I think as far as the collection , it doesn't seem to me l like , unreasonable to say that  in January , you know , ro roughly  which is roughly three months from now , we should have at least something like , you know , twenty - five , thirty hours . phd c: And we just don't know about the transcription part of that , professor e: So that 's postdoc b: Yeah , we need to I think that there 's a possibility that the transcript will need to be adjusted afterwards , phd c: so .  , it postdoc b: and  es especially since these people won't be  used to dealing with multi - channel  transcriptions . postdoc b: So I think that we 'll need to adjust some And also if we wanna add things like  , well , more refined coding of overlaps , then definitely I think we should count on having an extra pass through . There 'd be no reason why a person couldn't get together several  , you know , friends , and come and argue about a topic if they wanted to , right ? professor e: If they really have something they wanna talk about as opposed to something @ @  , what we 're trying to stay away from was artificial constructions , but I think if it 's a real Why not ? Yeah . postdoc b: You could do this , phd c: Well yeah , postdoc b: you know . phd c: or just if you 're if you ha If there are meetings here that happen that we can record even if we don't  have them do the digits , or maybe have them do a shorter digit thing like if it was , you know , one string of digits , or something , they 'd probably be willing to do . grad g: We don't have to do the digits at all if we don't want to . phd c: Then , having the data is very valuable , cuz I think it 's  politically better for us to say we have this many hours of audio data , especially with the ITR , if we put in a proposal on it .  , whether it 's transcribed or not  , is another issue , but there 's there are research questions you can answer without the transcriptions , or at least that you can start to answer . phd a: Would it help at all  , we 're already talking about sort of two levels of detail in meetings . One is   without doing the digits Or , I guess the full - blown one is where you do the digits , and everything , and then talk about doing it without digits , what if we had another level , just to collect data , which is without the headsets and we just did the table - mounted stuff . I 'm really scared grad g: It seems like it 's a big part of this corpus is to have the close - talking mikes . phd c:  or at least , like , me personally ? I would I couldn't use that data . phd c: So it 's a great idea , professor e: Yeah , I I b By the by the way , I don't think the transcriptions are actually , in the long run , such a big bottleneck . phd c: and if it were true than I would just do that , but it 's not that bad like the room is not the bottleneck , and we have enough time in the room , it 's getting the people to come in and put on the and get the setup going . Right ? And and  d Do you have any idea when when  the you 'll be able to send  the ten hours to them ? grad g: Well , I 've been burning two C Ds a day , which is about all I can do with the time I have . So early next week we send it to them , and then then we check with them to see if they 've got it and we we start , you know asking about the timing for it . professor e: So I think once they get it sorted out about how they 're gonna do it , which I think they 're pretty well along on , cuz they were able to read the files and so on . professor e: Right ? grad g: Yeah , but professor e: Well grad g: Yeah , who knows where they are . professor e: Yeah , but You know , so they they they have you know , they 're volunteering their time and they have a lot of other things to do , phd c: What if grad g: Yeah , you we can't complain . professor e: right ? But they But at any rate , they 'll I I think once they get that sorted out , they 're they 're making cassettes there , then they 're handing it to someone who they who 's who is doing it , and  I think it 's not going to be I don't think it 's going to be that much more of a deal for them to do thirty hours then to do one hour , I think . phd c: Really ? So it 's the amount of professor e: It 's it 's just getting it going . phd c: I don't know , if there 's any way without too much more overhead , even if we don't ship it right away to IBM even if we just collect it here for awhile , to record you know , two or three more meeting a week , just to have the data , even if they 're  not doing the digits , but they do wear the headphones ? professor e: But the lunch meetings are pretty much one person getting up and phd c: No , I meant , sorry , the meetings where people eat their lunch downstairs , maybe they don't wanna be recorded , but grad g: Oh , and we 're just chatting ? phd c: Just the ch the chatting . phd c: I actually I actually think that 's useful data ,  the chatting , grad g: Yeah , the problem with that is I would I think I would feel a little constrained to You know ?  , some of the meetings phd c: but OK . phd c: Alright , so I 'll just throw it out there , if anyone knows of one more m or two more wee meetings per week that happen at ICSI ,  that we could record , I think it would be worth it . Well , we should also check with Mari again , because they because they were really intending , you know , maybe just didn't happen , but they were really intending to be duplicating this in some level . And there 's a lot of different meetings at UW   really m a lot more than we have here right cuz we 're not right on campus , grad g: Right . phd a: Is the  , notion of recording any of Chuck 's meetings dead in the water , or is that still a possibility ? professor e: they seem to have some problems with it .  , but , again , Jerry is Jerry 's open So  , we have two speech meetings , one  network meeting ,  Jerry was open to it but I I s One of the things that I think is a little a little bit of a limitation , there is a think when the people are not involved  in our work , we probably can't do it every week . You know ? I I I I think that that people are gonna feel  are gonna feel a little bit constrained . Now , it might get a little better if we don't have them do the digits all the time . And the then so then they can just really sort of try to put the mikes on and then just charge in and grad g: Yep . phd c: What if we give people you know , we cater a lunch in exchange for them having their meeting here or something ? postdoc b: Well , you know , I I do think eating while you 're doing a meeting is going to be increasing the noise . postdoc b: But I had another question , which is  , you know , in principle , w  , I know that you don't want artificial topics , phd c: Alright , alright , alright . postdoc b: but  it does seem to me that we might be able to get subjects from campus to come down and do something that wouldn't be too artificial . postdoc b: and i you know , people who are Because , you know , there 's also this constraint . We d it 's like , you know , the the  goldibears goldi goldilocks , it 's like you don't want meetings that are too large , but you don't want meetings that are too small . And  a and it just seems like maybe we could exploit the subj human subject p p pool , in the positive sense of the word . phd a: Well , even  , coming down from campus is sort of a big thing , but what about postdoc b: We could pay subjects . phd a: or what about people in the in the building ? phd c: Yeah , I was thinking , there 's all these other peo phd a: there 's the State of California downstairs , and phd c: Yeah .  grad g: I just really doubt that  any of the State of California meetings would be recordable and then releasable to the general public . grad g: So I  I talked with some people at the Haas Business School who are i who are interested in speech recognition phd c: Alright , well . grad g: and , they sort of hummed and hawed and said " well maybe we could have meetings down here " , but then I got email from them that said " no , we decided we 're not really interested and we don't wanna come down and hold meetings . phd a: What about Joachim , maybe he can professor e: But but we c But I think , you know , we get some scattered things from this and that . professor e:  i I have better contacts in radio than in television , but phd a: You could get a lot of lively discussions from those radio ones . phd c: Well , and they 're already they 're these things are already recorded , grad g: Yep . phd c: we don't have to ask them to even and I 'm not sure wh how they record it , but they must record from individual professor e: n Well No , I 'm not talking about ones that are already recorded . I 'm talking about new ones phd c: Why why not ? professor e: because because because we would be asking them to do something different . I know for instance Mark Liberman was interested  in in LDC getting data , and professor e: Right , that 's the found data idea . professor e: But what I 'm saying is  if I talk to people that I know who do these th who produce these things we could ask them if they could record an extra channel , let 's say , of a distant mike . So , since I 'm interested in the distant mike stuff , I wanna make sure that there is at least that somewhere phd c: Right .   professor e: and  But if we ask them to do that they might be intrigued enough by the idea that they  might be e e willing to the I might be able to talk them into it . We 're getting towards the end of our disk space , so we should think about trying to wrap up here . Well I don't why don't we why d u why don't we   turn them turn grad g: OK , leave leave them on for a moment until I turn this off , cuz that 's when it crashed last time <doc-sep>And actually if everyone could cross out the R - nine next to " Session " , and write MR eleven . grad c: And let 's remember also to make sure that one 's gets marked as unread , unused . grad c: There 's lots of clicking I 'm sure as I 'm trying to get this to work correctly . Any agenda items today ? grad c: I wanna talk a little bit about getting how we 're gonna to get people to edit bleeps , parts of the meeting that they don't want to include . What I 've done so far , and I wanna get some opinions on , how to how to finish it up . postdoc f: I wanna ask about  , some aud audio monitoring on some of the  well some of the equipment . postdoc f: Ba - based on some of the tran  i In listening to some of these meetings that have already been recorded there are sometimes big spikes on particular things , and in pact in fact this one I 'm talking on is one of of the ones that showed up in one of the meetings , grad c: Oh really . postdoc f: so I phd b: " Spikes " , you mean like  , instantaneous click type spikes , or ? postdoc f:   grad c: It could be touching and fiddling , and the other thing is that it could the fact that it 's on a wired mike is suspicious . Well maybe Then we don't really have to talk about that as an phd b: You could try an experiment and say " OK , I 'm about to test for spikes " , postdoc f: I I take that off the agenda . phd b: and then wiggle the thing there , and then go and when they go to transcribe it , it could , ask them to come and get you . phd b: " Come get me when you transcribe this and see if there 's spikes . phd b: No I 'm just  professor e: were this a professional audio recording , what we would do what you would do is in testing it is , you would actually do all this wiggling and make sure that that that things are not giving that kind of performance . I guess I would like to have a discussion about you know where we are on  , recording , transcription you know , basically you know where we are on the corpus . professor e: And then  , the other thing which I would like to talk about which is a real meta - quest , I think , deal is , agendas . Andreas brought up the fact that he would kinda like to know , if possible , what we were gonna be talking about because he 's sort of peripherally involved to this point , and if there 's gonna be a topic about discussion about something that he  strongly cares about then he would come and And I think part of part of his motivation with this is that he 's trying to help us out , in the because of  the fact that the meetings are are tending to become reasonably large now on days when everybody shows up and so , he figures he could help that out by not showing grad c: Mmm . by not showing up if it 's a meeting that he 's he 's So ,  in order I 'd I think that this is a wish on his part . It 's actually gonna be hard because it seems like a lot of times  things come up that are unanticipated and and grad c: Right . professor e: But  , we could try anyway , do another try at coming up with the agenda  , at some point before the meeting , say the day before . grad c: Well maybe it would be a good idea for one of us to like on Wednesday , or Tuesday send out a reminder for people to send in agenda items . phd b: That 'll be I think that 'll help grad c: I 'll put that on my spare brain or it will not get done . professor e: Yeah , I have to tell you for the  for the admin meeting that we have , Lila does that  every time before an admin meeting . phd b: I would I would also guess that as we get more into processing the data and things like that there 'll be more things of interest to him . The other topic I was thinking of was the sta status on microphones and channels , and all that . grad c: Yeah , actually I I was going to say we need to talk about that too . First of all , if the other headsets are a lot more comfortable , we should probably just go ahead and get them . So we 'll have to evaluate that when they come in , phd a:  grad c: and get people 's opinions on on what they think of them . professor e: So how many channels do you get to have in a wireless setup ? grad c: well , I 'm pretty sure that you can daisy - chain them together so what we would do is replace the wired mikes with wireless . So we currently have one base station with six wireless mike , possibility of six wireless receivers , and apparently you can chain those together . And so we could replace our wired mikes with wireless if we bought another base station and more wireless mikes . grad c: And So , you know it 's still , it 's fifteen minus six . professor e: So let 's see we grad c: Right ? So we could have up to nine . grad c: And it 's  , about nine hundred dollars for the base station , and then eight hundred per channel . So yeah so the only Beyond the mike the cost of the mikes the only thing is the base station that 's nine hundred dollars . OK , so I 'll look into how you daisy - chain them and and then just go ahead and order them . grad c: No , we 're just replacing the wired the two wired that are still working , phd b: OK . grad c: along with a couple of the wired that aren't working , one of the wired that 's not working , with a wireless . phd b: Three wireds work , professor e: Basically we found phd b: right ? grad c: I I guess three wireds work , yeah . professor e: And so  , we we seem to have  , a reliable way of getting the data in , which is through the ra Sony radio mikes , as long as we 're conscious about the batteries . grad c: Everyone 's battery OK ? phd b: I checked them this morning , they should be . But the quality seems really good and  I heard from UW that they 're they 're  very close to getting their ,  setup purchased . They 're they 're they 're buying something that you can just sort of buy off the shelf . grad c: Well we should talk to them about it because I know that SRI is also in the process of looking at stuff , and so , you know , what we should try to keep everyone on the same page with that . grad c: They got sa apparent Well , Maybe this needs to be bleeped out ? I have no clue . But anyway there 's there 's there 's  ,  other activities that are going on there and and  and NIST and UW . But but yeah I thin I think that at least the message we can tell other people is that our experience is is quite positive with the Sony , grad c: Right . Now the one thing that you have said that actually concerns me a little is you 're talking about changing the headsets meaning changing the connector , which means some hand - soldering or something , right ? grad c: no , we 're having the them do it . professor e: No ? grad c: So it 's so hand - soldering it , but I 'm not doing it . professor e: OK , so that 's being done professionally and grad c: I  professor e: Yeah . professor e: Well , it could if they do a lot of it , it 's grad c:  i it 's just their repair shop . grad c: And ,  we 'll see , tomorrow , you know , what it looks like . So , Dave isn't here but he was going to start working on some things with the digits . I guess Was the decision last time was that the the  transcribers were going to be doing stuff with the digits as well ? Has that started , or is that ? postdoc f:   grad c: Right , so , the decision was that Jane did not want the transcribers to be doing any of the paperwork . And  , then I have a bunch of scripts that we 'll read those and let the  transcribers use different tools . So he has a nice set up that they it w it will be efficient for them to do that . professor e: So anyway grad c: So , you know , just  , a matter of a few days I suspect . professor e: So anyway I think we we have at least one  , user for the digits once they get done , which will be Dave . grad c: So if he wanted to , you know , just have a few to start with , he could . You know , and I also have a bunch of scripts that will , like , generate P - files and run recognition on them also .  , is Dave I don't know if Dave is on the list , if he 's invited to these meetings ,  if he knows . grad c: we don't have a active one but I 'll make sure he 's on the list . Should we call him ?  is he d is he definitely not available today ? professor e: I don't know . phd a: Yeah , he was in s postdoc f: He wasn't there at cof professor e: Yeah , so this might be a conflict for him . grad c: Yeah didn't he say his signal - processing class was like Tuesdays and Thursdays ? phd a: I think he has a class . grad d: Yeah , I think he 's taking two twenty - five A which is now . professor e: And the and the recordings also , postdoc f:  professor e: just where we are . postdoc f: Well , so  , should we we don't wan wanna do the recording status first , or ? grad c: Well , we have about thirty - two hours  as of , I guess a week and a half ago , so we probably now have about thirty - five hours . professor e: And and that 's that 's  How much of that is digits ? It 's  that 's including digits , grad c: That 's including digits . professor e: right ? grad c: I haven't separated it out so I have no clue how much of that is digits . So anyway there 's at least probably thirty hours , or something of There 's got to be more than thirty hour phd a: Mmm . grad c: Of of non - digits ? professor e: i it couldn't of Of non - digits . postdoc f: OK , and the transcribers h I , don't have the exact numbers , but I think it would come to about eleven hours that are finished  , transcribing from them right now . The next step is to that I 'm working on is to insure that the data are clean first , and then channelized . What  by clean is that they 're spell - checked , that the mark - up is consistent all the way throughout , and also that we now incorporate these additional conventions that  , Liz requested in terms of  ,  in terms of having a s a systematic handling of numbers , and acronyms which I hadn't been specific about . And you know , so how you could grad c: Nine two , postdoc f: e Exactly . postdoc f: So if you just say " nine two " , the there are many s ways that could have been expressed . An - and I just had them I  , a certain number of them did put the words down , but now we have a convention which also involves having it followed by , a gloss th and things . phd b: one suggestion and you may already be doing this , but I 've noticed in the past that when I 've gone through transcriptions and you know in in order to build lexicons and things , if you  , just take all the transcriptions and separate them into words and then alphabetize them , a lot of times just scanning down that list you 'll find a lot of inconsistencies and mis grad c: Misspelled . postdoc f: You 're talking about the type token frequency listings , and I use those too . Y you mean just  on each on each line there 's a one word right ? It 's one token from the from the corpus . postdoc f: Yeah , those are e extremely efficient and I and I I agree that 's a very good use of it . postdoc f: Well that 's that 's a way that 's You know , the spell - check basically does that but but in addition yes , that 's that 's exactly the strategy I wanna do in terms of locating these things which are you know colloquial spoken forms which aren't in the lexicon . Cuz a lot of times they 'll appear next to each other , and  , postdoc f: Exactly . phd b: i in alphabetized lists , they 'll appear next to each other and and so it makes it easier . And that was that 's my strategy for handling a lot of these things , in terms of things that need to be glossed . I didn't get to that point but So there are numbers , then there are acronyms , and then  , there 's a he she wants the  , actually a an explicit marker of what type of comment this is , so i curly b inside the curly brackets I 'm gonna put either " VOC " for vocalized , like cough or like laugh or whatever , " NONVOC " for door - slam , and " GLOSS " for things that have to do with if they said a s a spoken form with this m this pronunciation error . postdoc f: but I I haven't been asking these people to do it systematically cuz I think it most ha most efficiently handled by  by a a filter . So that , you know you get a whole long list exactly what you 're saying , you get a whole list of things that say " curly bracket laugh curly bracket " , phd b:   postdoc f: then y you know it 's it 's You you risk less error if you handle it by a filter , than if you have this transcriber ch laboriously typing in sort of a VOC space , phd b: Yeah . postdoc f: So ,  I 'm I 'm going to convert that via a filter , into these tagged  , subcategorized comments , and same thing with you know , we see you get a subset when you do what you 're saying , phd b:   postdoc f: you end up with a s with  , you 're collapsing across a frequency you just have the tokens phd b:   But the numbers and acronyms have to be handled by hand , because , you know  , jus grad c: You don't know what they could be . postdoc f: Yeah now TIMIT 's clear  and PLP is clear but  there are things that are not so well known , in or or have variant u u uses like the numbers you can say " nine two " or you can say " ninety - two " , grad c: So how are you doing the postdoc f: and  I 'd handle the numbers individually . grad c: How are you doing the  , acronyms so if I say PZM what would it appear on the transcript ? postdoc f: It would be separate The letters would be separated in space grad c: OK . postdoc f: and potentially they 'll have a curly bracket thing afterwards e but I 'm not sure if that 's necessary , clarifying what it is , grad c:   Maybe it 's a nice thing to do because of it then indicating this is  , a step away from i indicating that it really is intentional that those spaces are there , and indicating why they 're there to indicate that it 's  the you know ,  enumerated , or i grad c:   postdoc f: it 's not a good way of saying but it 's it 's the specific  way of stating these these letters . postdoc f: And so anyway , the clean those are those things and then channelized is to then  , get it into this multichannel format . But that 's been my top priority beyond getting it tanel channelized , the next step is to work on tightening up the boundaries of the time bins . postdoc f: And  , Thilo had a e e a breakthrough with this this last week in terms of getting the channel - based   s s speech - nonspeech segmentation  , up and running and I haven't I haven't been able to use that yet cuz I 'm working s re this is my top priority get the data clean , and channelized . phd a: I actually gave grad c: Have you also been doing spot checks , Jane ? postdoc f: Oh yes . I spent  actually  I have a segment of ten minutes that was transcribed by two of our transcribers , grad c: Oh good . postdoc f: and I went through it last night , it 's it 's almost spooky how similar these are , word for word . And there are some differences in commas cuz commas I I left them discretion at commas . postdoc f:  and so because it 's not part of our st of our ne needed conventions . postdoc f: And  , and so they 'll be a difference in commas , but it 's word - by - word the same , in in huge patches of the data . And and sometimes it turns out that one of these transcribers has a better ear for technical jargon , and the other one has a better ear for colloquial speech . So  , the one i i the colloquial speech person picked up " gobbledy - gook " . And on this side , this one 's picking up things like " neural nets " and the one that 's good on the sp o on th the vocabulary on the  colloquial didn't . phd b: When for the person who missed " gobbledy - gook " what did they put ? postdoc f: It was an interesting approximation , put in parentheses , cuz I have this convention that , i if they 're not sure what it was , they put it in parentheses . postdoc f: So they tried to approximate it , but it was phd b: Oh good . More of an attempt to  apparently it was very clear to her that these the a this this was a sound these are the sounds , grad c: It was a technical term that she didn't recognize , phd b: Yeah . But she even though her technical perception is just really  you know I 've I 'm tempted to ask her if she 's taken any courses in this area or if she 's taken cognitive science courses grad c: Right . postdoc f: then cuz " neural nets " and oh she has some things that are oh " downsampled " , she got that right . postdoc f: But ch ten solid  m ch s chunk of ten solid minutes where they both coded the same data . professor e: And and again the main track that you 're working with is elev eleven hours ? postdoc f: And  professor e: Is that right ? postdoc f: Yes exactly . professor e: So let 's say roughly ten hours or so of postdoc f:   postdoc f: It 'd be more than that because I my recollection is the minutes that da digits don't take more than half a minute . postdoc f: But  the the total set that I gave them is twelve hours of tape , professor e: Oh , I see . postdoc f: So they 're still working some of them are Two of them are still working on completing that . professor e: So what what what 's the deal with with your phd a: The channel u thing ? professor e: Yeah . phd a: Oh , it 's just  , I ran the recognizer  , the speech - nonspeech detector on different channels and , it 's just in  in this new multi - channel format and output , and I just gave one one meeting to to Liz who wanted to to try it for for the recognizer professor e: Oh , I see . phd a: as  , apparently the recognizer had problems with those long chunks of speech , which took too much memory or whatever , professor e: Right . So , I hope grad c: Is this anything different than the  system you were using before ? professor e: Yeah . phd a: There is some ,  as the energy is normalized across channels grad c: You know what  ? Across all of them . What are some of the other features ? Besides the energy ? You said you 're trying some different features , or something . phd a: Oh I just  Mmm , I just use  our loudness - based things now as they before there were they were some in in the log domain and I I changed this to the to the professor e: Cu - Cube root ? phd a: Yeah . To No , I changed this to the to the to the loudness thingy with the with the grad c:  . Yeah , and I and I tried t to normalize   the features , there 's loudness and modified loudness , within one channel , professor e: OK . phd a: because they 're , yeah to to be able to distinguish between foreground and background speech . I think the  Were were you basically done with the transcription part ? So I guess the next thing is this  bleep editing . So the The idea is that we need to have We need to provide the transcripts to every participant of every meeting to give them an opportunity to bleep out sections they don't want . So I 've written a bunch of tools that will generate web pages ,  with the transcription in it so that they can click on them and piece pieces and they can scroll through and read them , and then they can check on each one if they want it excluded . And then , it 's a form , HTML form , so they can submit it and it will end up sending me email with the times that they want excluded . And so I thought about this a little bit and I think the best way to do it is every participant will have a password , professor e: Yeah . And then each meeting , we 'll only allow the participants who were at that meeting to look at it . professor e: I I can't help but wonder if this is maybe a little more elaborate than is needed .  if people have  , for me I would actually want to have some pieces of paper that had the transcription and I would sort of flip through it . professor e: And , I   it depends how this really ends up working out , but I guess my thought was that the occasion of somebody wondering whether something was OK or not and needing to listen to it was gonna be extremely rare . grad c: Right ,  so th th th the fact that you could listen to it over the web is a minor thing that I had already done for other reasons . grad c: And so that that 's a minor part of it , I just wanted some web interface so that people you didn't actually have to send everyone the text . So m what my intention to do is that as the transcripts become ready ,  I would take them , and generate the web pages and send email to every participant or contact them using the contact method they wanted , and just  , tell them , " here 's the web page " , " you need a password " . So th th question number one is how do we distribute the passwords , and question number two is how else do we wanna provide this information if they want it . professor e: That 's I think what I was sort of saying is that if you just say " here is a here is "  this maybe it sounds paleolithic but but I just thought if you handed them some sheets of paper , that said , " here 's what was said in this transcription is it OK with you ? and if it is , here 's this other sheet of paper that you sign that says that it 's OK " . grad c: I think that  there are a subset of people who will want printouts that we can certainly provide . These are big , and I would much rather be ha be able to just sit and leaf through it . professor e: You find it easier to go through a large  how do you read books ? grad c: Well I certainly read books by hand . professor e: Really ?  , it grad c: Cuz you 're gonna get , you know , if I I 'm I 'm in a bunch of meetings and I don't wanna get a stack of these . professor e: Going to a web site is easy , but flipping through a hundred pounds a hundred pages of stuff is not easy on the web . grad c: Well , I don't think it 's that much harder than , paper . So are you thinking that  the person would have a transcript and go strictly from the transcript ? Because I I do think that there 's a benefit to being able to hear the tone of voice and the professor e: So here 's the way I was imagining it , and maybe I 'm wrong , postdoc f: Yeah . professor e: but the way I imagined it was that  , the largest set of people is gonna go " oh yeah , I didn't say anything funny in that meeting just go ahead , where 's the where 's the release ? " And then there 'll be a subset of people , right ? OK there 's  think of who it is we 've been recording mostly . professor e: OK there 'll be a subset of people , who  , will say  " well , yeah , I really would like to see that . " And for them , the easiest way to flip through , if it 's a really large document ,  unless you 're searching . Searching , of course , should be electronic , but if you 're not so if you provide some search mechanism you go to every place they said something or something like that , phd a: Yeah .  if if  you don't have search mechanisms you just sort of have this really , really long document ,  whenever I 've had a really , really long document that it was sitting on the web , I 've always ended up printing it out .  , so it 's it 's  , you you 're you 're not necessarily gonna be sitting at the desk all the time , you wanna figure you have a train ride , and there 's all these situations where where I  , this is how I was imagining it , anyway . And then I figured , that out of that group , there would be a subset who would go "  you know I 'm really not sure about this section here , " and then that group would need it S It seems like i if I 'm right in that , it seems like you 're setting it up for the most infrequent case , rather than for the most frequent case . So that  , now we have to worry about privacy , grad c: Well , no fre for the most professor e: we have to worry about all these passwords , for different people grad c: For the most frequent case they just say " it 's OK " and then they 're done . postdoc f: The other thing too is it seems like professor e: yeah , that 's true . grad c: cuz you don't have to visit the web page if you don't want to . grad c: Oh that was another thing I I had assumed that we didn't need their signature , that it that an email approval was sufficient . phd b: Are are people going to be allowed to bleep out sections of a meeting where they weren't speaking ? grad c: Yes . If someone feels strongly enough about it , then I I I think they should be allowed to do that . phd b: So that means other people are editing what you say ? professor e:  I don't know about that . grad c: Well , the only other choice is that the person would say " no , don't distribute this meeting at all " , and I would rather they were able to edit out other people then just say " don't distribute it at all " . professor e: But th what they signed in the consent form , was something that said you can use my voice . grad c: Well , but if if someone is having a conversation , and you only bleep out one side of it , that 's not sufficient . grad c: If I say " we were having a conversation , and I consider that conversation private , " and I consider that your side of it is enough for other people to infer , I wanna be able to bleep out your side . postdoc f: The I agree that the consent forms were  , I cons agree with what Adam 's saying , that  , the consent form did leave open this possibility that they could edit things which they found offensive whe whether they said them or didn't say them . postdoc f: And the other thing is from the standpoint of the l of the l I 'm not a law lawyer , but it strikes me that  , we wouldn't want someone to say " oh yes , I was a little concerned about it but it was too hard to access " . Now in terms of like editing it by hand ,  I think it 's i some people would find that easier to specify the bleep part by having a document they edited . But but it seems to me that sometimes  , you know i if a person had a bad day , and they had a tone in their voice that they didn't really like , you know it 's nice it 's nice to be able to listen to it and be sure that that was OK . professor e:   it 's also a mixture of people ,  some people are r do their work primarily by sitting at the computer , flipping around the web , and others do not . You know ? grad c: Well I think most of the people in the meetings are the former . grad c: And that Well , but if they want to print it out that 's alright . I I think grad c: OK , so does that mean that I can't use email ? Or what ? postdoc f: Cuz you could send it through email you 're thinking . professor e: I I th grad c: Well , I don't think I professor e: well we there was this grad c: well I don't think we can send the text through email because of the privacy issues . So giving them , you think a web site to say , " if you wanna print it out here it is " , is not sufficient ? postdoc f: Good point . I professor e: Certainly for everybody who 's been in the meetings so far it would be sufficient . grad c: Yeah , I 'm just thinking for people that that 's not sufficient for , what the only sufficient thing would be for me to walk up to them and hand it to them . professor e: I 'm just wondering about postdoc f: You could mail it to them . phd a: But postdoc f: But I think it 's easier to drop in the box . phd a: Just put the button on on the web page which say " please send me the the scripts " . phd b: What  When you display it on the web page , what are what are you showing them ? Utterances , or ? grad c:   phd b: And so can they bleep within an utterance ? grad c: No . grad c: And that was just convenience for my sake , that it 's  ,  it would end up being fairly difficult to edit the transcripts if we would do it at the sub - utterance level . Because this way I can just delete an entire line out of a transcript file rather than have to do it by hand . professor e: There 's another aspect to this which maybe is part of why this is bothering me .  , I think you 're really trying very hard to make this as convenient as possible for people to do this . grad c:  that 's why I did the web form , because for me that would be my most convenient . You 're gonna end up with all these little patchy things , whereas really what we want to do is have the the the bias towards letting it go . Because nob you know it There was a one or twi once or twice , in the re in the meetings we 've heard , where somebody said something that they might be embarrassed by , but overall people are talking about technical topics . You know , this is this we 're we 're covering We 're playing the lawyer 's game , and we 're playing we 're we 're we 're looking for the extreme case . If we really orient it towards that extreme case , make it really easy , we 're gonna end up encouraging a headache . That I think that 's I 'm sort of psyching myself out here , I I 'm trying to  grad c: I guess I don't see having a few phrases here and there in a meeting being that mu much of a headache , bleeped out . professor e: but I I think that 's Well , it 's grad c: So . phd b: I think what Morgan 's saying is the easier it is , the more is gonna be bleeped . I think some researchers who are gonna be working with this corpus years from now are really gonna be cursing the fact that there 's a bunch of stuff in there that 's missing from the dialogue . professor e: You know , it depends on the kind of research they 're doing , phd a: Yeah . And , you know where it 's really gonna hurt somebody , in some way the one who said it or someone who is being spoken about , we definitely want to allow the option of it being bleeped out . And and  , I am just a little worried about making it so easy for people to do , and so much fun ! that they 're gonna go through and bleep out stuff . professor e: and they can bleep out stuff they don't like too , right from somebody else , as you say , you know , so " well I didn't like what he said .  , we have to provi we have promised that we would provide them the transcript and that they can remove parts that they don't like . No , no , I I I don't grad c: The only question is professor e: You - you 've talked me into that , but I I just think that we should make it harder to do . grad c: The problem is if it 's harder for them it 's also harder for me . Whereas this web interface , I just get email , it 's all formatted , it 's all ready to go and I can just insert it . professor e: So maybe you don't give them access to the web interface unless they really need it . professor e: I 'm sorry so so So maybe this is a s a way out of it . professor e: You 've provided something that 's useful for you to do handle , and useful for someone else if they need it . But I think the issue of privacy and ease and so forth should be that  , they get access to this if they really need it . grad c: Well phd b: So you 're saying the the sequence would be more like first Adam goes to the contact lists , contacts them via whatever their preferred method is , to see if they want to review the meeting . grad c: Well , to some extent I have to do that anyway because as I said we have to distribute passwords . professor e: There 's there grad c: So , professor e: y but you don't necessarily have to distribute passwords is what I 'm saying . grad c: what I 'm saying is that I can't just email them the password because that 's not secure . professor e: We 're We 're trying i We 're trying to make it less of an obvious just l l l l  fall off a log , to do this . professor e: Right ? So th so what I would see , is that first you contact them and ask them if they would like to review it for to check for the postdoc f: Yeah . professor e: not just for fun , OK ? but to to check this for  things that they 're worried about having said or if they 're willing to just send an approval of it , at from their memory .  and , and we should think carefully actually we should review go through how that 's worded , OK ? Then , if someone  wants to review it , and I know you don't like this , but I 'm offering this as a suggestion , is that is that we then give them a print out . And then if they say that " I have a potential problem with these things , " then , you you say " OK well you might wanna hear this in context to s think if you need that , " you issue them a password , i in the grad c: But the the problem with what you 're suggesting is it 's not just inconvenient for them , it 's inconvenient for me . professor e: I know you 'd prefer it , but the proble grad c: Yeah . professor e: we have grad c: So I think you 're thinking people are going to arbitrarily start bleeping and I just don't think that 's gonna happen . postdoc f: I 'm also concerned about the spirit of the of the informed consent thing . Cuz I think if they feel that  , it 's I th I th You know , if it turns out that something gets published in this corpus that someone really should have eliminated and didn't detect , then it could have been because of their own negligence that they didn't pursue that next level and get the password and do that , but but they might be able to argue " oh well it was cumbersome , and I was busy and it was gonna take me too much time to trace it down " . So I 'm a little bit worried about  , making it harder for them , from the legal standpoint . professor e: Well you can go too far in that direction , and you need to find somewhere between I think , postdoc f: Yeah . grad c: It seems to me that sending them email , saying " if you have an O - OK reply to this email and say OK , professor e: because  - huh . grad c: If you have a problem with it contact me and I 'll give you a password " , seems like is a perfectly , reasonable compromise . postdoc f:  we could offer that but but there 's  , another aspect to that and that is that in the informed consent form , my impression is that they that we offered them at the very least that they definitely would have access to the transcript . postdoc f: I don't know that there 's a chance of really skipping that stage .  I I thought that you were Maybe I misinterpreted what you said but it 's professor e: Having access to it doesn't necessarily mean , that having it grad c: Having it . grad c: Well the in professor e: right ? It just means they have the right to have it . grad c: the consent form is right in there if anyone wants to look at it , postdoc f: Alright . grad c: D you want me to grab one ? postdoc f: Sh - sh well I could I 'm closer . grad c: Yeah , but you 're wired postdoc f: I could grad c: aren't you ? postdoc f: Yeah . Yeah ,  I don't wanna fool them , postdoc f: I don't know professor e: I just meant that e every ev any time you say anything to anyone there is in fact a a bias that is presented , postdoc f: Oh yeah yeah oh I know . professor e: right ? grad c: " If you agree to participate you 'll have the opportunity to have anything ex anything excised , which you would prefer not to have included in the data set . grad c: " Once a transcript is available we will ask your permission to include the data in the corpus for the r larger research community . grad c: There again you will be allowed to indicate any sections that you 'd prefer to have excised from the database , and they will m be removed both from the transcript and the recording . grad c: Well ,  it The one question is definitely clear with anything as opposed to just what you said . professor e: Yeah ,  no that it tha postdoc f: Tha - that 's true . postdoc f: And it doesn't really say we 'll send it to you , or wi it 'll be available for you on the web , or anything .  like like Morgan was saying they they grad c: They just have to make sure that it is available to them . wh  I think I have an idea that may be sat may satisfy both you and me in this which is , it 's a it we just go over carefully how these notes to people are worded . So I I just want it to be worded in such a way where it gives the strong impre it gives very ,  nothing hidden , v very strongly the bias that we would really like to use all of these data . professor e: That that we really would rather it wasn't a patchwork of things tossed out , postdoc f: Good . But if you really think something is gonna And I don't think there 's anything in the legal aspects that that is hurt by our expressing that bias . professor e: you know you might be right , it may be it was just paranoia on my part ,  but people just See I 'm @ @ worried about this interface so much fun that people start bleeping stuff out just as just because they can . grad c: It 's just a check box next to the text , it 's not any fun at all . professor e: I grad c: All no because it doesn't automatically bleep it at the time . professor e: Oh they won't ? grad c: It just sends me professor e: Oh good . professor e: OK , grad c: And then at some point I 'll incorporate them all and put bleeps .  I don't wanna have t ha do that yet until we actually release the data professor e: Yeah . grad c: because  , then we have to have two copies of every meeting and we 're already short on disk space . grad c: So I I wanna I just keep the times until we actually wanna release the data and then we bleep it . Alright , so I think Yeah so if we have if i Again let 's you know , sort of circulate the the wording on each of these things and get it right , grad c: Well since you seem to feel heart  , strongest about it , would you like to do the first pass ? professor e: but but OK . Turn about is fair play , postdoc f: Al - Also it ther there is this other question , the legal question that that Adam 's raised ,  about whether we need a concrete signature , or email c i suffices or whatever professor e: Sorry . i There 's something down there about " if you agree to " professor e: I 'm I 'm I 'm I thought I I thought about it with one of my background processes grad c: I don't think so . professor e: and I  it 's  it 's  , it 's fine to do the email . grad c: Yeah because thi th they 're signing here that they 're agreeing to the paragraph which says " you 'll be given an opportunity . professor e: And Well and furthermore I it 's now fairly routine in a lot of arrangements that I do with people on contracts and so forth that that  if it 's if it 's that sort of thing where you 're you 're saying  " OK I agree , we want eighty hours of this person at such - and - such amount , and I agree that 's OK , "  if it 's a follow up to some other agreement where there was a signature it 's often done in email now grad c: Right . grad c: So I guess I probably should at the minimum , think about how to present it in a printed form . The problem is a lot of them are really short , postdoc f: Well grad c: and so I don't necessarily wanna do one per line . postdoc f: Well I s I also have this I I think it 's nice you have it  , viewab her hearable on the on the web for those who might wonder about  , the non nonverbal side ,  I I agree that our bias should be as as expressed here , and but I I think it 's nice that a person could check . Cuz sometimes you know you the words on a on the page , come out soun sounding different in terms of the social dynamics if they hear it . postdoc f: And I realize we shouldn't emphasize that people you know , shouldn't borrow trouble . What it comes down to but grad c: Yeah I think actually my opinion probably is that the only time someone will need to listen to it is if the transcript is  not good . postdoc f: Oh , you know , or what if there was an error in the transcript that didn't get detected and there was a whole  i segment a against some personal i th grad c: Right . grad c: I think Microsoft is postdoc f: Yeah exactly phd a: Oh , grad c: Sorry transcribers . postdoc f: Or or even or even there was a a line you know about how "  - mmm - mmm Bill Gates duh - duh - duh - duh . postdoc f: but but it was all the words were all visible , but they didn't end up i some there was a slip in the transcript .  , I I guess we 're assuming that the transcript is a close enough approximation and that that my double checking will be so close to absolutely perfect that it that nothing will slip by . professor e: But it the some something might sometime , and they  if if it 's something that they said , they might i i  , you might be very accurate in putting down what they actually said , postdoc f:   professor e: but , when they hear it , themselves , they may hear something different because they know what they meant . phd b: how do you how do you indicate sarcasm ? postdoc f: Yeah that 's right . So the so i the so we might we might get some feedback from people that such - and - such was , you know , not not really what I said . grad c: So  , in terms of password distribution , I think phone is really the only way to do it , phone and in person . postdoc f: You know I just realized something , which is of e th this question about the  the possible mismatch of  i well , and actually also the lawyer saying that  , we shouldn't really have them have the people believing that they will be cleared by our checks . So it 's like i in a way it 's it 's nice to have the responsibility still on them to listen to the tape and and hear the transcript , to have that be the professor e: Well yeah , but you can't dep  , most people will not wanna take the time to do that , though . professor e: And they they have to postdoc f: So it 's not it 's not  Yeah , good . professor e: But  if you were at a meeting , and and you you don't think , at least , that you said anything funny and the meeting was about , you know , some some funny thing about semantics or something , or  grad c: You probably won't listen to it . postdoc f: It is true that tec that the content is technical , I and so i and we 're not having these discussions which professor e: Yeah . postdoc f: I  , when I listen to these things , I don't find things that are questionable , in other people 's speech or in my own . professor e:  we 're not talking about the energy crisis or something , people have postdoc f: Yeah . Actually , I was gonna Di - Did you have anything n that 's going on , or grad d: Not really .  , my project is going along but  , I 'm really just here to  fill the project  the overall progress . grad c:   professor e: Transcribers , he was rattling the b marbles in his brain back and forth just then this this grad c: Shall we do digits ? professor e: Oh yeah . grad c: oh by the way I did find a bunch grad d: It  grad c: we should count out how many more digits to forms do we have back there ? phd b: There were quite a few . I f I was going through them all and I found actually a lot filed in with them , that were blanks , that no one had actually read . postdoc f: Oh , did you do that ? phd a: Burger King grad c: I 'd like a burger with that , phd b: No I never did 
The group aims to have the question answering mock-up and transcriber interface ready for the DARPA meeting in July, along with available data. Speaker fe008 is currently working on ensuring clean and properly coded transcripts, while speaker mn014's multi-channel speech/non-speech segmenter is being utilized. To save time, speaker mn005 will only mark regions of overlapping speech in the transcribed data. The task of extracting digits will be assigned to the person working on acoustics for the Meeting Recorder project.
Summarize the discussion, requirements, and agreement regarding the transcripts.