phd d: there 's gon na be some zeros from this morning 's meeting because i noticed that barry , maybe you turned your mike off before the digits were , was it during digits ? , so it does n't matter . phd b: so it 's not that bad if it 's at the end , but it 's in the beginning , it 's bad . grad a: , you wanna keep them on so you get good noise floors , through the whole meeting . professor f: i probably just should have left it on . i did have to run , but grad e: where like you just do n't like if you if it starts catching zeros , like in the driver in the card , or somewhere in the hardware where if you start seeing zeros on w across one channel , you just add some random , @ noise floor like a small noise floor . grad a: certainly we could do that , but i do n't think that 's a good idea . we can do that in post - processing if the application needs it . professor f: , i u i actually what the default is anymore as to how we 're using the front - end but for when we use the icsi front - end , professor f: but , there is an o an option in rasta , which , in when i first put it in , back in the days when i actually wrote things , i did actually put in a random bit or so that was in it , professor f: but then i realized that putting in a random bit was equivalent to adding flat spectrum , professor f: and it was a lot faster to just add a constant to the spectrum . so then started doing that professor f: instead of calling " rand " , so . so it d it does that . gee ! here we all are ! grad a: so the only agenda items were jane was jane wanted to talk about some of the ibm transcription process . grad a: i condensed the three things you said into that . and then just i only have like , this afternoon and maybe tomorrow morning to get anything done before i go to japan for ten days . so if there 's anything that n , desperately needs to be done , you should let me know now . phd g: liz had sent them a note saying " could we have another " i , " three days " , and they said yes . grad a: , dave gelbart sent me email , he sent it to you too , that , there 's a special topic , section in si in eurospeech on new , corp corpors corpora . and it 's not due until like may fifteenth . grad a: i s forwarded it to jane as being the most relevant person . so , it was highly relevant grad a: i 'll help , but i ca n't , really do , most of it , phd g: there were some interesting results in this paper , though . that morgan , accounted for fifty - six percent of the robustness meetings in terms of number of words . postdoc c: because is it partly , c correctly identified words ? or is it or just overall volume ? professor f: i get these aarp things , but i ' m not se really senior yet , professor f: but , other than that delightful result , what was the rest of the paper about ? phd g: three kinds of results , if you will . , the one was that the just the amount of overlap phd g: s in terms of number of words and also we computed something called a " spurt " , which is essentially a stretch of speech with , no pauses exceeding five hundred milliseconds . , and we computed how many overlapped i spurts there were and how many overlapped words there were . , for four different corpora , the meeting recorder meetings , the robustness meetings switchboard and callhome , and , found and compared the numbers . and found that the , as you might expect the meeting recorder meetings had the most overlap , but next were switchboard and callhome , which both had roughly the same , almost identical , and the robustness meetings were had the least , so one unexpected result there is that two - party telephone conversations have about the same amount of overlap , phd g: in gen order of magnitude - wise as , as face - to - face meetings with multiple phd g: also , i in the levinson , the pragmatics book , in , textbook , there 's i found this great quote where he says , how people it talks about how people are so good at turn taking , and so they 're so good that generally , u the overlapped speech does not is less than five percent . phd g: , in real conversations , everyday conversations . it 's what these conversation analysts have been studying for years and years there . postdoc c: , no , it does n't necessarily go against what he said , cuz he said " generally speaking " . in order to go against that a claim you 'd have to big canvassing . postdoc c: it just says that it 's a bi bell curve , and that , you have something that has a range , in your sampling . phd g: so there are slight there are differences in how you measure it , but still it 's , the difference between that number and what we have in meetings , which is more like , close to in meetings like these , close to twenty percent . phd g: robustness meeting ? it was about half of the r so , in terms of number of words , it 's like seventeen or eigh eighteen percent for the meeting recorder meetings and about half that for , the robustness . grad a: but i if that 's really a fair way of comparing between , multi - party , conversations and two - party conversations . i . phd g: , we did n't get to look at that , but this obvious thing to see if there 's a dependence on the number of participants . grad a: i bet there 's a weak dependence . i ' m it 's not a real strong one . right ? because you grad a: you have a lot of two - party , subsets within the meeting . regardless it 's an interesting result regardless . phd g: and and then and we also d computed this both with and without backchannels , so you might think that backchannels have a special status because they 're essentially just grad a: so , did we all said " - " and nodded at the same time , phd g: r right . but , even if you take out all the backchannels so you treat backchannels l as nonspeech , as pauses , phd g: you still have significant overlap . , it goes down from maybe for switchboard it goes down from i f fourteen percent of the words to maybe i , eleven percent it 's not a dramatic change , so it 's anyway , that was that was one set of results , and then the second one was just the we had in the hlt paper on how overlaps effect the recognition performance . phd g: and we rescored things , a little bit more carefully . we also fixed the transcripts in numerous ways . , but mostly we added one number , which was what if you , score ignoring all so so the conjecture from the hlt results was that most of the added recognition error is from insertions due to background speech . so , we scored all the recognition results , in such a way that the phd g: ok , so if you have the foreground speaker speaking here , and then there 's some background speech , may be overlapping it somehow , and this is the time bin that we used , then you 're gon na get insertion errors here and here . right ? so we scored everything , and i must say the nist scoring tools are pretty for this , where you just ignore everything outside of the , region that was deemed to be foreground speech . and where that was we had to use the t forced alignment , results from s for that 's somewhat that 's somewhat subject to error , but still we , don did some ha hand - checking and we think that based on that , we think that the results are , valid , although , some error is gon na be in there . but what we found is after we take out these regions so we only score the regions that were certified as foreground speech , the recognition error went down to almost , the level of the non - overlapped speech . so that means that even if you do have background speech , if you can somehow separate out or find where it is , the recognizer does a good job , grad a: , i that does n't surprise me , because , with the close - talking mikes , the signal will be so much stronger . phd g: , we do , vtl vocal tract length normalization , w and we , make all the features have zero mean and unit variance . phd g: over the but . , now we did n't re - align the recognizer for this . we just took the old so this is actually a sub - optimal way of doing it , right ? so we took the old recognition output and we just scored it differently . so the recognizer did n't have the benefit of knowing where the foreground speech a start phd g: i should bring the table with results . maybe we can look at it monday . professor f: i would presume that you still would have somewhat higher error with the lapel for insertions than professor f: cuz again , looking forward to the non - close miked case , that we s still professor f: i it 's the high signal - to - noise ratio here that helps you . phd g: u s right . so that was number that was the second set of , the second section . and then , the third thing was , we looked at , what we call " interrupts " , although that 's that may be a misnomer , but we looked at cases where , so we used the punctuation from the original transcripts and we inferred the beginnings and ends of sentences . postdoc c: di - did you use upper - lower case also , or not ? u upper lower case or no ? phd g: ? we only used , periods , question marks and exclamation . and we know that there 's th that 's not a very g , we miss a lot of them , phd g: no commas . no . and then we looked at locations where , if you have overlapping speech and someone else starts a sentence , where do these where do other people start their turns not turns really , but , sentences , so we only looked at cases where there was a foreground speaker and then at the to at the so the foreground speaker started into their sentence and then someone else started later . phd g: , so that such that there was overlap between the two sentences . so , the question was how can we what can we say about the places where the second or actually , several second speakers , start their " interrupts " , as we call them . phd g: so so we had u to for the purposes of this analysis , we tagged the word sequences , and we time - aligned them . and we considered it interrupt if it occurred in the middle of a word , we , considered that to be a interrupt as if it were at the beginning of the word . so that , if any part of the word was overlapped , it was considered an interrupted word . and then we looked at the locatio the , , the features that the tags because we had tagged these word strings , that occurred right before these , interrupt locations . phd g: and the tags we looked at are the spurt tag , which says or actually end of spurt . so whether there was a pause essentially here , because spurts are a defined as being , five hundred milliseconds or longer pauses , and then we had things like discourse markers , backchannels , disfluencies . , filled pauses so disfluen the d 's are for , the interruption points of a disfluency , so , where you hesitate , or where you start the repair there . , what else do we had . , repeated words is another of that disfluencies and . so we had both the beginnings and ends of these so , the end of a filled pause and the end of a discourse marker . and we just eyeballed we did n't really hand - tag all of these things . we just looked at the distribution of words , and so every " so " , and " ok " , and " - " were the were deemed to be backchannels and " so " and " right " , were not " right " . " right " is a backchannel . but so , we just based on the lexical , identity of the words , we tagged them as one of these things . and the d the interruption points we got from the original transcripts . and then we looked at the disti so we looked at the distribution of these different kinds of tags , overall , and particularly at the interruption points . and , we found that there is a marked difference that after so at the end after a discourse marker or after backchannel or after filled pause , you 're much more likely to be interrupted than before . and also after spurt ends , which means in p inside pauses . so pauses are always an opportunity for so we have this little histogram which shows these distributions and , phd d: i wonder about the and effect there . in other words if you were n't going to pause you will because you 're g being interrupted . professor f: but he , he 's right , y maybe you were n't intending to pause , but you were intending to stop for fifty - seven milliseconds , but then chuck came in and so you paused for a second phd g: , and that was it . and and we so we wrote this and then , we found we were at six pages , and then we started cutting furiously and threw out half of the material again , and played with the latex and phd g: the gap between the two columns is like ten millimeters , so i d shrunk it to eight millimeters and that helped some . and like that . phd d: was n't there was n't there some result , andreas maybe liz presented this at some conference a while ago about , backchannels and that they tend to happen when the pitch drops . you get a falling pitch . and so that 's when people tend to backchannel . - i do you rem phd g: now that we have the data and we have the alignment , this is purely based on the words phd d: it made me think about a little device that could be built to handle those people that call you on the phone and just like to talk and talk . and you just have this little detector that listens for these drops in pitch and gives them the backchannel . and so then you hook that to the phone and go off and do the do whatever you r wanna do , while that thing keeps them busy . phd g: there 's actually there 's this a former student of here from berkeley , nigel ward . do him ? he did a system , in he lives in japan now , and he did this backchanneling , automatic backchanneling system . it 's a very phd g: so , exactly what you describe , but for japanese . and it 's for japa - in japanese it 's really important that you backchannel . it 's really impolite if you do n't , so . professor f: actually for a lot of these people you could just backchannel continuously and it would be fine . grad a: there was there was a monty python sketch with that . where the barber who was afraid of scissors was playing a tape of clipping sounds , and saying " - " , how about them sports teams ? phd g: anyway . so the paper 's on - line y i cc ' ed a message to meeting recorder with the url so you can get it . phd g: , one more thing . so i ' m actually about to send brian kingbury an email saying where he can find the s the m the material he wanted for the s for the speech recognition experiment , but i have n't sent it out yet because actually my desktop locked up , like i ca n't type anything . b so if there 's any suggestions you have for that i was just gon na send him the phd g: he prefe he said he would prefer ftp and also , the other person that wants it there is one person at sri who wants to look at the , the data we have so far , and so i figured that ftp is the best approach . so what i did is i @ i made a n new directory after chuck said that would c that was gon na be a good thing . , so it 's " ftp pub phd g: and then under there actually and this directory , is not readable . it 's only , accessible . so , in other words , to access anything under there , you have to be told what the name is . so that 's a g quick and dirty way of doing access control . , and the directory for this i call it i " asr zero point one " because it 's meant for recognition . phd g: and then in there i have a file that lists all the other files , so that someone can get that file and then know the file names and therefore download them . if you the file names you ca n't phd g: dash . so all i was gon na do there was stick the transcripts after we the way that we munged them for scoring , because that 's what he cares about , and also and then the waveforms that don segmented . , just tar them all up f w for each meeting i tar them all into one tar file and g - zip them and stick them there . phd d: so we could point mari to this also for her march o - one request ? phd d: she was saying that it would be if we had they had a or was she talking she was saying it would be if they had the same set , so that when they did experiments they could compare . phd d: we need to give brian the beeps file , so i was gon na probably put it grad e: so . , so either we should regenerate the original versions , or , we should just make a note of it . grad e: , that 's the first meeting i cut both versions . just to check which w if there is a significant difference . phd g: and so i ok so but for the other meetings it 's the downsampled version that you have . phd g: that 's th important to know , we should probably give them the non - downsampled versions . grad e: , because liz decided to go ahead with the downsampled versions cuz we can there was no s like , r significant difference . grad e: it does take up less disk space , and it did even better than the original versions , which , is just , probably random . grad a: ok , we 're losing , don and andreas at three - thirty , right ? ok . professor f: so , that 's why it was good to have andreas , say these things so , we should probably talk about the ibm transcription process that postdoc c: so , that adam created , a b a script to generate the beep file ? to then create something to send to ibm . and , you should probably talk about that . but but you were gon na to use the originally transcribed file because i tightened the time bins and that 's also the one that they had already in trying to debug the first stage of this . and , my understanding was that , i have n't listened to it yet , but it sounded very good and i understand that you guys were going to have a meeting today , before this meeting . grad a: it was just to talk about how to generate it . , just so that while i ' m gone , you can regenerate it if you decide to do it a different way . so , chuck and thilo should , now more or less know how to generate the file grad a: and , the other thing chuck pointed out is that , since this one is hand - marked , there are discourse boundaries . so so when one person is speaking , there 's breaks . whereas thilo 's wo n't have that . so what we 're probably gon na do is just write a script , that if two , chunks are very close to each other on the same channel we 'll just merge them . grad a: so , and that will get around the problem of , the , " one word beep , one word beep " . phd d: , after our meeting , this morning thilo came in and said that , there could be other differences between the already transcribed meeting with the beeps in it and one that has just r been run through his process . phd d: so tomorrow , when we go to make the , chunked file for ibm , we 're going to actually compare the two . so he 's gon na run his process on that same meeting , phd d: and then we 're gon na do the beep - ify on both , and listen to them and see if we notice any real differences . postdoc c: ok , now one thing that prevented us from apply you from applying the training so that is the training meeting . phd d: w and we know that . wel - we just wanna if there 're any major differences between doing it on the hand phd g: so this training meeting , w un is that some data where we have very , accurate time marks ? for postdoc c: i went back and hand - marked the ba the bins , i ment i mentioned that last week . phd d: but the but there 's , but there is this one issue with them in that there 're there are time boundaries in there that occur in the middle of speech . phd d: so like when we went t to when i was listening to the original file that adam had , it 's like you hear a word then you hear a beep and then you hear the continuation of what is the same sentence . grad a: that 's not gon na be true of the foreground speaker . that 'll only be if it 's the background speaker . phd d: right . so you 'll have a chunk of , channel a which starts at zero and ends at ten , and then the same channel starting at eleven , ending at fifteen , and then again , starting at sixteen , ending at twenty . so that 's three chunks where actually we w can just make one chunk out of that which is a , zero , twenty . phd d: so if you were to use these , you have to be careful not to pull out these individual phd g: it w what i would i was interested in is having a se having time marks for the beginnings and ends of speech by each speaker . phd g: it i do n't care that , there 's actually abutting segments that we have to join together . that 's fine . phd g: but what we do care about is that the beginnings and ends are actually close to the speech inside of that postdoc c: they were , reasonably tight , but not excruciatingly tight . that would ' ve taken more time . wanted to get it so tha so that if you have like " in a swimming in a big bin , then it 's phd g: actually i it 's f that 's fine because we do n't want to th that 's perfectly fine . it 's good . you always want to have a little bit of pause or nonspeech around the speech , say for recognition purposes . but just u w get an i d wanted to have an idea of the of how much extra you allowed so that interpret the numbers if i compared that with a forced alignment segmentation . postdoc c: i ca n't answer that , but my main goal was , in these areas where you have a three - way overlap and one of the overlaps involves " , and it 's swimming in this huge bin , i wanted to get it so that it was clo more closely localized . phd g: but are we talking about , i , a tenth of a second ? a ? how how much extra would you allow at most postdoc c: i wanted to i wanted it to be able to l he be heard normally , so that if you play back that bin and have it in the mode where it stops at the boundary , it sounds like a normal word . it does n't sound like the person i it sounds normal . it 's as if the person could ' ve stopped there . and it would n't have been an awkward place to stop . now sometimes , it 's these are involved in places where there was no time . and so , there would n't be a gap afterwards because some cases , there 're some people , who have very long segments of discourse where , they 'll breath and then i put a break . but other than that , it 's really pretty continuous and this includes things like going from one sentence into the u one utterance into the next , one sentence into the next , w without really stopping . i they , i in writing you have this two spaces and a big gap but but i some people are planning and , , a lot we always are planning what we 're going to say next . but , in which case , the gap between these two complete syntactic units , which n spoken things are not always complete syntactically , but it would be a shorter p shorter break than maybe you might like . but the goal there was to not have the text be so crudely parsed in a time bin . , because from a discourse m purpose it 's more useful to be able to see and also , from a speech recognition purpose my impression is that if you have too long a unit , it 's it does n't help you very much either , cuz of the memory . postdoc c: so , that means that the amount of time after something is variable depending partly on context , but my general goal when there was sufficient space , room , pause after it to have it be a natural feeling gap . which i c i what it would be quantified as . , wally chafe says that , in producing narratives , the spurts that people use tend to be , that the what would be a pause might be something like two seconds . postdoc c: and , that would be , one speaker . the discourse the people who look at turn taking often do use i was interested that you chose , the that you use cuz that 's a unit that would be more consistent with sociolinguistics . phd g: we chose , half a second because if you go much larger , you have a y , your statement about how much overlap there is becomes less , precise , because you include more of actual pause time into what you consider overlap speech . so , it 's a compromise , phd b: . , i also used something around zero point five seconds for the speech - nonspeech detector phd g: and it 's also based liz suggested that value based on the distribution of pause times that you see in switchboard and other corpora . postdoc c: in any case , this , meeting that i hand - adjusted two of them i mentioned before , and i sent email , so phd g: maybe using those as references because , what you would do is you would play with different parameters . and to get an object you need an objective measure of how closely you can align the models to the actual speech . and that 's where your data would be very important to have . so , i will phd b: and hopefully the new meetings which will start from the channelized version will have better time boundaries and alignments . postdoc c: but i like this idea of , for our purposes for the ibm preparation , n having these joined together , and it makes a lot of sense . and in terms of transcription , it would be easy to do it that way . the way that they have with the longer units , not having to fuss with adding these units at this time . phd b: whi - which could have one drawback . if there is a backchannel in between those three things , the n the backchannel will occur at the end of those three . and and in the previous version where in the n which is used now , there , the backchannel would be in - between there somewhere , so . postdoc c: that 's right , but , thi this brings me to the other f stage of this which i discussed with you earlier today , which is the second stage is , w what to do in terms of the transcribers adjustment of these data . i discussed this with you too . , the tr so the idea initially was , we would get , for the new meetings , so the e edu meetings , that thilo ha has now presegmented all of them for us , on a channel by channel basis . so , i ' ve assigned them to our transcribers and , so far i ' ve discussed it with one , with and i had a about an hour discussion with her about this yesterday , we went through edu - one , at some extent . and it occurred to me that what we have in this a format is you could consider it as a staggered mixed file , we had some discussion over the weekend a about at this other meeting that we were all a at , about whether the tran the ibm transcribers should hear a single channel audio , or a mixed channel audio . and , in a way , by having this chunk and then the backchannel after it , it 's like a stagal staggered mixed channel . and , it occurred to me in my discussion with her yesterday that , the maximal gain , it 's from the ibm people , may be in long stretches of connected speech . so it 's a whole bunch of words which they can really do , because of the continuity within that person 's turn . so , what i ' m thinking , and it may be that not all meetings will be good for this , but what i ' m thinking is that in the edu meetings , they tend to be driven by a couple of dominant speakers . and , if the chunked files focused on the dominant speakers , then , when it got s patched together when it comes back from ibm , we can add the backchannels . it seems to me that , the backchannels per - se would n't be so hard , but then there 's this question of the time @ , marking , and whether the beeps would be y and i ' m not exactly how that would work with the backchannels . and certainly things that are intrusions of multiple words , taken out of context and displaced in time from where they occurred , that would be hard . so , m my thought is i i ' m having this transcriber go through the edu - one meeting , and indicate a start time { nonvocalsound } f for each dominant speaker , endpoi end time for each dominant speaker , and the idea that these units would be generated for the dominant speakers , and maybe not for the other channels . grad a: the only , disadvantage of that is , then it 's hard to use an automatic method to do that . the advantage is that it 's probably faster to do that than it is to use the automated method and correct it . so . postdoc c: i think , the original plan was that the transcriber would adjust the t the boundaries , and all that for all the channels but , that is so time - consuming , and since we have a bottleneck here , we want to get ibm things that are usable s as soon as possible , then this seemed to me it 'd be a way of gett to get them a flood of data , which would be useful when it comes back to us . and also , at the same time she when she goes through this , she 'll be if there 's anything that was encoded as a pause , but really has something transcribable in it , then she 's going to , make a mark w , so that bin would be marked as it as double dots and she 'll just add an s . and in the other case , if it 's marked as speech , and really there 's nothing transcribable in it , then she 's going to put a s dash , and i 'll go through and it and , with a substitution command , get it so that it 's clear that those are the other category . i 'll just , recode them . but , the transcribable events that , i ' m considering in this , continue to be laugh , as speech , and cough and things like that , so i ' m not stripping out anything , just , being very lenient in what 's considered speech . ? phd d: jane ? in terms of the this new procedure you 're suggesting , u what is the phd d: so i ' m a little confused , because how do we know where to put beeps ? is it i d y is it postdoc c: so what it involves is really a s , the original pr procedure , but only applied to , a certain strategically chosen s aspect of the data . postdoc c: and then , the transcriber , instead of going painstakingly through all the channels and moving the boundaries around , and deciding if it 's speech or not , but not transcribing anything . ok ? instead of doing that , which was our original plan , the tra they focus on the dominant speaker postdoc c: so what they do is they identify who 's the di dominant speaker , and when the speaker starts . postdoc c: so we 're it 's based on your se presegmentation , that 's the basic thing . phd b: and you just use the s the segments of the dominant speaker then ? for for sending to ibm or ? phd d: so , now jane , my question is when they 're all done adjusting the w time boundaries for the dominant speaker , have they then also erased the time boundaries for the other ones ? postdoc c: that 's that 's why she 's notating the start and end points of the dominant speakers . so , on a , so i in edu - one , i as far as i listened to it , you start off with a s section by jerry . so jerry starts at minute so - and - so , and goes until minute so - and - so . and then mark paskin comes in . and he starts at minute such - and - such , and goes on till minute so - and - so . and then meanwhile , she 's listening to both of these guys ' channels , determining if there 're any cases of misclassification of speech as nothing , and nothing as speech , and a and adding a tag if that happens . postdoc c: but , i wanted to say , his segmentation is so good , that , the part that i listened to with her yesterday did n't need any adjustments of the bins . postdoc c: so far we have n't . so this is not gon na be a major part of the process , at least not in not on ones that really phd d: so if you do n't have to adjust the bins , why not just do what it for all the channels ? postdoc c: there 's the question o of whether she i it 's a question of how much time we want our transcriber to invest here when she 's gon na have to invest that when it comes back from ibm anyway . so if it 's only inserting " - " s here and there , then , would n't that be something that would be just as efficient to do at this end , instead of having it go through i b m , then be patched together , then be double checked here . phd b: but then we could just use the output of the detector , and do the beeping on it , and send it to i b professor f: that sounds like a good idea since as you say you have to do with the other end anyway . postdoc c: now , you were saying that they differ in how they work depending on channel s sys systems and . phd b: so we should perhaps just select meetings on which the speech - nonspeech detection works , phd b: and just use , those meetings to send to ibm and , do the other ones . professor f: what 's the problem the l i forget . is the problem the lapel , or phd b: , it really depends . , my impression is that it 's better for meetings with fewer speakers , and it 's better for meetings where nobody is breathing . phd d: so the one suggestion is we run thilo 's thing and then we have somebody go and adjust all the time boundaries and we send it to ibm . the other one is we just run his thing and send it to ibm . there 's a another possibility if we find that there are some problems , and that is if we go ahead and we just run his , and we generate the beeps file , then we have somebody listen beeps file . phd d: and they listen to each section and say " yes , no " whether that section is phd d: i intelligible or not . and it just , there 's a little interface which will for all the " yes " - es it then that will be the final beep file . phd d: it would n't be that much fun for a transcriber to sit there , hear it , beep , yes or no . phd d: but there 's no adjusting . and that 's what 's slow . there 's no adjusting of time boundaries . professor f: , what 's the worst that happens ? do the transcribers as long as th on the other end they can say there 's something conventions so that they say " ? " phd d: we can just catch it at the catch everything at this side . maybe that 's the best way to go , just postdoc c: so i was gon na say , edu - one is good enough , maybe we could include it in this set of , this we send . grad a: , we wo n't know until we generate a bunch of beep files automatically , listen to them and see how bad they are . phd d: because there 's a part of the process of the beep file which requires knowing the normalization coefficients . grad a: that 's not hard to do . just it takes , it just takes five minutes rather than , taking a second . phd d: except i do n't think that the c the instructions for doing that was in that directory , right ? i did n't see where you had gener professor f: but but i have another suggestion on that , which is , since , really what this is , is trying to in the large , send the right thing to them and there is gon na be this post - processing step , why do n't we check through a bunch of things by sampling it ? right ? in other words , rather than , , saying we 're gon na listen to everything grad a: i did n't mean listen to everything , i meant , just see if they 're any good . professor f: so y you do a bunch of meetings , you listen to a little bit here and there , if it sounds like it 's almost always right and there 's not any big problem you send it to them . professor f: and , then they 'll send us back what we w what they send back to us , professor f: and we 'll fix things up and some meetings will cost more time to fix up than others . grad a: we should and we should just double - check with brian on a few simple conventions on how they should mark things . phd d: when they when there 's either no speech in there , they do n't understand , things like that . grad a: , cuz @ what i had originally said to brian was they 'll have to mark , when they ca n't distinguish between the foreground and background , because that was gon na be the most prevalent . but if we send them without editing , then we 're also gon na hafta have m , notations for words that are cut off , and other sorts of , acoustic problems . phd d: and they may just at what those cut - off words are , but w we 're gon na adjust everything when we come back grad a: but what we would like them to do is be conservative so that they should only write down the transcript if they 're . and otherwise they should mark it so that we can check . professor f: i can i maybe have an order of it 's probably in your paper that i have n't looked at lately , professor f: , an order of magnitude notion of how on a good meeting , how often , do you get segments that come in the middle of words and , and in a bad meeting how often ? professor f: so i wanted to get some sense of what almost always meant . and then , in a bad meeting , or p some meetings where he said he 's had some problems , what does that mean ? so does one of the does it mean one percent and ten percent ? or does it mean five percent and fifty percent ? phd b: the problem is that , nnn , the numbers ian gave in the paper is just , some frame error rate . so that 's not really what will be effective for the transcribers , is they have to , in they have to insure that 's a real s spurt . and but , the numbers let me think . so the speech the amount of speech that is missed by the detector , for a good meeting , i th is around or under one percent , i would say . but there can be for but there can be more there 's more amount speech , more amount of , the detector says there is speech , but there is none . so that can be a lot when it 's really a breathy channel . professor f: but that 's less of a problem . they 'll just listen . it 's just wasted time . and th and that 's for a good meeting . now what about in a meeting that you said we ' ve you ' ve had some more trouble with ? phd b: i ca n't really hhh , tsk . i do n't have really representative numbers , . that 's really i did this on four meetings and only five minutes of every meet of these meetings so , it 's not that representative , but , it 's perhaps , fff . , it 's perhaps then it 's perhaps five percent of something , which s the frames speech frames which are missed , but , i ca n't really tell . professor f: so i so i sometime , we might wanna go back and look at it more in terms of how many times is there a spurt that 's , interrupted ? something like that ? postdoc c: the other problem is , that when it d i on the breathy ones , where you get breathing , inti indicated as speech . and i we could just indicate to the transcribers not to encode that if they we could still do the beep file . professor f: again that is probably less of a problem because if you 're if there 's if if a word is split , then they might have to listen to it a few times to really understand that they ca n't quite get it . whereas if they listen { nonvocalsound } to it and there 's do n't hear any speech they 'd probably just listen to it once . so there 'd you 'd think there 'd be a factor of three or four in , cost function , between them . phd b: so but that 's n that really does n't happen very often that a word is cut in the middle . that 's that 's really not normal . professor f: so so what you 're saying is that nearly always what happens when there 's a problem is that , there 's some , nonspeech that is b interpreted as speech . professor f: because that does n't do any harm . , if they hear , a dog bark and they say what was the word , they phd b: i also thought of there are really some channels where it is almost , only bre breathing in it . and to re - run 's phd b: , . i ' ve got a p - a method with loops into the cross - correlation with the pzm mike , and then to reject everything which seems to be breath . so , i could run this on those breathy channels , and perhaps throw out professor f: but i th again , that would be good , and what that 'll do is just cut the time a little further . but none of this is that really needs somebody doing these , explicit markings . postdoc c: , i 'd be delighted with that , i was very impressed with the result . professor f: , cuz the other thing that was concerning me about it was that it seemed specialized to the edu meeting , and that then when you get a meeting like this , and you have a b a bunch of different dominant speakers professor f: , how are you gon na handle it . whereas this sounds like a more general solution postdoc c: i was just trying to find a way cuz i do n't think the staggered mixed channel is awfully good as a way of handling overlaps . but phd d: good . that that really simplifies thing then . and we can just , get the meeting , process it , put the beeps file , send it off to ibm . ? with very little work on our side . professor f: i would just use some samples , make you do n't send them three hours of " bzzz " . phd d: that would be very good . and then we can that 'll oughta be a good way to get the pipeline going . phd b: and there 's one point which i , which i r we covered when i r listened to one of the edu meetings , phd b: and i the speech - nonspeech detector just assigns randomly the speech to one of the channels , - i have n't - i did n't think of s of this before , phd b: but , sometimes the laptop is in the background and some somebody is talking , and , that 's really a little bit confusing , phd b: that 's a second question , what will different transcribers do with the laptop sound ? postdoc c: so my standard approach has been if it 's not someone close - miked , then , they do n't end up on one of the close - miked channels . they end up on a different channel . and we have any number of channels available , it 's an infinite number of channels . so just put them on some other channel . phd b: when thi when this is sent to the i m - , i b m transcribers , i if they can tell that 's really postdoc c: , they have a convention , in their own procedures , which is for a background sound . grad a: right , but , in general i do n't think we want them transcribing the background , cuz that would be too much work . for it because in the overlap sections , then they 'll phd d: i do n't think jane 's saying they 're gon na transcribe it , but they 'll just mark it as being there 's some background there , grad a: but that 's gon na be all over the place . how w how will they tell the difference between that background and the dormal normal background of two people talking at once ? grad a: because otherwise it 's gon na be too much work for them to mark it . they 'll be marking it all over the place . grad a: but how are they gon na tell bet the difference between that and two people just talking at the same time ? professor f: , is n't there a category something like , " sounds for someone for whom there is no i close mike " ? grad a: but how do we d how do we do that for the i b m folks ? how can they tell that ? grad a: yes , that 's my opinion as . so we do n't do anything for it with it . grad a: and they 'll just mark it however they mark it , and we 'll correct it when it comes back . postdoc c: , as it comes back , we have a when we can use the channelized interface for encoding it , then it 'll be easy for us to handle . but but if out of context , they ca n't tell if it 's a channeled speak , a close - miked speaker or not , then that would be confusing to them . i , i it does n't i do n't either way would be fine with me , i do n't really care . postdoc c: i have o i have one question . do you think we should send the that whole meeting to them and not worry about pre - processing it ? postdoc c: or , what is we should leave the part with the audio in the , beep file that we send to ibm for that one , or should we start after the that part of the meeting is over in what we send . postdoc c: w if we have speech from the laptop should we just , excise that from what we send to ibm , or should we i give it to them and let them do with it what they can ? phd d: we should just it 's gon na be too much work if we hafta worry about that . professor f: cuz , i would n't do n't think we would mind having that transcribed , if they did it . grad a: as i say , we 'll just have to listen to it and see how horrible it is . sample it , rather . grad a: - , and they 're very it 's very audible ? on the close - talking channels ? professor f: that 's a fast way to do it . one , two , three , go ! postdoc c: it 's interesting if there 're any more errors in these , than we had the first set .
the main topics of the agenda were a paper submitted to eurospeech and the organising of the recording transcriptions to be done by ibm. the results presented in the former show a significant percentage of overlapping speech even without counting in backchanneling. additionally , the high error rate in the recognition of such overlapping speech by the sri recogniser was minimised simply by changing the scoring method used. finally , a strong correlation between pauses and interruptions was confirmed. all these measurements were based on the sample of available transcripts. other features , like prosody , will be studied in the near future. an ftp directory containing such experimental data is being set up for the benefit of other researchers. regarding the transcriptions to be carried out by ibm , the discussion mainly concerned the format of the recordings that should be sent to them. suggestions included sending only the channels with the dominant speakers for transcription , but it was finally agreed on sending the original files with minimal modifications , as there will be extensive in-house post-processing. within this discussion , the rationale behind the coding of the time bins according to the flow of discourse was also explained. the files made available in the ftp directory will be the original ones ( before down-sampling ) , as these seem to be wanted by other parties. moreover , as files may have been modified through different processing , tests will be carried out in order to ensure the generation of beep files in a consistent way. also towards this goal , some of the time bins will need to be merged. on the other hand , the two meetings where time bins have been hand-coded in detail will be used to fine-tune the forced alignments. recordings will be sent to ibm for transcription. before that , the files will be automatically pre-segmented into speech/non-speech bins and the beeps will be inserted. in order to make things easier for the transcribers , breathy channels , which are erroneously marked as speech , will be re-classified correctly with other methods. all this pre-processing will have to be evaluated first by checking a sample of the output files. other issues , like whether and how synthesised speech off a laptop needs be transcribed , will be resolved during the in-house post-processing of the transcriptions. there is a slight worry about the acceptance of the paper submitted to eurospeech as the deadline was exceeded. as to the content of the paper , the overlap statistics have not been normalised against the number of participants in the conversation , although the dependency is probably going to be a weak one. additionally , the correlation between pauses in speech and interruptions does not provide a cause-and-effect link for these phenomena. the preparation of files for transcription by ibm is facing some minor difficulties , as some features ( hand-coded time boundaries , multiplicity of channels etc ) may complicate the generation of beep files. besides this , the automatic pre-segmentation has been deemed to be good , but there are still no specific measurements to verify this. the pre-segmentation tool also classifies synthesised speech used in a recording as "normal speech" and assigns a random channel to it. the transcribers at ibm may not be able to differentiate between the two. a paper has been submitted to eurospeech , which also includes a section on new corpora. the statistics in the paper are based on the transcripts of two meeting and two telephone conversation corpora. in the first two , the overlapped words vary between 9% and 18%. the telephone conversation results were in-between and very similar to each other. on the other hand , the automatic recognition errors affected by overlaps were reduced dramatically by focusing on regions with the foreground speech. furthermore , it was shown that after spurts , backchannels , disfluencies and discourse markers , the likelihood of interruption by other speakers was much higher. files with the recordings , as well as some experimental data will be available for other researchers in an ftp directory that is being set up. parts of the recordings will have to be beeped out by a script that has already been developed. finally , edu meetings already recorded have now been pre-segmented and assigned to the transcribers at icsi.
###dialogue: phd d: there 's gon na be some zeros from this morning 's meeting because i noticed that barry , maybe you turned your mike off before the digits were , was it during digits ? , so it does n't matter . phd b: so it 's not that bad if it 's at the end , but it 's in the beginning , it 's bad . grad a: , you wanna keep them on so you get good noise floors , through the whole meeting . professor f: i probably just should have left it on . i did have to run , but grad e: where like you just do n't like if you if it starts catching zeros , like in the driver in the card , or somewhere in the hardware where if you start seeing zeros on w across one channel , you just add some random , @ noise floor like a small noise floor . grad a: certainly we could do that , but i do n't think that 's a good idea . we can do that in post - processing if the application needs it . professor f: , i u i actually what the default is anymore as to how we 're using the front - end but for when we use the icsi front - end , professor f: but , there is an o an option in rasta , which , in when i first put it in , back in the days when i actually wrote things , i did actually put in a random bit or so that was in it , professor f: but then i realized that putting in a random bit was equivalent to adding flat spectrum , professor f: and it was a lot faster to just add a constant to the spectrum . so then started doing that professor f: instead of calling " rand " , so . so it d it does that . gee ! here we all are ! grad a: so the only agenda items were jane was jane wanted to talk about some of the ibm transcription process . grad a: i condensed the three things you said into that . and then just i only have like , this afternoon and maybe tomorrow morning to get anything done before i go to japan for ten days . so if there 's anything that n , desperately needs to be done , you should let me know now . phd g: liz had sent them a note saying " could we have another " i , " three days " , and they said yes . grad a: , dave gelbart sent me email , he sent it to you too , that , there 's a special topic , section in si in eurospeech on new , corp corpors corpora . and it 's not due until like may fifteenth . grad a: i s forwarded it to jane as being the most relevant person . so , it was highly relevant grad a: i 'll help , but i ca n't , really do , most of it , phd g: there were some interesting results in this paper , though . that morgan , accounted for fifty - six percent of the robustness meetings in terms of number of words . postdoc c: because is it partly , c correctly identified words ? or is it or just overall volume ? professor f: i get these aarp things , but i ' m not se really senior yet , professor f: but , other than that delightful result , what was the rest of the paper about ? phd g: three kinds of results , if you will . , the one was that the just the amount of overlap phd g: s in terms of number of words and also we computed something called a " spurt " , which is essentially a stretch of speech with , no pauses exceeding five hundred milliseconds . , and we computed how many overlapped i spurts there were and how many overlapped words there were . , for four different corpora , the meeting recorder meetings , the robustness meetings switchboard and callhome , and , found and compared the numbers . and found that the , as you might expect the meeting recorder meetings had the most overlap , but next were switchboard and callhome , which both had roughly the same , almost identical , and the robustness meetings were had the least , so one unexpected result there is that two - party telephone conversations have about the same amount of overlap , phd g: in gen order of magnitude - wise as , as face - to - face meetings with multiple phd g: also , i in the levinson , the pragmatics book , in , textbook , there 's i found this great quote where he says , how people it talks about how people are so good at turn taking , and so they 're so good that generally , u the overlapped speech does not is less than five percent . phd g: , in real conversations , everyday conversations . it 's what these conversation analysts have been studying for years and years there . postdoc c: , no , it does n't necessarily go against what he said , cuz he said " generally speaking " . in order to go against that a claim you 'd have to big canvassing . postdoc c: it just says that it 's a bi bell curve , and that , you have something that has a range , in your sampling . phd g: so there are slight there are differences in how you measure it , but still it 's , the difference between that number and what we have in meetings , which is more like , close to in meetings like these , close to twenty percent . phd g: robustness meeting ? it was about half of the r so , in terms of number of words , it 's like seventeen or eigh eighteen percent for the meeting recorder meetings and about half that for , the robustness . grad a: but i if that 's really a fair way of comparing between , multi - party , conversations and two - party conversations . i . phd g: , we did n't get to look at that , but this obvious thing to see if there 's a dependence on the number of participants . grad a: i bet there 's a weak dependence . i ' m it 's not a real strong one . right ? because you grad a: you have a lot of two - party , subsets within the meeting . regardless it 's an interesting result regardless . phd g: and and then and we also d computed this both with and without backchannels , so you might think that backchannels have a special status because they 're essentially just grad a: so , did we all said " - " and nodded at the same time , phd g: r right . but , even if you take out all the backchannels so you treat backchannels l as nonspeech , as pauses , phd g: you still have significant overlap . , it goes down from maybe for switchboard it goes down from i f fourteen percent of the words to maybe i , eleven percent it 's not a dramatic change , so it 's anyway , that was that was one set of results , and then the second one was just the we had in the hlt paper on how overlaps effect the recognition performance . phd g: and we rescored things , a little bit more carefully . we also fixed the transcripts in numerous ways . , but mostly we added one number , which was what if you , score ignoring all so so the conjecture from the hlt results was that most of the added recognition error is from insertions due to background speech . so , we scored all the recognition results , in such a way that the phd g: ok , so if you have the foreground speaker speaking here , and then there 's some background speech , may be overlapping it somehow , and this is the time bin that we used , then you 're gon na get insertion errors here and here . right ? so we scored everything , and i must say the nist scoring tools are pretty for this , where you just ignore everything outside of the , region that was deemed to be foreground speech . and where that was we had to use the t forced alignment , results from s for that 's somewhat that 's somewhat subject to error , but still we , don did some ha hand - checking and we think that based on that , we think that the results are , valid , although , some error is gon na be in there . but what we found is after we take out these regions so we only score the regions that were certified as foreground speech , the recognition error went down to almost , the level of the non - overlapped speech . so that means that even if you do have background speech , if you can somehow separate out or find where it is , the recognizer does a good job , grad a: , i that does n't surprise me , because , with the close - talking mikes , the signal will be so much stronger . phd g: , we do , vtl vocal tract length normalization , w and we , make all the features have zero mean and unit variance . phd g: over the but . , now we did n't re - align the recognizer for this . we just took the old so this is actually a sub - optimal way of doing it , right ? so we took the old recognition output and we just scored it differently . so the recognizer did n't have the benefit of knowing where the foreground speech a start phd g: i should bring the table with results . maybe we can look at it monday . professor f: i would presume that you still would have somewhat higher error with the lapel for insertions than professor f: cuz again , looking forward to the non - close miked case , that we s still professor f: i it 's the high signal - to - noise ratio here that helps you . phd g: u s right . so that was number that was the second set of , the second section . and then , the third thing was , we looked at , what we call " interrupts " , although that 's that may be a misnomer , but we looked at cases where , so we used the punctuation from the original transcripts and we inferred the beginnings and ends of sentences . postdoc c: di - did you use upper - lower case also , or not ? u upper lower case or no ? phd g: ? we only used , periods , question marks and exclamation . and we know that there 's th that 's not a very g , we miss a lot of them , phd g: no commas . no . and then we looked at locations where , if you have overlapping speech and someone else starts a sentence , where do these where do other people start their turns not turns really , but , sentences , so we only looked at cases where there was a foreground speaker and then at the to at the so the foreground speaker started into their sentence and then someone else started later . phd g: , so that such that there was overlap between the two sentences . so , the question was how can we what can we say about the places where the second or actually , several second speakers , start their " interrupts " , as we call them . phd g: so so we had u to for the purposes of this analysis , we tagged the word sequences , and we time - aligned them . and we considered it interrupt if it occurred in the middle of a word , we , considered that to be a interrupt as if it were at the beginning of the word . so that , if any part of the word was overlapped , it was considered an interrupted word . and then we looked at the locatio the , , the features that the tags because we had tagged these word strings , that occurred right before these , interrupt locations . phd g: and the tags we looked at are the spurt tag , which says or actually end of spurt . so whether there was a pause essentially here , because spurts are a defined as being , five hundred milliseconds or longer pauses , and then we had things like discourse markers , backchannels , disfluencies . , filled pauses so disfluen the d 's are for , the interruption points of a disfluency , so , where you hesitate , or where you start the repair there . , what else do we had . , repeated words is another of that disfluencies and . so we had both the beginnings and ends of these so , the end of a filled pause and the end of a discourse marker . and we just eyeballed we did n't really hand - tag all of these things . we just looked at the distribution of words , and so every " so " , and " ok " , and " - " were the were deemed to be backchannels and " so " and " right " , were not " right " . " right " is a backchannel . but so , we just based on the lexical , identity of the words , we tagged them as one of these things . and the d the interruption points we got from the original transcripts . and then we looked at the disti so we looked at the distribution of these different kinds of tags , overall , and particularly at the interruption points . and , we found that there is a marked difference that after so at the end after a discourse marker or after backchannel or after filled pause , you 're much more likely to be interrupted than before . and also after spurt ends , which means in p inside pauses . so pauses are always an opportunity for so we have this little histogram which shows these distributions and , phd d: i wonder about the and effect there . in other words if you were n't going to pause you will because you 're g being interrupted . professor f: but he , he 's right , y maybe you were n't intending to pause , but you were intending to stop for fifty - seven milliseconds , but then chuck came in and so you paused for a second phd g: , and that was it . and and we so we wrote this and then , we found we were at six pages , and then we started cutting furiously and threw out half of the material again , and played with the latex and phd g: the gap between the two columns is like ten millimeters , so i d shrunk it to eight millimeters and that helped some . and like that . phd d: was n't there was n't there some result , andreas maybe liz presented this at some conference a while ago about , backchannels and that they tend to happen when the pitch drops . you get a falling pitch . and so that 's when people tend to backchannel . - i do you rem phd g: now that we have the data and we have the alignment , this is purely based on the words phd d: it made me think about a little device that could be built to handle those people that call you on the phone and just like to talk and talk . and you just have this little detector that listens for these drops in pitch and gives them the backchannel . and so then you hook that to the phone and go off and do the do whatever you r wanna do , while that thing keeps them busy . phd g: there 's actually there 's this a former student of here from berkeley , nigel ward . do him ? he did a system , in he lives in japan now , and he did this backchanneling , automatic backchanneling system . it 's a very phd g: so , exactly what you describe , but for japanese . and it 's for japa - in japanese it 's really important that you backchannel . it 's really impolite if you do n't , so . professor f: actually for a lot of these people you could just backchannel continuously and it would be fine . grad a: there was there was a monty python sketch with that . where the barber who was afraid of scissors was playing a tape of clipping sounds , and saying " - " , how about them sports teams ? phd g: anyway . so the paper 's on - line y i cc ' ed a message to meeting recorder with the url so you can get it . phd g: , one more thing . so i ' m actually about to send brian kingbury an email saying where he can find the s the m the material he wanted for the s for the speech recognition experiment , but i have n't sent it out yet because actually my desktop locked up , like i ca n't type anything . b so if there 's any suggestions you have for that i was just gon na send him the phd g: he prefe he said he would prefer ftp and also , the other person that wants it there is one person at sri who wants to look at the , the data we have so far , and so i figured that ftp is the best approach . so what i did is i @ i made a n new directory after chuck said that would c that was gon na be a good thing . , so it 's " ftp pub phd g: and then under there actually and this directory , is not readable . it 's only , accessible . so , in other words , to access anything under there , you have to be told what the name is . so that 's a g quick and dirty way of doing access control . , and the directory for this i call it i " asr zero point one " because it 's meant for recognition . phd g: and then in there i have a file that lists all the other files , so that someone can get that file and then know the file names and therefore download them . if you the file names you ca n't phd g: dash . so all i was gon na do there was stick the transcripts after we the way that we munged them for scoring , because that 's what he cares about , and also and then the waveforms that don segmented . , just tar them all up f w for each meeting i tar them all into one tar file and g - zip them and stick them there . phd d: so we could point mari to this also for her march o - one request ? phd d: she was saying that it would be if we had they had a or was she talking she was saying it would be if they had the same set , so that when they did experiments they could compare . phd d: we need to give brian the beeps file , so i was gon na probably put it grad e: so . , so either we should regenerate the original versions , or , we should just make a note of it . grad e: , that 's the first meeting i cut both versions . just to check which w if there is a significant difference . phd g: and so i ok so but for the other meetings it 's the downsampled version that you have . phd g: that 's th important to know , we should probably give them the non - downsampled versions . grad e: , because liz decided to go ahead with the downsampled versions cuz we can there was no s like , r significant difference . grad e: it does take up less disk space , and it did even better than the original versions , which , is just , probably random . grad a: ok , we 're losing , don and andreas at three - thirty , right ? ok . professor f: so , that 's why it was good to have andreas , say these things so , we should probably talk about the ibm transcription process that postdoc c: so , that adam created , a b a script to generate the beep file ? to then create something to send to ibm . and , you should probably talk about that . but but you were gon na to use the originally transcribed file because i tightened the time bins and that 's also the one that they had already in trying to debug the first stage of this . and , my understanding was that , i have n't listened to it yet , but it sounded very good and i understand that you guys were going to have a meeting today , before this meeting . grad a: it was just to talk about how to generate it . , just so that while i ' m gone , you can regenerate it if you decide to do it a different way . so , chuck and thilo should , now more or less know how to generate the file grad a: and , the other thing chuck pointed out is that , since this one is hand - marked , there are discourse boundaries . so so when one person is speaking , there 's breaks . whereas thilo 's wo n't have that . so what we 're probably gon na do is just write a script , that if two , chunks are very close to each other on the same channel we 'll just merge them . grad a: so , and that will get around the problem of , the , " one word beep , one word beep " . phd d: , after our meeting , this morning thilo came in and said that , there could be other differences between the already transcribed meeting with the beeps in it and one that has just r been run through his process . phd d: so tomorrow , when we go to make the , chunked file for ibm , we 're going to actually compare the two . so he 's gon na run his process on that same meeting , phd d: and then we 're gon na do the beep - ify on both , and listen to them and see if we notice any real differences . postdoc c: ok , now one thing that prevented us from apply you from applying the training so that is the training meeting . phd d: w and we know that . wel - we just wanna if there 're any major differences between doing it on the hand phd g: so this training meeting , w un is that some data where we have very , accurate time marks ? for postdoc c: i went back and hand - marked the ba the bins , i ment i mentioned that last week . phd d: but the but there 's , but there is this one issue with them in that there 're there are time boundaries in there that occur in the middle of speech . phd d: so like when we went t to when i was listening to the original file that adam had , it 's like you hear a word then you hear a beep and then you hear the continuation of what is the same sentence . grad a: that 's not gon na be true of the foreground speaker . that 'll only be if it 's the background speaker . phd d: right . so you 'll have a chunk of , channel a which starts at zero and ends at ten , and then the same channel starting at eleven , ending at fifteen , and then again , starting at sixteen , ending at twenty . so that 's three chunks where actually we w can just make one chunk out of that which is a , zero , twenty . phd d: so if you were to use these , you have to be careful not to pull out these individual phd g: it w what i would i was interested in is having a se having time marks for the beginnings and ends of speech by each speaker . phd g: it i do n't care that , there 's actually abutting segments that we have to join together . that 's fine . phd g: but what we do care about is that the beginnings and ends are actually close to the speech inside of that postdoc c: they were , reasonably tight , but not excruciatingly tight . that would ' ve taken more time . wanted to get it so tha so that if you have like " in a swimming in a big bin , then it 's phd g: actually i it 's f that 's fine because we do n't want to th that 's perfectly fine . it 's good . you always want to have a little bit of pause or nonspeech around the speech , say for recognition purposes . but just u w get an i d wanted to have an idea of the of how much extra you allowed so that interpret the numbers if i compared that with a forced alignment segmentation . postdoc c: i ca n't answer that , but my main goal was , in these areas where you have a three - way overlap and one of the overlaps involves " , and it 's swimming in this huge bin , i wanted to get it so that it was clo more closely localized . phd g: but are we talking about , i , a tenth of a second ? a ? how how much extra would you allow at most postdoc c: i wanted to i wanted it to be able to l he be heard normally , so that if you play back that bin and have it in the mode where it stops at the boundary , it sounds like a normal word . it does n't sound like the person i it sounds normal . it 's as if the person could ' ve stopped there . and it would n't have been an awkward place to stop . now sometimes , it 's these are involved in places where there was no time . and so , there would n't be a gap afterwards because some cases , there 're some people , who have very long segments of discourse where , they 'll breath and then i put a break . but other than that , it 's really pretty continuous and this includes things like going from one sentence into the u one utterance into the next , one sentence into the next , w without really stopping . i they , i in writing you have this two spaces and a big gap but but i some people are planning and , , a lot we always are planning what we 're going to say next . but , in which case , the gap between these two complete syntactic units , which n spoken things are not always complete syntactically , but it would be a shorter p shorter break than maybe you might like . but the goal there was to not have the text be so crudely parsed in a time bin . , because from a discourse m purpose it 's more useful to be able to see and also , from a speech recognition purpose my impression is that if you have too long a unit , it 's it does n't help you very much either , cuz of the memory . postdoc c: so , that means that the amount of time after something is variable depending partly on context , but my general goal when there was sufficient space , room , pause after it to have it be a natural feeling gap . which i c i what it would be quantified as . , wally chafe says that , in producing narratives , the spurts that people use tend to be , that the what would be a pause might be something like two seconds . postdoc c: and , that would be , one speaker . the discourse the people who look at turn taking often do use i was interested that you chose , the that you use cuz that 's a unit that would be more consistent with sociolinguistics . phd g: we chose , half a second because if you go much larger , you have a y , your statement about how much overlap there is becomes less , precise , because you include more of actual pause time into what you consider overlap speech . so , it 's a compromise , phd b: . , i also used something around zero point five seconds for the speech - nonspeech detector phd g: and it 's also based liz suggested that value based on the distribution of pause times that you see in switchboard and other corpora . postdoc c: in any case , this , meeting that i hand - adjusted two of them i mentioned before , and i sent email , so phd g: maybe using those as references because , what you would do is you would play with different parameters . and to get an object you need an objective measure of how closely you can align the models to the actual speech . and that 's where your data would be very important to have . so , i will phd b: and hopefully the new meetings which will start from the channelized version will have better time boundaries and alignments . postdoc c: but i like this idea of , for our purposes for the ibm preparation , n having these joined together , and it makes a lot of sense . and in terms of transcription , it would be easy to do it that way . the way that they have with the longer units , not having to fuss with adding these units at this time . phd b: whi - which could have one drawback . if there is a backchannel in between those three things , the n the backchannel will occur at the end of those three . and and in the previous version where in the n which is used now , there , the backchannel would be in - between there somewhere , so . postdoc c: that 's right , but , thi this brings me to the other f stage of this which i discussed with you earlier today , which is the second stage is , w what to do in terms of the transcribers adjustment of these data . i discussed this with you too . , the tr so the idea initially was , we would get , for the new meetings , so the e edu meetings , that thilo ha has now presegmented all of them for us , on a channel by channel basis . so , i ' ve assigned them to our transcribers and , so far i ' ve discussed it with one , with and i had a about an hour discussion with her about this yesterday , we went through edu - one , at some extent . and it occurred to me that what we have in this a format is you could consider it as a staggered mixed file , we had some discussion over the weekend a about at this other meeting that we were all a at , about whether the tran the ibm transcribers should hear a single channel audio , or a mixed channel audio . and , in a way , by having this chunk and then the backchannel after it , it 's like a stagal staggered mixed channel . and , it occurred to me in my discussion with her yesterday that , the maximal gain , it 's from the ibm people , may be in long stretches of connected speech . so it 's a whole bunch of words which they can really do , because of the continuity within that person 's turn . so , what i ' m thinking , and it may be that not all meetings will be good for this , but what i ' m thinking is that in the edu meetings , they tend to be driven by a couple of dominant speakers . and , if the chunked files focused on the dominant speakers , then , when it got s patched together when it comes back from ibm , we can add the backchannels . it seems to me that , the backchannels per - se would n't be so hard , but then there 's this question of the time @ , marking , and whether the beeps would be y and i ' m not exactly how that would work with the backchannels . and certainly things that are intrusions of multiple words , taken out of context and displaced in time from where they occurred , that would be hard . so , m my thought is i i ' m having this transcriber go through the edu - one meeting , and indicate a start time { nonvocalsound } f for each dominant speaker , endpoi end time for each dominant speaker , and the idea that these units would be generated for the dominant speakers , and maybe not for the other channels . grad a: the only , disadvantage of that is , then it 's hard to use an automatic method to do that . the advantage is that it 's probably faster to do that than it is to use the automated method and correct it . so . postdoc c: i think , the original plan was that the transcriber would adjust the t the boundaries , and all that for all the channels but , that is so time - consuming , and since we have a bottleneck here , we want to get ibm things that are usable s as soon as possible , then this seemed to me it 'd be a way of gett to get them a flood of data , which would be useful when it comes back to us . and also , at the same time she when she goes through this , she 'll be if there 's anything that was encoded as a pause , but really has something transcribable in it , then she 's going to , make a mark w , so that bin would be marked as it as double dots and she 'll just add an s . and in the other case , if it 's marked as speech , and really there 's nothing transcribable in it , then she 's going to put a s dash , and i 'll go through and it and , with a substitution command , get it so that it 's clear that those are the other category . i 'll just , recode them . but , the transcribable events that , i ' m considering in this , continue to be laugh , as speech , and cough and things like that , so i ' m not stripping out anything , just , being very lenient in what 's considered speech . ? phd d: jane ? in terms of the this new procedure you 're suggesting , u what is the phd d: so i ' m a little confused , because how do we know where to put beeps ? is it i d y is it postdoc c: so what it involves is really a s , the original pr procedure , but only applied to , a certain strategically chosen s aspect of the data . postdoc c: and then , the transcriber , instead of going painstakingly through all the channels and moving the boundaries around , and deciding if it 's speech or not , but not transcribing anything . ok ? instead of doing that , which was our original plan , the tra they focus on the dominant speaker postdoc c: so what they do is they identify who 's the di dominant speaker , and when the speaker starts . postdoc c: so we 're it 's based on your se presegmentation , that 's the basic thing . phd b: and you just use the s the segments of the dominant speaker then ? for for sending to ibm or ? phd d: so , now jane , my question is when they 're all done adjusting the w time boundaries for the dominant speaker , have they then also erased the time boundaries for the other ones ? postdoc c: that 's that 's why she 's notating the start and end points of the dominant speakers . so , on a , so i in edu - one , i as far as i listened to it , you start off with a s section by jerry . so jerry starts at minute so - and - so , and goes until minute so - and - so . and then mark paskin comes in . and he starts at minute such - and - such , and goes on till minute so - and - so . and then meanwhile , she 's listening to both of these guys ' channels , determining if there 're any cases of misclassification of speech as nothing , and nothing as speech , and a and adding a tag if that happens . postdoc c: but , i wanted to say , his segmentation is so good , that , the part that i listened to with her yesterday did n't need any adjustments of the bins . postdoc c: so far we have n't . so this is not gon na be a major part of the process , at least not in not on ones that really phd d: so if you do n't have to adjust the bins , why not just do what it for all the channels ? postdoc c: there 's the question o of whether she i it 's a question of how much time we want our transcriber to invest here when she 's gon na have to invest that when it comes back from ibm anyway . so if it 's only inserting " - " s here and there , then , would n't that be something that would be just as efficient to do at this end , instead of having it go through i b m , then be patched together , then be double checked here . phd b: but then we could just use the output of the detector , and do the beeping on it , and send it to i b professor f: that sounds like a good idea since as you say you have to do with the other end anyway . postdoc c: now , you were saying that they differ in how they work depending on channel s sys systems and . phd b: so we should perhaps just select meetings on which the speech - nonspeech detection works , phd b: and just use , those meetings to send to ibm and , do the other ones . professor f: what 's the problem the l i forget . is the problem the lapel , or phd b: , it really depends . , my impression is that it 's better for meetings with fewer speakers , and it 's better for meetings where nobody is breathing . phd d: so the one suggestion is we run thilo 's thing and then we have somebody go and adjust all the time boundaries and we send it to ibm . the other one is we just run his thing and send it to ibm . there 's a another possibility if we find that there are some problems , and that is if we go ahead and we just run his , and we generate the beeps file , then we have somebody listen beeps file . phd d: and they listen to each section and say " yes , no " whether that section is phd d: i intelligible or not . and it just , there 's a little interface which will for all the " yes " - es it then that will be the final beep file . phd d: it would n't be that much fun for a transcriber to sit there , hear it , beep , yes or no . phd d: but there 's no adjusting . and that 's what 's slow . there 's no adjusting of time boundaries . professor f: , what 's the worst that happens ? do the transcribers as long as th on the other end they can say there 's something conventions so that they say " ? " phd d: we can just catch it at the catch everything at this side . maybe that 's the best way to go , just postdoc c: so i was gon na say , edu - one is good enough , maybe we could include it in this set of , this we send . grad a: , we wo n't know until we generate a bunch of beep files automatically , listen to them and see how bad they are . phd d: because there 's a part of the process of the beep file which requires knowing the normalization coefficients . grad a: that 's not hard to do . just it takes , it just takes five minutes rather than , taking a second . phd d: except i do n't think that the c the instructions for doing that was in that directory , right ? i did n't see where you had gener professor f: but but i have another suggestion on that , which is , since , really what this is , is trying to in the large , send the right thing to them and there is gon na be this post - processing step , why do n't we check through a bunch of things by sampling it ? right ? in other words , rather than , , saying we 're gon na listen to everything grad a: i did n't mean listen to everything , i meant , just see if they 're any good . professor f: so y you do a bunch of meetings , you listen to a little bit here and there , if it sounds like it 's almost always right and there 's not any big problem you send it to them . professor f: and , then they 'll send us back what we w what they send back to us , professor f: and we 'll fix things up and some meetings will cost more time to fix up than others . grad a: we should and we should just double - check with brian on a few simple conventions on how they should mark things . phd d: when they when there 's either no speech in there , they do n't understand , things like that . grad a: , cuz @ what i had originally said to brian was they 'll have to mark , when they ca n't distinguish between the foreground and background , because that was gon na be the most prevalent . but if we send them without editing , then we 're also gon na hafta have m , notations for words that are cut off , and other sorts of , acoustic problems . phd d: and they may just at what those cut - off words are , but w we 're gon na adjust everything when we come back grad a: but what we would like them to do is be conservative so that they should only write down the transcript if they 're . and otherwise they should mark it so that we can check . professor f: i can i maybe have an order of it 's probably in your paper that i have n't looked at lately , professor f: , an order of magnitude notion of how on a good meeting , how often , do you get segments that come in the middle of words and , and in a bad meeting how often ? professor f: so i wanted to get some sense of what almost always meant . and then , in a bad meeting , or p some meetings where he said he 's had some problems , what does that mean ? so does one of the does it mean one percent and ten percent ? or does it mean five percent and fifty percent ? phd b: the problem is that , nnn , the numbers ian gave in the paper is just , some frame error rate . so that 's not really what will be effective for the transcribers , is they have to , in they have to insure that 's a real s spurt . and but , the numbers let me think . so the speech the amount of speech that is missed by the detector , for a good meeting , i th is around or under one percent , i would say . but there can be for but there can be more there 's more amount speech , more amount of , the detector says there is speech , but there is none . so that can be a lot when it 's really a breathy channel . professor f: but that 's less of a problem . they 'll just listen . it 's just wasted time . and th and that 's for a good meeting . now what about in a meeting that you said we ' ve you ' ve had some more trouble with ? phd b: i ca n't really hhh , tsk . i do n't have really representative numbers , . that 's really i did this on four meetings and only five minutes of every meet of these meetings so , it 's not that representative , but , it 's perhaps , fff . , it 's perhaps then it 's perhaps five percent of something , which s the frames speech frames which are missed , but , i ca n't really tell . professor f: so i so i sometime , we might wanna go back and look at it more in terms of how many times is there a spurt that 's , interrupted ? something like that ? postdoc c: the other problem is , that when it d i on the breathy ones , where you get breathing , inti indicated as speech . and i we could just indicate to the transcribers not to encode that if they we could still do the beep file . professor f: again that is probably less of a problem because if you 're if there 's if if a word is split , then they might have to listen to it a few times to really understand that they ca n't quite get it . whereas if they listen { nonvocalsound } to it and there 's do n't hear any speech they 'd probably just listen to it once . so there 'd you 'd think there 'd be a factor of three or four in , cost function , between them . phd b: so but that 's n that really does n't happen very often that a word is cut in the middle . that 's that 's really not normal . professor f: so so what you 're saying is that nearly always what happens when there 's a problem is that , there 's some , nonspeech that is b interpreted as speech . professor f: because that does n't do any harm . , if they hear , a dog bark and they say what was the word , they phd b: i also thought of there are really some channels where it is almost , only bre breathing in it . and to re - run 's phd b: , . i ' ve got a p - a method with loops into the cross - correlation with the pzm mike , and then to reject everything which seems to be breath . so , i could run this on those breathy channels , and perhaps throw out professor f: but i th again , that would be good , and what that 'll do is just cut the time a little further . but none of this is that really needs somebody doing these , explicit markings . postdoc c: , i 'd be delighted with that , i was very impressed with the result . professor f: , cuz the other thing that was concerning me about it was that it seemed specialized to the edu meeting , and that then when you get a meeting like this , and you have a b a bunch of different dominant speakers professor f: , how are you gon na handle it . whereas this sounds like a more general solution postdoc c: i was just trying to find a way cuz i do n't think the staggered mixed channel is awfully good as a way of handling overlaps . but phd d: good . that that really simplifies thing then . and we can just , get the meeting , process it , put the beeps file , send it off to ibm . ? with very little work on our side . professor f: i would just use some samples , make you do n't send them three hours of " bzzz " . phd d: that would be very good . and then we can that 'll oughta be a good way to get the pipeline going . phd b: and there 's one point which i , which i r we covered when i r listened to one of the edu meetings , phd b: and i the speech - nonspeech detector just assigns randomly the speech to one of the channels , - i have n't - i did n't think of s of this before , phd b: but , sometimes the laptop is in the background and some somebody is talking , and , that 's really a little bit confusing , phd b: that 's a second question , what will different transcribers do with the laptop sound ? postdoc c: so my standard approach has been if it 's not someone close - miked , then , they do n't end up on one of the close - miked channels . they end up on a different channel . and we have any number of channels available , it 's an infinite number of channels . so just put them on some other channel . phd b: when thi when this is sent to the i m - , i b m transcribers , i if they can tell that 's really postdoc c: , they have a convention , in their own procedures , which is for a background sound . grad a: right , but , in general i do n't think we want them transcribing the background , cuz that would be too much work . for it because in the overlap sections , then they 'll phd d: i do n't think jane 's saying they 're gon na transcribe it , but they 'll just mark it as being there 's some background there , grad a: but that 's gon na be all over the place . how w how will they tell the difference between that background and the dormal normal background of two people talking at once ? grad a: because otherwise it 's gon na be too much work for them to mark it . they 'll be marking it all over the place . grad a: but how are they gon na tell bet the difference between that and two people just talking at the same time ? professor f: , is n't there a category something like , " sounds for someone for whom there is no i close mike " ? grad a: but how do we d how do we do that for the i b m folks ? how can they tell that ? grad a: yes , that 's my opinion as . so we do n't do anything for it with it . grad a: and they 'll just mark it however they mark it , and we 'll correct it when it comes back . postdoc c: , as it comes back , we have a when we can use the channelized interface for encoding it , then it 'll be easy for us to handle . but but if out of context , they ca n't tell if it 's a channeled speak , a close - miked speaker or not , then that would be confusing to them . i , i it does n't i do n't either way would be fine with me , i do n't really care . postdoc c: i have o i have one question . do you think we should send the that whole meeting to them and not worry about pre - processing it ? postdoc c: or , what is we should leave the part with the audio in the , beep file that we send to ibm for that one , or should we start after the that part of the meeting is over in what we send . postdoc c: w if we have speech from the laptop should we just , excise that from what we send to ibm , or should we i give it to them and let them do with it what they can ? phd d: we should just it 's gon na be too much work if we hafta worry about that . professor f: cuz , i would n't do n't think we would mind having that transcribed , if they did it . grad a: as i say , we 'll just have to listen to it and see how horrible it is . sample it , rather . grad a: - , and they 're very it 's very audible ? on the close - talking channels ? professor f: that 's a fast way to do it . one , two , three , go ! postdoc c: it 's interesting if there 're any more errors in these , than we had the first set .
###summary: the main topics of the agenda were a paper submitted to eurospeech and the organising of the recording transcriptions to be done by ibm. the results presented in the former show a significant percentage of overlapping speech even without counting in backchanneling. additionally , the high error rate in the recognition of such overlapping speech by the sri recogniser was minimised simply by changing the scoring method used. finally , a strong correlation between pauses and interruptions was confirmed. all these measurements were based on the sample of available transcripts. other features , like prosody , will be studied in the near future. an ftp directory containing such experimental data is being set up for the benefit of other researchers. regarding the transcriptions to be carried out by ibm , the discussion mainly concerned the format of the recordings that should be sent to them. suggestions included sending only the channels with the dominant speakers for transcription , but it was finally agreed on sending the original files with minimal modifications , as there will be extensive in-house post-processing. within this discussion , the rationale behind the coding of the time bins according to the flow of discourse was also explained. the files made available in the ftp directory will be the original ones ( before down-sampling ) , as these seem to be wanted by other parties. moreover , as files may have been modified through different processing , tests will be carried out in order to ensure the generation of beep files in a consistent way. also towards this goal , some of the time bins will need to be merged. on the other hand , the two meetings where time bins have been hand-coded in detail will be used to fine-tune the forced alignments. recordings will be sent to ibm for transcription. before that , the files will be automatically pre-segmented into speech/non-speech bins and the beeps will be inserted. in order to make things easier for the transcribers , breathy channels , which are erroneously marked as speech , will be re-classified correctly with other methods. all this pre-processing will have to be evaluated first by checking a sample of the output files. other issues , like whether and how synthesised speech off a laptop needs be transcribed , will be resolved during the in-house post-processing of the transcriptions. there is a slight worry about the acceptance of the paper submitted to eurospeech as the deadline was exceeded. as to the content of the paper , the overlap statistics have not been normalised against the number of participants in the conversation , although the dependency is probably going to be a weak one. additionally , the correlation between pauses in speech and interruptions does not provide a cause-and-effect link for these phenomena. the preparation of files for transcription by ibm is facing some minor difficulties , as some features ( hand-coded time boundaries , multiplicity of channels etc ) may complicate the generation of beep files. besides this , the automatic pre-segmentation has been deemed to be good , but there are still no specific measurements to verify this. the pre-segmentation tool also classifies synthesised speech used in a recording as "normal speech" and assigns a random channel to it. the transcribers at ibm may not be able to differentiate between the two. a paper has been submitted to eurospeech , which also includes a section on new corpora. the statistics in the paper are based on the transcripts of two meeting and two telephone conversation corpora. in the first two , the overlapped words vary between 9% and 18%. the telephone conversation results were in-between and very similar to each other. on the other hand , the automatic recognition errors affected by overlaps were reduced dramatically by focusing on regions with the foreground speech. furthermore , it was shown that after spurts , backchannels , disfluencies and discourse markers , the likelihood of interruption by other speakers was much higher. files with the recordings , as well as some experimental data will be available for other researchers in an ftp directory that is being set up. parts of the recordings will have to be beeped out by a script that has already been developed. finally , edu meetings already recorded have now been pre-segmented and assigned to the transcribers at icsi.