input: What did the team discuss about evaluation?
  PhD E: OK.
  Professor B: OK, so {pause} We {disfmarker} we had a meeting with, uh {disfmarker} with Hynek, um, in {disfmarker} in which, uh, uh, Sunil and Stephane, uh {vocalsound} summarized where they were and {disfmarker} and, uh, talked about where we were gonna go. So that {disfmarker} that happened sort of mid - week. Uh.
  PhD E: D did {disfmarker} did you guys get your code pushed together?
  PhD D: Oh, yeah. Yeah. It's {disfmarker} it's {disfmarker} it's {disfmarker} it was updated yesterday,
  PhD E: Cool.
  PhD D: right?
  PhD A: Yeah.
  PhD D: Yeah.
  PhD A: You probably received the mail.
  PhD E: Oh, right, I saw {disfmarker} I saw the note.
  PhD A: Yeah.
  PhD E: Mm - hmm.
  Professor B: What was the update?
  PhD A: What was the update? So there is th then {disfmarker} the {disfmarker} all the new features that go in.
  Professor B: Yeah.
  PhD A: The, um, noise suppression, the re - synthesis of speech after suppression. These are the {disfmarker}
  PhD E: Is the, um {disfmarker} the CVS mechanism working {pause} well?
  PhD A: Yeah.
  PhD E: Are {disfmarker} are people, uh, up at OGI grabbing code uh, via that?
  PhD D: Uh, I don't think {disfmarker} I don't think {disfmarker}
  PhD E: Or {disfmarker}?
  PhD A: I don't know if they use it, but.
  PhD D: Yeah, I I don't think anybody up there is like {pause} working on it right now.
  PhD E: Uh - huh. Mmm.
  Professor B: I think it more likely that what it means is that when Sunil is up there {vocalsound} he will grab it.
  PhD D: Yeah. Yeah. So right now nobody's working on Aurora there.
  PhD E: Yeah.
  Professor B: They're {disfmarker} Yeah. They're working on a different task.
  PhD E: I see. I see.
  PhD D: Yeah.
  PhD E: OK.
  Professor B: But what'll happen is {disfmarker} is he'll go back up there and, uh, Pratibha will come back from {disfmarker} from, uh, the east coast. Uh.
  PhD E: Mm - hmm.
  Professor B: And, uh {disfmarker} and {disfmarker} and I guess actually, uh, after Eurospeech for a little bit, uh, he'll go up there too. So, actually everybody {vocalsound} who's working on it {comment} will be up there for at least a little while. So they'll remotely access it {vocalsound} from there.
  PhD E: So has {disfmarker} Has anybody tried remotely accessing the CVS using, uh, uh, SSH?
  Professor B: Yeah.
  PhD A: Um, I don't know if Hari did that or {disfmarker} You d
  PhD D: I {comment} can actually do it today. I mean, I can just log into {disfmarker}
  PhD E: Have you tried it yet?
  PhD D: No, I didn't. So I I'll try it today.
  PhD E: OK.
  Professor B: Good idea.
  PhD A: Actually I {disfmarker} I tried wh while {disfmarker} when I installed the {pause} repository, I tried from Belgium.
  Professor B: Yeah.
  PhD D: Yeah.
  PhD A: I logged in there and I tried {pause} to import {disfmarker}
  PhD E: Yeah? It worked good?
  PhD A: Yeah, it works.
  PhD E: Oh, good!
  PhD A: But it's {disfmarker} So, right now it's the mechanism with SSH.
  PhD D: Oh.
  PhD E: Great!
  PhD A: I don't {pause} s I didn't set up {disfmarker} You can also set up a CVS server {pause} on a new port. It's like well {pause} uh, a main server, or d You can do a CVS server.
  PhD E: Yeah. Right. Then that's using the CVS password mechanism and all that,
  PhD A: But. Yeah, right.
  PhD E: right?
  PhD A: But I didn't do that because I was not sure about {pause} security problems. I {disfmarker} I would have to {disfmarker}
  PhD E: So w when you came in from Belgian {disfmarker} {comment} Belgium, using SSH, uh, was it asking you for your own {pause} password into ICSI? So if yo you can only do that if you have an account at ICSI?
  PhD A: Right. Yeah.
  PhD E: OK.
  PhD A: Yeah.
  PhD E: Cuz there is an {disfmarker} a way to set up anonymous CVS right?
  PhD A: Yeah, you ha in this way you ca you have to set up a CVS server but then, yeah, you can access it.
  PhD E: So that {disfmarker} Oh, OK.
  PhD A: you {disfmarker} you can set up priorities.
  PhD E: So the anonymous mechanism {disfmarker}
  PhD A: You can access them and mostly if you {disfmarker} if y the set the server is set up like this.
  PhD E: OK. Because a lot of the open source stuff works with anonymous CVS and I'm just wondering {disfmarker} Uh, I mean, for our transcripts we may want to do that.
  PhD A: Mm - hmm.
  Professor B: Yeah.
  PhD E: Uh.
  Professor B: Yeah, for this stuff I don't think we're {pause} quite up to that. I mean, we're still so much in development.
  PhD E: Mm - hmm. Yeah,
  Professor B: We want to have just the insiders.
  PhD E: yeah, yeah. Oh, I wasn't suggesting for this. I'm {pause} thinking of the Meeting Recorder {comment} stuff
  Professor B: Yeah.
  PhD E: but. Yeah. OK. Cool.
  Professor B: Yeah. So, uh {disfmarker}
  PhD E: What's new?
  Professor B: Well, I mean, I think maybe the thing to me might be {disfmarker} I me I'm sure you've just been working on {disfmarker} on, uh, details of that since the meeting, right? And so {disfmarker}
  PhD A: Mmm, since the meeting, well, I {disfmarker} I've been {disfmarker} I've been train training a new VAD and a new {pause} feature net.
  Professor B: That was {disfmarker} that was Tuesday. OK.
  PhD A: So they should be ready. Um.
  Professor B: But I guess maybe the thing {disfmarker} since you weren't {disfmarker} yo you guys weren't at that {disfmarker} that meeting, might be just {disfmarker} just to, um, sort of recap, uh, the {disfmarker} the conclusions of the meeting.
  PhD A: Mm - hmm.
  PhD E: Oh, great.
  Professor B: So.
  PhD E: You're talking about the meeting with Hynek?
  Professor B: Yeah. Cuz that was sort of, uh {disfmarker} we {disfmarker} we'd sort of been working up to that, that {disfmarker} that, uh, he would come here this week and {disfmarker} and we would sort of {disfmarker}
  PhD E: Uh - huh.
  Professor B: Since he's going out of town like now, and I'm going out town in a couple weeks, uh, and time is marching, sort of, given all the mu many wonderful things we could be working on, what {disfmarker} what will we actually focus on?
  PhD E: Mm - hmm.
  Professor B: And, uh {disfmarker} and what do we freeze? And, you know, what do we {disfmarker}? So, um. I mean, this {pause} software that these guys created was certainly a {disfmarker} a key part. So then there's something central and there aren't at least a bunch of different versions going off in {disfmarker} in ways that {pause} differ {pause} trivially. Uh, um, and, um,
  PhD E: Yeah. That's {disfmarker} that's nice.
  Professor B: and then within that, I guess the idea was to freeze a certain set of options for now, to run it, uh, a particular way, and decide on what things are gonna be experimented with, as opposed to just experimenting with everything. So keep a certain set of things constant. So, um. Uh, maybe describe roughly what {disfmarker} what we are keeping constant for now, or {disfmarker}?
  PhD A: Yeah. Well. So we've been working like six weeks on {disfmarker} on the noise compensation and we end up with something that seems reasonable. Um.
  PhD E: Are you gonna use {disfmarker} which of the two techniques?
  PhD A: So finally it's {disfmarker} it's, um, Wiener filtering on FFT bins. And it uses, uh, two steps, smoothing of the transfer function, the first step, that's along time, which use recursion. And {vocalsound} after this step there is a further smoothing along frequency, which use a sliding window of twenty FFT bins. Mmm. And, uh {disfmarker}
  PhD E: So this is on the {disfmarker} uh, before any mel scaling has been done?
  PhD A: Yeah, yeah.
  PhD E: This is {disfmarker}
  PhD A: It was {disfmarker}
  Professor B: This {disfmarker} this smoothing is done on the estimate, um, of what you're going to subtract? Or on the thing that has already had something subtracted?
  PhD A: Yeah. Uh, {vocalsound} it's on the transfer function. So {disfmarker}
  Professor B: Oh, it's on the transfer function for the Wiener filter.
  PhD A: Yeah.
  Professor B: Yeah, OK.
  PhD A: Yeah, so basically we tried {vocalsound} different configuration within this idea. We tried u u applying this on mel bands, having spectral subtraction instead of wiener filtering. Um. Well, finally we end up with {pause} this configuration that works, uh, quite well. So we are going to fix this for the moment and work on the other aspects of {vocalsound} the whole system.
  PhD E: Mm - hmm.
  PhD A: So {disfmarker}
  Professor B: Actually, let me int eh, Dave isn't here to talk about it, but let me just interject. This module, in principle, i I mean, you would know whether it's {vocalsound} true in fact, is somewhat independent from the rest of it. I mean, because you {disfmarker} you re - synthesize speech, right?
  PhD A: Mm - hmm.
  Professor B: So, um. Uh, well you don't {disfmarker} I guess you don't re - synthesize speech, but you could {disfmarker}
  PhD A: We {disfmarker} we do not fo
  Professor B: Uh, but you could.
  PhD A: Well {disfmarker} well, we do, but we don't {disfmarker} don't re - synthesize. In {disfmarker} in the program we don't re - synthesize and then re - analyze once again. We just use the clean FFT bins.
  Professor B: But you have a re - synthesized thing that you {disfmarker} that's an {disfmarker} an option here.
  PhD A: This is an option that {disfmarker} then you can {disfmarker} Yeah.
  Professor B: Yeah, I gu I guess my point is that, um, i in some of the work he's doing in reverberation, one of the things that we're finding is that, uh, it's {disfmarker} it's {disfmarker} for the {disfmarker} for an artificial situation, we can just deal with the reverberation and his techniques work really well. But for the real situation uh, problem is, is that you don't just have reverberation, you have reverberation in noise. And if you don't include that in the model, it doesn't work very well. So in fact it might be a very nice thing to do, to just take the noise removal part of it and put that in front of what he's looking at. And, uh, generate new files or whatever, and {disfmarker} and, uh, uh {disfmarker} and then do the reverberation part.
  PhD A: Mm - hmm.
  Professor B: So it's {disfmarker}
  PhD D: Mmm.
  Professor B: Anyway.
  PhD E: So Dave hasn't {pause} tried that yet?
  Professor B: No, no. He's {disfmarker} I mean, e
  PhD E: I guess he's busy with {disfmarker}
  Professor B: Yeah, prelims, right.
  Grad C: Pre - prelim hell.
  Professor B: Yeah.
  PhD E: Yeah.
  Professor B: So.
  PhD E: Yeah.
  Professor B: Uh, but {disfmarker} but, you know, that'll {disfmarker} uh, it's clear that we, uh {disfmarker} we are not {disfmarker} with the real case that we're looking at, we can't just look at reverberation in isolation because the interaction between that and noise is {disfmarker} is considerable. And that's I mean, in the past we've looked at, uh, and this is hard enough, the interaction between channel effects and {disfmarker} and, uh {disfmarker} and additive noise, uh, so convolutional effects and {disfmarker} and additive effects. And that's hard enough. I mean, I don't think we really {disfmarker} I mean, we're trying to deal with that. In a sense that's what we're trying to deal with in this Aurora task. And we have, uh, the, uh, uh, LDA stuff that in principle is doing something about convolutional effects. And we have the noise suppression that's doing something about noise. Uh, even that's hard enough. And {disfmarker} and the on - line normalization as well, in that s category. i i There's all these interactions between these two and that's part of why these guys had to work so hard on {disfmarker} on juggling everything around. But now when you throw in the reverberation, it's even worse, because not only do you have these effects, but you also have some long time effects. And, um, so Dave has something which, uh, is doing some nice things under some conditions with {disfmarker} with long time effects but when it's {disfmarker} when there's noise there too, it's {disfmarker} it's {disfmarker} it's pretty hard. So we have to start {disfmarker} Since any {disfmarker} almost any real situation is gonna have {disfmarker} uh, where you have the microphone distant, is going to have both things, we {disfmarker} we actually have to think about both at the same time.
  PhD E: Hmm.
  Professor B: So, um {disfmarker} So there's this noise suppression thing, which is sort of worked out and then, uh, maybe you should just continue telling what {disfmarker} what else is in the {disfmarker} the form we have.
  PhD A: Yeah, well, {vocalsound} the, um, the other parts of the system are the {disfmarker} the blocks that were already present before and that we did not modify a lot.
  Professor B: So that's {disfmarker} again, that {disfmarker} that's the Wiener filtering, followed by, uh {disfmarker} uh, that's done at the FFT level. Then {disfmarker}
  PhD A: Yeah, th then the mel filter bank,
  Professor B: Mm - hmm.
  PhD A: then the log operation,
  Professor B: Mm - hmm.
  PhD A: Mmm.
  Professor B: The {disfmarker} the {disfmarker} the filtering is done in the frequency domain?
  PhD A: Yeah.
  Professor B: Yeah, OK. And then the mel and then the log, and then the
  PhD A: Then the LDA filter,
  Professor B: LDA filter.
  PhD A: mmm, then the downsampling,
  Professor B: And then uh downsample,
  PhD A: DCT,
  Professor B: DCT,
  PhD A: then, um, on - line normalization,
  Professor B: on - line norm,
  PhD A: followed by {pause} upsampling. Then finally, we compute delta and we put the neural network also.
  Professor B: Right, and then in parallel with {disfmarker} an {disfmarker} a neural net. And then following neural net, some {disfmarker} probably some orthogonalization.
  PhD A: Yeah.
  Professor B: Uh {disfmarker} Um.
  PhD A: And finally frame dropping, which um, {vocalsound} would be a neural network also, used for estimated silence probabilities. And the input of this neural network would be somewhere between log {pause} mel bands or one of the earlier stages of the processing.
  Professor B: Mm - hmm. So that's sort of {disfmarker} most of this stuff is {disfmarker} yeah, is operating parallel with this other stuff.
  PhD A: Mm - hmm.
  Professor B: Yeah. So the things that we, um, uh, I guess we sort of {disfmarker} uh, There's {disfmarker} there's some, uh, neat ideas for {vocalsound} V A So, I mean, in {disfmarker} I think there's sort of like {disfmarker} There's a bunch of tuning things to improve stuff. There's questions about {pause} various places where there's an exponent, if it's the right exponent, or {pause} ways that we're estimating noise, that we can improve estimating noise. And there's gonna be a host of those. But structurally it seemed like the things {disfmarker} the main things that {disfmarker} that we brought up that, uh, are {disfmarker} are gonna need to get worked on seriously are, uh, uh, a {disfmarker} {vocalsound} a significantly better VAD, uh, putting the neural net on, um, which, you know, we haven't been doing anything with, the, uh, neural net at the end there, and, uh, the, uh, {vocalsound} opening up the second front. Uh.
  PhD E: The other half of the channel?
  Professor B: Yeah, yeah, I mean, cuz we {disfmarker} we have {disfmarker} we have, uh, uh, half the {disfmarker} the, uh, data rate that they allow.
  PhD E: That what you mean?
  Professor B: And, uh, so the initial thing which came from, uh, the meeting that we had down south was, uh, that, um, we'll initially just put in a mel spectrum as the second one. It's, you know, {pause} cheap, easy. Uh. There's a question about exactly how we do it. We probably will go to something better later, but the initial thing is that cepstra and spectra behave differently,
  PhD E: Mm - hmm.
  Professor B: so. Um, {comment} I think Tony Robinson used to do {disfmarker} I was saying this before. I think he used to do mel, uh, spectra and mel cepstra. He used them as alternate features. Put them together.
  PhD E: Hmm.
  Professor B: Uh.
  PhD E: So if you took the system the way it is now, the way it's fro you're gonna freeze it, and it ran it on the last evaluation, where it would it be?
  PhD A: Mm - hmm. It, uh,
  PhD E: In terms of ranking?
  PhD A: Ri - right now it's second.
  PhD D: Second.
  PhD A: Um.
  PhD E: Mm - hmm.
  Professor B: Although you {disfmarker} you know, you haven't tested it actually on the German and Danish, have you?
  PhD A: No, we didn't. No, um.
  Professor B: Yeah.
  PhD E: So on the ones that you did test it on it would have been second?
  Professor B: Yeah. Would it {disfmarker} I mean {disfmarker} But {disfmarker} When you're saying second, you're comparing to the numbers that the, uh {disfmarker} that the best system before got on, uh {disfmarker} also without German and Danish?
  PhD A: Yeah, yeah.
  Professor B: Yeah, OK.
  PhD D: And th the ranking actually didn't change after the German and Danish. So, yeah.
  Professor B: Well ranking didn't before, but I'm just asking where this is to where theirs was without the German and Danish,
  PhD A: Yeah.
  PhD D: Yeah.
  PhD A: Mmm.
  PhD D: Yeah.
  Professor B: right?
  PhD D: Yeah, yeah.
  Professor B: So.
  PhD E: Where {disfmarker} where {disfmarker} where were we actually on the last test?
  Professor B: Oh, we were also esp essentially second, although there were {disfmarker} there were {disfmarker} I mean, we had a couple systems and they had a couple systems. And so, I guess by that {pause} we were third, but I mean, there were two systems that were pretty close, that came from the same place.
  PhD E: Uh - huh. I see. OK.
  Professor B: Uh, so institutionally we were {disfmarker} {vocalsound} we were second, with, uh, the third {disfmarker} third system.
  PhD E: We're {disfmarker} so this second that you're saying now is system - wide second?
  Professor B: See {disfmarker} Uh, no I think it's also institutional, isn't it?
  PhD E: Still institutionally second?
  Professor B: Right? I mean, I think both of their systems probably {disfmarker}
  PhD A: Uh, we are between their two systems. So
  Professor B: Oh, are we?
  PhD A: I {disfmarker} It is a triumph.
  PhD D: Yeah.
  Professor B: Is it?
  PhD D: Their {disfmarker} their first system is fifty - four point something. And, uh, we are fifty - three point something.
  PhD A: But everything is {pause} within the range of one {disfmarker} one percent.
  PhD D: And their second system is also fifty - three point something. Yeah, one percent.
  Professor B: Yeah, so {disfmarker} so basically they're all {disfmarker} they're all pretty close.
  PhD E: Oh, wow!
  PhD A: So.
  PhD E: That's very close.
  PhD D: Yeah.
  PhD E: Yeah.
  Professor B: And {disfmarker} and, {vocalsound} um, you know, in some sense we're all doing fairly similar things. Uh, I mean, one could argue about the LDA and so forth but I {disfmarker} I think, you know, in a lot of ways we're doing very similar things. But what {disfmarker} what {disfmarker}
  PhD E: So how did they fill up this {disfmarker} all these {disfmarker} these bits? I mean, if we're u
  Professor B: Um, why are we using half? Well, so you could {disfmarker} you c
  PhD E: Yeah. Or how are they using more than half, I guess maybe is what I {disfmarker}
  Professor B: Yeah, so I {disfmarker} I think {disfmarker} uh, you guys are closer to it than me, so correct me if I'm wrong, but I {disfmarker} I think that what's going on is that in {disfmarker} in both cases, some kind of normalization is done to deal with convola convolutional effects. Uh, they have some cepstral {pause} modification,
  PhD A: Mm - hmm.
  Professor B: right? In our case we have a couple things. We have the on - line normalization and then we have the LDA RASTA. And {pause} they seem to comple complement each other enough and be different enough that they both seem to help {disfmarker} help us. But in any event, they're both doing the same sort of thing. But there's one difference. The LDA RASTA, uh, throws away high modulation frequencies. And they're not doing that.
  PhD E: So th So {disfmarker}
  Professor B: So that if you throw away high modulation frequencies, then you can downsample.
  Grad C: Get down.
  PhD E: I see. I see.
  Professor B: So {disfmarker}
  PhD E: So what if you didn't {disfmarker} So do you explicitly downsample then? Do we explicitly downsample?
  Professor B: Yeah.
  PhD A: Yeah.
  PhD E: And what if we didn't do that? Would we get worse performance?
  PhD A: Um {pause} Yeah, not better, not worse.
  Professor B: I think it doesn't affect it, does it?
  PhD E: I see. OK.
  Professor B: Yeah. So I think the thing is, since we're not evidently throwing away useful information, let's try to put in some useful information.
  PhD E: Yeah. Yeah.
  Professor B: And, uh, so I {disfmarker} you know, we {disfmarker} we've found in a lot of ways for quite a while that having a second stream uh, helps a lot. So that's {disfmarker} that's put in, and you know, it may even end up with mel spectrum even though I'm saying I think we could do much better, just because it's simple.
  PhD E: Mm - hmm.
  Professor B: Um. And you know, in the long run having something everybody will look at and say," oh, yeah, I understand" , is {disfmarker} is very helpful.
  PhD E: So you would {disfmarker} you're {disfmarker} You're thinking to put the, uh, mel spectrum in before any of the noise removal stuff? or after?
  Professor B: Well, that's a question. I mean, we were talking about that. It looks like it'd be straightforward to {disfmarker} to, uh, remove the noise, um, and, uh,
  PhD E: Cuz that happens before the mel conversion, right?
  Professor B: Yeah. So, I mean, to do it after the mel conversion {disfmarker} uh, after the noise removal, after the mel conversion. There's even a question in my mind anyhow of whether th you should take the log or not. Uh. I sort of think you should, but I don't know.
  PhD A: What about norm normalizing also?
  Professor B: Right. Uh. Well, but normalizing spectra instead of cepstra?
  PhD A: Yeah.
  Professor B: Yeah, probably. Some kind would be good. You know? I would think.
  PhD D: Well, it {disfmarker} it {disfmarker} it {disfmarker} it {disfmarker} so it actually makes it dependent on the overall energy of the {disfmarker} uh, the frame.
  Professor B: If you do or don't normalize?
  PhD D: If yo if you don't normalize and {disfmarker} if {disfmarker} if you don't normalize.
  Professor B: Right. Yes, so I mean, one would think that you would want to normalize. But I {disfmarker} I {disfmarker} w w My thought is, uh, particularly if you take the log, try it. And then if {disfmarker} if normalization helps, then y you have something to compare against, and say," OK, this much effect" {disfmarker} I mean, you don't want to change six things and then see what happens. You want to change them one at a time.
  PhD D: Mm - hmm.
  Professor B: So adding this other stream in, that's simple in some way. And then {pause} saying, oh {disfmarker} uh {disfmarker} particularly because we've found in the past there's all these {disfmarker} these {disfmarker} these different results you get with slight modifications of how you do normalization. Normalization's a very tricky, sensitive thing and {pause} you learn a lot. So, I would think you would wanna {pause} have some baseline that says," OK, we don't normalize, this is what we get" , when we do this normalization, when we do that normalization. But {disfmarker} but the other question is {disfmarker} So I think ultimately we'll wind up doing some normalization. I agree.
  PhD E: So this second stream, will it add latency to the system
  Professor B: No, it's in parallel.
  PhD E: or {disfmarker}?
  Grad C: Para
  Professor B: We're not talking about computation time here.
  PhD E: S
  Professor B: We're ta I think we're pretty far out.
  PhD E: Yeah.
  Professor B: So it's just in terms of what data it's depending on. It's depending on the same data as the other.
  PhD E: Same data.
  Professor B: So it's in parallel.
  PhD E: OK.
  Professor B: Uh - huh.
  Grad C: So with this, uh, new stream would you train up a VAD on both {disfmarker} both features, somehow?
  PhD D: No, I guess the VAD has its own set of features.
  Grad C: OK. that's {disfmarker}
  PhD D: I mean, which could be this {disfmarker} one of these streams, or it can be something derived from {pause} these streams.
  Professor B: Yeah.
  Grad C: OK.
  PhD A: And there is also the idea of using TRAPS, maybe, for the VAD, which, um {disfmarker}
  PhD D: Yeah, that's also {disfmarker}
  PhD A: Well, Pratibha apparently showed, when, she was at IBM, that it's a good idea. So.
  Grad C: Would {disfmarker} would that fit on the handset, or {disfmarker}? Oh!
  PhD A: I have no idea.
  Grad C: OK.
  PhD D: Well, it has t I mean the {disfmarker} th
  PhD A: It would have to fit but {disfmarker} Yeah.
  PhD D: Yeah, if it has to fit the delays and all this stuff.
  Professor B: Well, there's the delays and the storage,
  Grad C: OK.
  Professor B: yeah. But I don't think the storage is so big for that.
  Grad C: Right.
  PhD D: Yeah.
  Professor B: I think th the biggest we've run into for storage is the neural net. Right?
  PhD D: Yeah.
  Professor B: Yeah. Um. And so I guess the issue there is, are we {disfmarker} are we using neural - net - based TRAPS, and {disfmarker} and how big are they? So that'll {disfmarker} that'll be, you know, an issue.
  Grad C: Oh, right.
  Professor B: Maybe they can be little ones.
  Grad C: Yeah. Cuz sh Right.
  Professor B: Mini - TRAPS.
  Grad C: Cuz she also does the, uh {disfmarker} the correlation - based, uh, TRAPS, with without the neural net, just looking at the correlation between {disfmarker}
  Professor B: Right. And maybe for VAD they would be OK. Yeah. Yeah.
  Grad C: Yeah.
  Professor B: That's true.
  PhD D: Yeah.
  Professor B: Or a simple neural net, right? I mean, the thing is, if you're doing correlation, you're just doing a simple {disfmarker} uh, uh {disfmarker} uh, dot product, you know, with some weights which you happened to learn from this {disfmarker} learn from the data.
  Grad C: Mm - hmm.
  Professor B: And so, uh, putting a nonlinearity on it is, {pause} you know, not that big a deal. It certainly doesn't take much space.
  Grad C: Mm - hmm. Right.
  Professor B: So, uh, the question is, how complex a function do you need? Do you need to have an added layer or something? In which case, uh, potentially, you know, it could be big. So.
  Grad C: Mm - hmm.
  Professor B: So, uh, uh {disfmarker} So what's next? Maybe s s remind us.
  PhD E: So the meeting with Hynek that you guys just had was to decide exactly what you were gonna freeze in this system? Is that {disfmarker}? Or was there {disfmarker}? Were you talking about what t new stuff, or {disfmarker}?
  Professor B: What to freeze and then what to do after we froze.
  PhD E: Mmm.
  Professor B: Yeah. And like I was saying, I think the {disfmarker} you know, the basic directions are, uh, uh {disfmarker} I mean, there's lots of little things, such as improve the noise estimator but the bigger things are adding on the neural net and, uh, the second stream. And then, uh, improving the VAD. Uh. So.
  PhD D: So, I'll, um {disfmarker} I'll actually {disfmarker} after the meeting I'll add the second stream to the VAD and maybe I'll start with the feature net in that case. It's like, you're looking at the VAD, right?
  PhD A: Uh, yeah. I I've a new feature net ready also.
  PhD D: I'll {disfmarker} For the VAD?
  PhD A: No, uh. Well p two network, one VAD and one {pause} feature net.
  PhD D: Oh, you already have it?
  PhD A: Mm - hmm.
  PhD D: OK, so just figure how to take the features from the final {disfmarker}
  PhD A: Yeah.
  PhD D: OK.
  PhD A: Um. But, yeah, I think there are plenty of issues to work on for the feature net @ @.
  Grad C: Feature net.
  PhD E: What about the, um {disfmarker} uh, the new part of the evaluation, the, uh, Wall Street Journal part?
  Professor B: Right. Right. Um. Have you ever {disfmarker}? Very good question. Have you ever worked with the Mississippi State h uh, software?
  PhD A: Sorry.
  PhD E: No. Not yet.
  Professor B: Oh. Well you {disfmarker} you may be called upon to help, uh, uh, on account of, uh, all the work in this stuff here has been, uh, with small vocabulary.
  PhD E: OK. Mm - hmm. So what {disfmarker} how is the, uh, interaction supposed to happen? Uh, I remember the last time we talked about this, it was sort of up in the air whether they were going to be taking, uh, people's features and then running them or they were gonna give the system out or {disfmarker}
  PhD D: Yeah. Yeah.
  PhD E: Oh, so they're gonna just deliver a system basically.
  PhD D: Yeah, yeah.
  Professor B: Do we already have it?
  PhD D: Yeah, th I {disfmarker} I guess it's almost ready.
  PhD E: Uh - huh.
  PhD D: So {disfmarker} That's what {disfmarker} So they have released their, uh, document, describing the system.
  Professor B: Maybe you could, uh, point it {pause} at Chuck,
  PhD E: I see.
  Professor B: because, I mean {disfmarker}
  PhD D: Sure.
  PhD E: So we'll have to grab this over CVS or something?
  PhD D: It - no, it's just downloadable from their {disfmarker} from their web site.
  PhD E: Is that how they do it? OK.
  Professor B: Cuz one of the things that might be helpful, if you've {disfmarker} if you've got time in all of this is, is if {disfmarker} if these guys are really focusing on improving, uh, all the digit stuff, uh, maybe {disfmarker} and you got the front - end from them, maybe you could do the runs for the {disfmarker}
  PhD E: OK. Mm - hmm.
  Professor B: and {disfmarker} and, you know, iron out hassles that {disfmarker} that you have to, uh, tweak Joe about or whatever,
  PhD E: Sure.
  Professor B: because you're more experienced with running the large vocabulary stuff.
  PhD E: OK.
  Professor B: S
  PhD D: So I'll point you to the web site and the mails corresponding. So I
  PhD E: And it {disfmarker} but it's not ready yet, the system?
  PhD D: Uh, I {disfmarker} I think they are still, uh, tuning something on that. So they're like, d they're varying different parameters like the insertion penalty and other stuff, and then seeing what's the performance.
  PhD E: Are those going to be parameters that are frozen, nobody can change? Or {disfmarker}?
  PhD D: Uh, w I guess there is, uh, time during which people are gonna make suggestions.
  PhD E: Oh, but everybody's gonna have to use the same values.
  PhD D: After that.
  PhD E: Oh! Interesting.
  PhD D: Yeah, I guess.
  PhD E: OK.
  PhD D: So these sugges these {disfmarker} this, uh, period during which people are gonna make suggestions is to know whether it is actually biased towards any set of features or {disfmarker}
  Professor B: Yeah, so I th th certainly the thing that I would want to know about is whether we get really hurt, uh, on in insertion penalty, language model, scaling, sorts of things.
  PhD E: Using our features.
  Professor B: Yeah, yeah.
  PhD E: Yeah.
  Professor B: Uh, in which case, um, H Hari or Hynek will need to, you know, push the case {pause} more about {disfmarker} about this.
  PhD E: Mm - hmm.
  Professor B: Um.
  PhD E: And we may be able to revisit this idea about, you know, somehow modifying our features to work with {disfmarker}
  Professor B: Yes. In this case, that's right.
  PhD E: Yeah.
  Professor B: That's right. Um, some of that may be, uh, a last minute rush thing because if the {disfmarker} if our features are changing {disfmarker} Uh.
  PhD E: Yeah.
  Professor B: Uh. But, um. Yeah, the other thing is that even though it's months away, uh, it's starting to seem to me now like November fifteenth is right around the corner. And, um, if they haven't decided things like this, like what the parameters are gonna be for this, uh, when" deciding" is not just somebody deciding. I mean, in fact there should be some understanding behind the, uh, {vocalsound} deciding, which means some experiments and {disfmarker} and so forth. It {disfmarker} it {disfmarker} it seems pretty tight to me.
  PhD E: So wha what's the significance of November fifteenth?
  Professor B: That's when the evaluation is.
  PhD E: OK.
  Professor B: Yeah. So, yeah, so after {disfmarker} But, you know, they may even decide in the end to push it off. It wouldn't, you know, entirely surprise me. But, uh, due to other reasons, like some people are going away, I'm {disfmarker} I'm hoping it's not pushed off for {vocalsound} a l a long while. That would be, uh {disfmarker} put us in an awkward position. But {disfmarker} Anyway.
  PhD E: OK.
  Professor B: Great. Yeah, I think that'll be helpful. There's {disfmarker} there's not anybody OGI currently who's {disfmarker} who's, uh, working with this and {disfmarker} and
  PhD E: Is {disfmarker} is this part of the evaluation just a small part, or ho how important is this to the overall {disfmarker}?
  Professor B: I {disfmarker} I think it's {disfmarker} it's, um {disfmarker} it depends how badly {vocalsound} you do. I mean, I think that it {disfmarker} it is {disfmarker} Uh.
  PhD D: b
  PhD E: This is one of those things that will be debated afterwards?
  Professor B: Yeah. Well, I mean, it's {disfmarker} it's {disfmarker} Conceptually, it {disfmarker} my impression, again, you guys correct me if I'm wrong, but {pause} my impression is that, um, they want it as a double check. That you haven't come across {disfmarker} you haven't invented features which are actually gonna do badly for a {disfmarker} a significantly different task, particularly one with larger vocabulary. And, um, but it's not the main emphasis.
  PhD E: Mmm.
  Professor B: I mean, the truth is, most of the applications they're looking at are pretty small vocabulary.
  PhD E: Mmm.
  Professor B: So it's {disfmarker} it's a double check. So they'll probably assign it some sort of low weight.
  PhD E: Seems to me that if it's a double check, they should give you a one or a zero. Y you passed the threshold or you didn't pass the threshold, and they shouldn't even care about what the score is.
  Professor B: Yeah. But, I mean, we'll {disfmarker} we'll {disfmarker} we'll see what they come up with. Uh, but in {disfmarker} in the current thing, for instance, where you have this well - matched, moderately - matched, and {disfmarker} and mis highly - mismatched, uh, the emphasis is somewhat on the {disfmarker} on the well - matched, but it's only a {disfmarker} a marginal,
  PhD E: Yeah.
  Professor B: right? It's like forty, thirty - five, twenty - five, or something like that. So you still {disfmarker} if you were way, way off on the highly - mismatched, it would have a big effect.
  PhD D: Yeah.
  PhD E: Mm - hmm.
  Professor B: And, um, it wouldn't surprise me if they did something like that with this. So again, if you're {disfmarker} if you get {disfmarker} If it doesn't help you much, uh, for noisy versions of this {disfmarker} of large vocabulary data, then, uh, you know, it may not hurt you that much.
  PhD E: Oh.
  Professor B: But if it {disfmarker} if you don't {disfmarker} if it doesn't help you much at all, um, or to put it another way, if it helps some people a lot more than it helps other people, uh, if their strategies do, then {disfmarker}
  PhD E: Mm - hmm. So is this, uh {disfmarker}? Uh, Guenter was putting a bunch of Wall Street Journal data on our disks.
  Professor B: That's it.
  PhD E: So that's the data that we'll be running on?
  Professor B: Yeah.
  PhD E: I see. OK.
  Professor B: Yeah. So {pause} we have the data, just not the recognizer. OK.
  PhD E: So this test may take quite a while to run, then. May - judging by the amount of data that he was putting on.
  Professor B: Uh, well there's training and test, right?
  PhD E: I {disfmarker} I guess, I'm not sure.
  Professor B: No, I mean, if it's like the other things, there's {disfmarker} there's data for training the H M Ms and {disfmarker} and data for testing it.
  PhD E: I just {disfmarker}
  Professor B: So I wouldn't {disfmarker} So it {disfmarker} it's {disfmarker}
  PhD E: OK. So there's {disfmarker}
  Professor B: So, training the recognizer, but, um Um. But I think it's trained on clean and {disfmarker} Is it trained on clean and {disfmarker} and test on {disfmarker}?
  PhD D: The Wall Street?
  Professor B: Yeah.
  PhD A: Apparently, no. It's training on a range between ten and twenty DB, I think, and testing between five and fifteen.
  PhD D: Mm - hmm. Yeah.
  PhD A: That's what I got {pause} on {disfmarker}
  Professor B: OK.
  PhD D: It's, uh {disfmarker} It's like a medium {disfmarker} medium - mismatch condition, sort of.
  PhD A: Yeah,
  Professor B: I see.
  PhD A: and {disfmarker} So the noise is {disfmarker} There is a range of different noises also {disfmarker} um {disfmarker} which are selected randomly and added randomly, uh, to the files. And there are noises that are different from the noises used {pause} on TI - digits.
  Professor B: Yeah. Yeah. I mean, I wouldn't imagine that the amount of testing data was that huge. They probably put training {disfmarker} uh, almost certain they put training data there too. Maybe not. So. That's that. Anybody have anything else?
  PhD E: Uh, one {disfmarker} one last question on that. When did they estimate that they would have that system available for download?
  PhD D: Um, I guess {disfmarker} I guess one {disfmarker} some preliminary version is already there.
  PhD E: Oh, so there's w something you can download to just learn?
  PhD D: Yeah, it's already there. Yeah.
  PhD E: OK,
  PhD D: But they're actually parallel - y doing some modifications also, I think.
  PhD E: good.
  PhD D: So I guess the f final system will be frozen by middle of, like, one more week maybe.
  PhD E: OK.
  Professor B: Oh, well that's pretty soon.
  PhD D: Yeah, that's just one more.
  Grad C: Is this their, um, SVM recognizer?
  PhD D: No, it's just a straightforward HMM.
  Professor B: You know, their {disfmarker} their {disfmarker} They have a lot of options {pause} in their recognizer and {disfmarker} and the SVM is one of the things they've done with it, but it's not their more standard thing.
  Grad C: Oh, OK. Uh - huh.
  Professor B: For the most part it's {disfmarker} it's Gaussian mixtures.
  Grad C: Oh, OK. Oh, OK.
  Professor B: Yeah.
  PhD D: It's just a HMM, Gaussian mixture model.
  Grad C: Gaussian mixture HMM.
  Professor B: Yeah.
  Grad C: OK.
  Professor B: Yeah, the SVM thing was an HMM also. It was just a {disfmarker} it {disfmarker} it {disfmarker} it was like a hybrid, like {disfmarker}
  Grad C: Mm - hmm.
  PhD D: Yeah, this is a g yeah, this i
  Professor B: what?
  PhD D: yeah.
  Professor B: Yeah.
  PhD E: So, just so that I understand, they're providing scripts and everything so that basically, uh, you {disfmarker} you push a button and it does training, and then it does test, and everything? Is that {pause} the idea?
  PhD D: I {disfmarker} I {disfmarker} I think {disfmarker} yeah, I {disfmarker} I guess something like that. It's like {vocalsound} {disfmarker} as painless as possible,
  PhD E: Mm - hmm.
  PhD D: is what {disfmarker} Do they provide all the scripts, everything, and then {disfmarker} Just,
  PhD E: I see. Hmm. Somehow yo there's hooks to put your features in and {disfmarker}
  PhD D: ju Yeah, I th I think.
  PhD E: Hmm.
  Professor B: Hmm. Yeah, um. In fact, I mean, if you look into it a little bit, it might be reasonable {disfmarker} You know Joe, right? Yeah.
  PhD E: Mm - hmm.
  Professor B: Just to sort of ask him about the issue of, um, different features having different kinds of, uh, scaling characteristics and so on. So that, you know, w w possibly having entirely different optimal values for {disfmarker} for the usual twiddle factors and what's {disfmarker} what's the plan about that?
  PhD E: OK.
  PhD D: So sh shall we, like, add Chuck also to the mailing lists? It may be better, I mean, in that case if he's going to {disfmarker}
  Professor B: Yeah.
  PhD D: Because there's a mailing list for this.
  Professor B: Is that OK?
  PhD E: Yeah, that'd be great.
  PhD D: Yeah, I guess maybe Hari or Hynek, one of them, has to {pause} send a mail to Joe. Or maybe if you {disfmarker}
  PhD E: I {disfmarker} I could send him an email.
  PhD D: Well, yeah, to add or maybe wh
  PhD E: I {disfmarker} I know him really well.
  PhD D: Yeah, so that's just fine.
  PhD E: I {disfmarker} I was just talking with him on email the other day actually.
  PhD D: So {disfmarker}
  Professor B: Uh, yeah, and just, um, se maybe see.
  PhD D: So {disfmarker}
  PhD E: About other things, but.
  Professor B: Do you have Hari's, uh {disfmarker}?
  PhD E: I have Hari's {disfmarker}
  Professor B: Yeah, so maybe just CC Hari and say that you've just been asked to handle the large vocabulary part here, and, uh, you know,
  PhD E: OK. Would it be better if I asked Hari to ask Joe?
  Professor B: Uh. Why don't you just ask Joe but CC Hari, and then in the note say," Hari, hopefully this is OK with you" .
  PhD E: OK.
  Professor B: And then if Joe feels like he needs a confirmation, Hari can answer it.
  PhD E: OK.
  PhD D: Yeah.
  Professor B: That way you can get started asking {comment} Joe quickly while he's {disfmarker} while he's maybe still, you know, putting in nails and screws and Yeah.
  PhD D: And there is an, uh, archive of all the mails that has been {vocalsound} gon that has gone, uh, between these people {disfmarker} among these people. So just you can see all this {pause} mails in the ISIP web site {disfmarker}
  PhD E: OK.
  PhD D: Mississippi web site.
  PhD E: OK. Is that a password controlled {disfmarker}?
  PhD D: Yeah, it's password protected.
  PhD E: OK.
  PhD D: So, like {disfmarker} like, it's, like {disfmarker}
  Professor B: Have you thought about {pause} how long {pause} would be uh, most useful for you to go up to OGI?
  PhD A: I don't know, uh. We can {disfmarker} {vocalsound} For September, we can set up a work schedule and we can maybe work independently. And then at some point it maybe be better to work together again.
  Professor B: Oh, so you're {disfmarker} you're imagining more that you would come back here first for a while and then {disfmarker} and then go up there?
  PhD A: I {disfmarker}
  Professor B: I mean, it's to you.
  PhD A: Maybe, yeah.
  Professor B: I ju you guys are Well, y anyway, you don't have to decide this second but thi think about it {disfmarker} about what {disfmarker} what you would think would be the {disfmarker} the best way to work it. I'll
  PhD A: But, uh {pause} Huh. Mm - hmm.
  Professor B: support it either way, so.
  PhD A: Mm - hmm Right.
  Professor B: OK. Uh. Got anything to tell us?
  Grad C: Um. Well, I've been reading some literature about clustering of data. Just, um, I guess, let me put it in context. OK, so we're talking about discovering intermediate categories to, um {disfmarker} to classify. And, uh, I was looking at some of the work that, uh, Sangita was doing on these TRAPS things. So she has, um {disfmarker} she has temporal patterns for, um, a certain set of phonemes, from {disfmarker} from TIMIT, right? the most common phonemes. And each one of them has {disfmarker} has a {disfmarker} a nice pattern over time, a one {disfmarker} one second window. And it has {disfmarker} has these patterns. Um, so she has, um a TRAP for each one of the phonemes, um, times fifteen, for each of the fifteen critical bands. And, um, {vocalsound} she does this agglomerative hierarchical clustering which {disfmarker} which basically, um, is a clustering algorithm that, uh, starts with many, many, many different points {disfmarker} many different clusters {disfmarker} uh, corresponding to the number of data, uh, patterns that you have in the data. And then you have this distance mej metric which, uh, measures how {disfmarker} how closely related they are. And you start, um {vocalsound} by merging the patterns that are most closely related.
  PhD E: And you create a tree.
  Grad C: And y yeah, yeah, a dendrogram tree.
  PhD E: Mm - hmm.
  Grad C: Um.
  PhD E: And then you can pick, uh, values anywhere along that tree to fix your set of clusters.
  Grad C: Right, usually it's when, um {disfmarker} when the sol similarity measures, um, don't go down as much.
  PhD E: Mm - hmm.
  Grad C: And so, uh {disfmarker} so you stop at that point. And what she found was, sh um, was there were five broad, um {disfmarker} broad categories, uh, corresponding to, uh, things like, uh, fricatives and, uh, vocalic, um, and, uh, stops.
  Professor B: Mm - hmm.
  Grad C: And, uh, one for silence and {disfmarker} and another one for schwa {disfmarker} schwa sounds. Um, and, um, I was thinking about ways to {disfmarker} to generalize this because w you're {disfmarker} it's sort of like a {disfmarker} it's not a completely automatic way of clustering, because yo beforehand you have these {disfmarker} these TRAPS and you're saying that {disfmarker} that these frames correspond to this particular phoneme. Um, and that's {disfmarker} that's constraining your {disfmarker} your clustering to {disfmarker} to the set of phonemes that you already have. Um, whereas maybe we want to just take {disfmarker} take a look at, um, arbitrary windows in time, um, of varying length, um, and cluster those.
  PhD E: Mm - hmm.
  Grad C: And I'm thinking if we {disfmarker} if we do that, then we would probably, um, at some point in the clustering algorithm find that we've clustered things like, OK, thi this is a transition, um, this is a relatively stable {disfmarker} stable point.
  PhD E: Mm - hmm.
  Grad C: Um, and I'm hoping to find other things of {disfmarker} of similarity and maybe use these things as the intermediate, um {disfmarker} intermediate categories that, uh, um, I'll later classify.
  PhD E: Mm - hmm.
  Professor B: Are you looking at these in narrow bands?
  Grad C: Um, right. F um, I'm {disfmarker}
  Professor B: Cuz that's what you're gonna be using, right?
  Grad C: Yeah, yeah. I {disfmarker} I haven't exactly figured out, um, the exact details for that but, uh, the {disfmarker} the representation of the data that I was thinking of, was using, um, critical band, um, energies, {vocalsound} um, over different lengths of time. So {disfmarker} Yeah.
  Professor B: Yeah, I mean, it seems somehow that needs th uh, there's a couple things that I wonder about with this. I mean, so one is {disfmarker} is, {pause} again, looking at the same representation,
  Grad C: OK.
  Professor B: I mean, if you're going for this sort of thing where you have {pause} uh, little detectors that are looking at narrow bands, then what you're going to be looking for should be some category that you can find with the narrow bands.
  Grad C: Mm - hmm.
  Professor B: That {disfmarker} that seems to be kind of fundamental to it. Um, and then the other thing, uh, is {disfmarker} that I wonder about with it, and {disfmarker} and don't take this in the wrong way, like I {disfmarker} I know what I'm doing or anything,
  Grad C: Right.
  Professor B: but, I mean. {vocalsound} Um, just wondering really.
  Grad C: Mm - hmm.
  Professor B: Um, the sort of standard answer about this sort of thing is that if you're trying to find {pause} the right system in some sense, whether you're trying by categories or {disfmarker} or parameters {pause} um, and your goal is discrimination, then having choices based on discrimination as opposed to, um, unsupervised nearness of things, um, is actually better.
  Grad C: Hmm.
  Professor B: Um, and I don't know if that {disfmarker} I mean, since you're dealing with issues of robustness, you know, maybe {disfmarker} maybe this isn't right, but it'd be something I'd be concerned about. Because, for instance, you can imagine, uh, uh, i i if you remember from {disfmarker} from, uh {disfmarker} from your {disfmarker} your quals, John Ohala saying that, uh," buh" {comment} and" puh" {comment} differed, uh, not really cuz of voicing but because of aspiration. I mean, in as far as wha what's really there in the acoustics.
  Grad C: Mm - hmm.
  Professor B: So, um, if you looked {disfmarker} if you were doing some coarse clustering, you probably would put those two sounds together. And yet, I would gue I would guess that many of your recognition errors were coming from, uh, um, pfft, {comment} screwing up on this distinction.
  Grad C: Mm - hmm.
  Professor B: So, in fact, it's a little hard because recognizers, to first order, sort of work. And the reasons we're doing the things we're doing is because they don't work as well as we'd like. And since they sort of work, uh, it means that they are already doing {disfmarker} if you go and take any recognizer that's already out there and you say," how well is it distinguishing between {pause} schwas and stops?" Grad C: Mm - hmm.
  Professor B: Boy, I bet they're all doing nearly perfectly on this, right?
  Grad C: Mm - hmm.
  Professor B: So these {disfmarker} these big categories that differ in huge obvious ways, we already know how to do. So, what are we bringing to the party? I mean, in fact what we wanna do is have something that, particularly in the presence of noise, uh, is better at distinguishing between, uh, categories that are actually close to one another, and hence, would probably be clustered together.
  Grad C: Mmm.
  Professor B: So that's th that's the hard thing. I mean, I understand that there's this other constraint that you're considering, is that you wanna have categories that, uh {disfmarker} that would be straightforward for, say, a human being to mark if you had manual annotation. And it's something that you really think you can pick up. But I think it's also essential that you wanna look at what are the {vocalsound} confusions that you're making and how can you come up with, uh, categories that, uh, can clarify these confusions.
  Grad C: Mm - hmm. Hmm.
  Professor B: So, I mean, the standard sort of way of doing that is take a look at the algorithms you're looking at, but then throw in some discriminative aspect to it. Y y this is more like, you know, how does LDA differ from PCA? I mean, they're the same sort of thing. They're both orthogonalizing.
  Grad C: Right.
  Professor B: But, you know {disfmarker} and {disfmarker} and, um, this is a little harder because you're not just trying to find parameters. You're actually trying to find the {disfmarker} the {disfmarker} the {disfmarker} the categories themselves. Uh, so a little more like brain surgery, I think on yourself. Uh. So, uh
  Grad C: Yeah.
  Professor B: Um, anyway. That's my {pause} thought.
  Grad C: OK.
  Professor B: You've been thinking about this problem for a long time actually. I mean, well {disfmarker} W actually, you stopped thinking about it for a long time, but you used to think about it {vocalsound} a lot.
  PhD E: Yeah.
  Professor B: And you've been thinking about it more now,
  PhD D: Yeah.
  Professor B: these categories.
  PhD E: Yeah.
  Professor B: Mm - hmm.
  PhD E: I guess {disfmarker} I don't {disfmarker} I don't {disfmarker} um, it's not clear to me how to reconcile, you know, what you're saying, which I think is right, with {pause} the way I've been looking at it. That it's {disfmarker} it's {disfmarker} it's all not very clear to me. But it seems to me that the desire {disfmarker} the desirable feature to have is something that, um, is bottom - up. You know, however we do that.
  Professor B: Mm - hmm.
  PhD E: And and so I guess what I don't understand is how to do that and still be discriminative, because to be discriminative you have to have categories and the only categories that we know of to use are sort of these human {disfmarker} human sig significant {disfmarker} categories that are significant to humans, like phonemes, things like that.
  Professor B: Right.
  PhD E: But that's sort of what you want to avoid. And so that feels {disfmarker} I don't know how to get out of this.
  Professor B: Well, here's a {disfmarker} here's a, uh, uh Here's a generic and possibly useless thought, which is, {vocalsound} um, what do you really {disfmarker} I mean, in a sense the only s s systems that make sense, uh, are ones that {disfmarker} that have something from top - down in th in them. Right? Because if e even the smallest organism that's trying to learn to do anything, if it doesn't have any kind of reward for doing {disfmarker} or penal penalty for doing anything, then it's just going to behave randomly.
  PhD E: Mm - hmm.
  Professor B: So whether you're talking about something being learned through evolution or being learned through experience, it's gotta have something come down to it that gives its reward or, you know, at least some reinforcement learning,
  PhD E: Right.
  Professor B: right?
  PhD E: So the question is, how far down?
  Professor B: And
  PhD E: We could stop at words, but we don't, right? We go all the way down to phonemes.
  Professor B: Right, but I me I {disfmarker} I think that maybe in some ways part of the difficulty is {disfmarker} is trying to deal with the {disfmarker} with these phonemes. You know, and {disfmarker} and {disfmarker} and i it's almost like you want categories if {disfmarker} if our {disfmarker} if our, uh, um, {vocalsound} metric of {disfmarker} of goodness, uh, i if our {disfmarker}
  PhD E: Mm - hmm.
  Professor B: correction {disfmarker} if our metric of badness {vocalsound} is word error rate then, um, maybe we should be looking at words.
  PhD E: Mm - hmm.
  Professor B: I mean, for {disfmarker} for {disfmarker} for very nice, uh, reasons we've looked for a while at syllables, and they have a lot of good properties, but i i i if you go all the way to words, I mean, that's really {disfmarker} I mean, d w In many applications you wanna go further. You wanna go to concepts or something, or have {disfmarker} have {disfmarker} have concepts, actions, this sort of thing.
  PhD E: Yeah. But words would be a nice {disfmarker}
  Professor B: But, words aren't bad, yeah. And {disfmarker} and
  PhD E: Yeah, so the common {disfmarker} right, the common wisdom is you can't do words because there's too many of them, right? So you have to have some smaller set that you can use, uh, and {disfmarker} and so everybody goes to phonemes. But the problem is that we {disfmarker} we build models of words in terms of phonemes and these models are {disfmarker} are really cartoon - ish, right? So when you look at conversational speech, for example, you don't see the phonemes that you {disfmarker} that you have in your word models.
  Professor B: Yeah. But {disfmarker} but {disfmarker} but we're not trying for models of words here. See, so her here's maybe where {disfmarker} If the issue is that we're trying to come up with, um, some sort of intermediate categories which will then be useful for later stuff, uh, then {pause} maybe it doesn't matter that we can't have enough {disfmarker}
  PhD E: Mm - hmm. Mm - hmm.
  Professor B: I mean, what you wanna do is {disfmarker} is build up these categories that are {disfmarker} that are best for word recognition.
  PhD E: Right. Right.
  Professor B: And {disfmarker} and somehow if that's built into the loop of what the categories {disfmarker} I mean, we do this every day in this very gross way of {disfmarker} of running o a thousand experiments
  PhD E: Right.
  Professor B: because we have fast computers and picking the thing that has the best word error rate.
  PhD E: Yeah.
  Professor B: In some way {disfmarker} I mean, we derive that all the time. In some ways it's really not {comment} a bad {disfmarker} bad thing to do because it tells you in fact how your adjustments at the very low level affect the {disfmarker} the final goal.
  PhD E: Mm - hmm. Mm - hmm.
  Professor B: Um, so maybe there's a way to even put that in in a much more automatic way,
  PhD E: Right.
  Professor B: where you take, you know, something about the error at the level of the word or some other {disfmarker} it could be syllable {disfmarker} but in some large unit,
  PhD E: Uh - huh.
  Professor B: uh, and uh {disfmarker} yeah, you may not have word models, you have phone models, whatever, but you sort of {pause} don't worry about that, and just somehow feed it back through.
  PhD E: Mm - hmm.
  Professor B: You know, so that's, uh, wh what I called a useless comments because I'm not really telling you how to do it. But I mean, it's a {disfmarker} {vocalsound} it's {disfmarker} it's, you know {disfmarker} it
  PhD E: No, but I think the important part in there is that, you know, if you want to be discriminative, you have to have uh, you know, categories.
  Professor B: Right.
  PhD E: And I think this {disfmarker} the important categories are the words, and {pause} not the phones.
  Professor B: Yeah. Yeah.
  PhD E: Maybe. And so {disfmarker} Right. If you can put the words in to the loop somehow for determining goodness of your sets of clusters {disfmarker} Uh {disfmarker}
  Professor B: Now, that being said, I think that {disfmarker} that if you have something that is, um {disfmarker} i Once you start dealing with spontaneous speech, all the things you're saying are {disfmarker} are really true.
  PhD E: Mm - hmm.
  Professor B: If you {pause} have read speech that's been manually annotated, like TIMIT, then, you know, i i you the phones are gonna be right, actually, {vocalsound} for the most part.
  PhD E: Yeah. Yeah,
  Professor B: So {disfmarker} so, uh, it doesn't really hurt them to {disfmarker} to do that, to put in discrimination at that level.
  PhD E: yeah.
  Professor B: Um, if you go to spontaneous speech then it's {disfmarker} it's trickier and {disfmarker} and {disfmarker} and, uh, the phones are {disfmarker} uh, you know, it's gonna be based on bad pronunciation models that you have of {disfmarker}
  PhD E:
  Professor B: and, um {disfmarker} And it won't allow for the overlapping phenomenon
  PhD E: Mmm. So it's almost like there's this mechanism that we have that, you know, when {disfmarker} when we're hearing read speech and all the phonemes are there you know, we {disfmarker} we deal with that, but {disfmarker} but when we go to conversational, and then all of a sudden not all the phonemes are there, it doesn't really matter that much to us as humans because we have some kind of mechanism that allows for these word models, whatever those models are, to be {pause} munged, you know, and {disfmarker} and it doesn't really hurt, and I'm not sure how {disfmarker} {vocalsound} how to build that in. Uh.
  Professor B: Yeah, I mean, I guess the other thing i is {disfmarker} is to think of a little bit {disfmarker} I mean, we when y when you start looking at these kind of results I think it usually is {disfmarker} is pretty intuitive, but start looking at um, what are the kinds of confusions that you do make, uh, you know, between words if you want or {disfmarker} or {disfmarker} or, uh, even phones in {disfmarker} in {disfmarker} in {disfmarker} in read speech, say, uh, when there is noise. You know, so is it more across place or more across manner? Or is it cor you know, is it {disfmarker}?
  Grad C: Mm - hmm.
  Professor B: I mean, I know one thing that happens is that you {disfmarker} you {disfmarker} you, uh, you lose the, um, uh, low energy phones. I mean, if there's added noise then low energy phones {vocalsound} sometimes don't get heard. And if that {disfmarker} if that is {disfmarker} if it {disfmarker} uh, if that turns it into another word or {disfmarker} or different {disfmarker} you know, or another pair of words or something, then it's more likely to happen. But, um, I don't know, I w I would {disfmarker} I would guess that you'd {disfmarker}
  Grad C: Mm - hmm.
  Professor B: W I don't know. Anyway, that's {disfmarker}
  PhD E: I think part of the difficulty is that a l a lot of the robustness that we have is probably coming from a much higher level.
  Professor B: Mm - hmm.
  PhD E: You know, we understand the context of the situation when we're having a conversation. And so if there's noise in there, you know, our brain fills in and imagines what {disfmarker} what should be there.
  Professor B: Well that {disfmarker}
  Grad C: Yeah. We're {disfmarker} we're doing some sort of prediction of what {disfmarker}
  PhD E: Yeah, exactly.
  Professor B: Oh, sure, that's really big.
  Grad C: Yeah.
  Professor B: Uh, but I mean, even if you do um, uh, diagnostic rhyme test kind of things, you know, where there really isn't an any information like that, uh, people are still better in noise than they {disfmarker} than they are in {disfmarker} in, uh {disfmarker} uh, than the machines are.
  PhD E: Hmm.
  Professor B: So, I mean, that's {disfmarker} i Right. We can't {disfmarker} we can't get it at all without any language models. Language models are there and important but {disfmarker} but, uh {disfmarker} Uh. If we're not working on that then {vocalsound} we should work on something else and improve it, but {disfmarker} especially if it looks like the potential is there. So {disfmarker} Should we do some digits?
  PhD E: Yeah.
  Professor B: Since we're here?
  PhD E: Go ahead, Morgan.
  Professor B: OK.
  PhD E: OK.
  Professor B: That's all folks.
output: The professor informed the team that the evaluation was on November 13. He encouraged the team to run experiments to learn which factors were the most important. The team was concerned about how the evaluation would be weighted, but they did not expect it to be significantly different. There was a chance that the evaluation would be pushed.