Sparse Expert Models (Switch Transformers, GLAM, and more... w/ the Authors)
#nlp #sparsity #transformers

This video is an interview with Barret Zoph and William Fedus of Google Brain about Sparse Expert Models.
Sparse Expert models have been hugely successful at distributing parts of models, mostly Transformers, across large array of machines and use a routing function to effectively route signals between them. This means that even though these models have a huge number of parameters, the computational load for a given signal does not increase because the model is only sparsely activated. Sparse expert models, such as Switch Transformers and GLAM can scale up to trillions of parameters and bring a number of desirable properties. We discuss everything from the fundamentals, history, strengths and weaknesses, up to the current state of the art of these models.

OUTLINE:
0:00 - Intro
0:30 - What are sparse expert models?
4:25 - Start of Interview
5:55 - What do you mean by sparse experts?
8:10 - How does routing work in these models?
12:10 - What is the history of sparse experts?
14:45 - What does an individual expert learn?
19:25 - When are these models appropriate?
22:30 - How comparable are sparse to dense models?
26:30 - How does the pathways system connect to this?
28:45 - What improvements did GLAM make?
31:30 - The "designing sparse experts" paper
37:45 - Can experts be frozen during training?
41:20 - Can the routing function be improved?
47:15 - Can experts be distributed beyond data centers?
50:20 - Are there sparse experts for other domains than NLP?
52:15 - Are sparse and dense models in competition?
53:35 - Where do we go from here?
56:30 - How can people get started with this?

Papers:
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (https://arxiv.org/abs/2101.03961)
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts (https://arxiv.org/abs/2112.06905)
Designing Effective Sparse Expert Models (https://arxiv.org/abs/2202.08906)

Links:
Merch: http://store.ykilcher.com
TabNine Code Completion (Referral): http://bit.ly/tabnine-yannick
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
BitChute: https://www.bitchute.com/channel/yannic-kilcher
LinkedIn: https://www.linkedin.com/in/ykilcher
BiliBili: https://space.bilibili.com/2017636191

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Hello, today I'm having an interview about the topic of sparse experts. Now, ironically, the people are absolute experts in this type of models. These models, they are huge, they're usually language models, but they don't have to be they're usually transformers, but they don't have to be what they do have in common is this notion of sparse experts. These models go up to the trillions of parameters, and they achieve this via sparsity. Now I want to do a very, very brief introduction of what sparse expert models are. And then we'll dive into the interview right away, because I don't want to keep it from you. So let's look at a transformer model. Usually, I have some sort of an input that is tokens, a sequence of tokens, which are represented here by circles. And what I'm going to do with these tokens is I'm going to alternatingly push them through different layers. Now one big layer type that is common in transformers is the attention layer. We're not going to talk about the attention layer today, all you have to know is that it takes in a sequence of tokens, and it outputs a sequence of tokens again, ideally the same amount as went in, which I failed to draw here. The other very common big type of layer in these transformers is what's called the feed forward layer. Now the feed forward layer is just a linear layer and every token goes through this linear layer by itself. So every token individually goes through the same transformation. And thus as we do this with all tokens, again, we end up with a sequence of as many tokens as we input. Now a sparse expert model isn't very different than this, the attention layers commonly aren't really touched. So that works just the same. However, in the feed forward layer, we see a big difference. Notably, we don't only have one feed forward layer, we have many. So here is feed forward one, here is feed forward two, here is feed forward three, and here is feed forward four, each one representing a different individual linear transformation of a token. Now when we talk about sparse experts, these things here are called the experts. They're called the experts because they're thought to specialize in very specific tasks. And the goal in sparse expert models is to route the tokens to the corresponding correct experts. So every token goes through what's known as a routing function. We're going to talk about this routing function in the interview, but in essence, it is a very simple, usually something like a linear function or a simple transformation that decides to which of the experts any given token is routed. So sometimes even in sparse expert models, a token is routed to multiple experts. But in the newest iterations, the tokens are simply routed to one single experts and none of the other. Usually this is done, as I said, by some sort of a linear transformation, followed by a softmax to decide where the token goes. So every token would be assigned to one expert. And that gives the possibility of scaling these models up dramatically. Not only do you save a lot of compute, because the tokens only go to one place, ergo, you only need to compute that one thing for that particular token. But also, there's the opportunity to massively shard and parallelize these different experts across different machines, as you only need to route the token to one place, that means you dramatically reduce these big all to all reductions, they still happen, but not as much. So as I already said, the biggest models have trillions of parameters, you need to take a little bit of care of how you then aggregate the tokens once they come out of the experts. So essentially, what you want to do is you want to carry over the likelihood from the routing function up here. But this is a minor detail, a minor details are important, but you know, so I know it doesn't look like much, but these sparse expert models really have the potential to massively scale up our current efforts in AI. And I have no doubt that they're going to play a role in the near future, when we're looking at bigger and bigger models, because at some point, the purely dense models will reach sort of the limit of what's physically doable. And then it's a good opportunity that we have models that can go even larger. Alright, so without further ado, let's jump into the interview. I hope you're enjoying yourself. If you do have any sort of comments, please leave a comment, share the video around if you like it, and I'll see you around. Bye bye. Hello, everyone, my guests today are William Fedis and Barrett Zoff, who are engineers and researchers at Google, Google Brain, and have been diving into large models, specifically sparse expert models, which are models that, well, feature this notion of experts, and also have a notion of sparsity. And hopefully today, we'll discover what this is all about. Specifically, we'll talk broadly about three papers in a long line of work. One is the Switch Transformers paper, which was really, I believe, one of the first papers that just had like massive amounts of parameter. Was that like trillion, probably trillion parameters? It was big. 1.6 trillion parameters. That's right. Yeah, yeah, insane. And then there's there's GLAM, which demonstrated really nice scaling laws with these sparse experts. And more recently, there is designing effective sparse expert models, which as far as I can see, is also a bit of a of a maybe a summary recommendations, more of a what we learned type of thing. So William and Barrett, welcome to the channel. Thanks so much for being here. Yeah, thanks for having us. So can you give us just a little bit of context what you mean when you say sparse expert models? Yeah, sure. So this is a great question, especially since the word sparsity crops up in like many different aspects of deep learning, whether it's like sparse attention or various other sparse paradigms. So yeah, sparsity in our case means that each input can get different subsets of parameters. So that's kind of like the main sparsity that we're talking about here. And it's like, you know, it's a very natural concept, right? Like normally in like a dense transformer, for example, you have, you know, a word embedding and, you know, any word will have the same parameters and compute applied to it. And in sparse models, typically what happens is you have the same amount of compute, but you can have different subsets of the model parameters be like, you know, acting on the model inputs. And what does that mean in practice? So we're talking mainly about, let's say, transformer models here. No, is that a good characterization of things? Or do you see sparse expert models in a more general sense? Yeah, I mean, these things actually almost sort of like cropped up originally as almost like in the context of like ensemble type methods where you have a bunch of like almost like fully independent models. And then you're sort of using these as like, you know, each model is an expert. But the common paradigm as of like 2022 is sort of experts as a layer. So this was like really popularized by Noam Chazir's work in 2017, outrageously large models. And in that context, they were actually inserting it in between LSTM layers, which is like the prevailing like recurrent architecture at the time. Most of the things just because like the world has sort of shifted towards transformers in it seems like almost all modalities now, we're often thinking about experts as a layer inside transformers. Typically, we're sort of doing this at the feed forward. So these blocks that just sort of independently apply on the different like tokens, but we've also kind of considered it in self attention layers. It's just sort of like a very general concept. But yeah, typically in transformers. So you have this notion of an expert, which you say is is sort of a specialized function or something like this. And then there's often this thing called a router. How does information find its way through these experts? What are the general principles in that? And why would I even consider doing something like this? Yeah, so great question. So yeah, so you have this figure up here. And so one thing to notice that basically, if you only have a single expert, it essentially reduces to just a normal dense transformer. So the interpretation is pretty natural. And in almost all of the ways people are doing sparse expert model nowadays, there's some notion of a learned mechanism that for you know, embedding at the current layer, you figure out what expert you should send this representation to. And this can be ranging from very simple to just like a simple softmax function over the total number of experts to very complicated, linear programming type solutions that have a more like globally optimal solution. So yeah, so this is kind of like the paradigm and I think it's a pretty natural one. So even if you want to only, you know, yeah, apply one set of weights per representation, now you have the option of just instead of always applying the same weight matrix, now you can, you know, maybe have a selection of in this figure for different weight matrices. And the way that, you know, we've done this in our work, and I think is the most common is just as a single feed forward network. So you take your input representation, and then you just, you know, apply it with something that's going to be like, you know, the model dimension by the number of experts. And then you apply like a softmax function to get like a probability over all of the different experts. In our switch transformer work, the routing was extremely simple where it's just like you just send it to the highest, like the highest expert with the highest probability. And then, you know, you just simply route it to that expert, then the output of that computation gets scaled by the router probability. So if it was like, oh, with 0.9 send it to experts, the expert too, then when you have the output of that computation, you scale it all by 0.9. Do I remember correctly that there was some paper that it was this an older paper, and this might be getting very technical for a second. But was there an older paper that said something like you always needed to send it to at least two of these experts, otherwise, it's kind of unstable. Is that an older paper or a newer than yours? It actually wasn't instability that they're clashing against. It was more this idea that we're doing this like weird discretized operations. So instead of using like reinforcement learning to sort of like update on the experts, we're kind of doing this like kind of hacky back propagation through these like softmax operations, which had been masked. And the idea that top two or greater was necessary because they were thinking, well, I'm creating a probability distribution for this token for this word over the available experts. If I don't have at least two, I can't tell whether expert I or J was sort of better for this one. So it's like, in order to have, you know, the hypothesis was sort of like a useful gradient signal for the router, it has to know, well, should I have sent it to I or J? And then we just sort of didn't follow convention and did one. And it also seems to work just fine. I think in part because you're sort of doing this sort of normalization. So you can still get an up weighting or a down weighting if you select an expert. So it's like, oh, if that expert selection worked out well for you or worked out poorly for you, you can then sort of adjust the embedding for that expert. And then you at the next pass, if you saw that same token, you're still doing this like softmax distribution. So you're kind of like up weighting or down weighting it. So I think that's sort of like the gist of the mechanism. And this, I think this idea was at least from 2017, it may have predated it. Could you maybe now that we're talking about history, trace the evolution of this line of research a little bit? You already mentioned this existed as sort of ensemble methods inside of it. I'm talking specifically about sparse experts within transformers, which are the things that allow us to really scale up to these these giant models. What's the what's sort of the line of research? What are the original things? I'm going to guess this this work is among them. And what were the improvements that happened since then in this field? Do you want me to go or you? Go for it, Liam. Yeah, so I mean, like, going back 30 years, like you have like Jordans and Jacob. This obviously predates transformer because transformer was a 2017 development. So I mean, the concept is very, very old. I think it just kind of like resurged in popularity, I'd say the first. Yeah, the very first sort of use of mixture of experts and transformer was left in it all in 2020. So this is G shard. And it just showed really remarkable improvements in translation. What they were doing was, you know, analogous to switch transforming these other works is they just sort of substitute these feedforward blocks with experts. And in that case, sort of also similar with switch transformer, they had many, many experts, I think, in that case, it was thousands. And they were showing really significant improvements over state of the art translation models. I think as the field has sort of evolved, as we've sort of like learned a bit more about it, there seem to be this like kind of general trend of like, okay, cool, we can pre train these models or like in the case of translation, there's no big distribution shift. When you're training to translate, you're also doing inference to translate. But in switch transformer, we found okay, we'll pre train to, you know, improve the perplexity improve the prediction of next token. And we were getting significant improvements. But then when we took it under a data distribution shift to fine tuning, it was performing quite badly with many experts. So I think there's been this trend to try to balance the computation and the parameters a bit more. So I think some of the prevailing models have actually in transformers have actually gone towards fewer experts. So 1632 64 experts, not thousands of experts. So that's kind of like the lineage of mixture of experts and then like mixture of experts in the context of transformers. And what is so in that context? If one expert is the classic transformer model, and that seems to not work as well as many experts, but too many don't work? What is the abstraction that I can think of for an expert? Like what does an expert learn? What is an expert responsible for? Approximately, do you have any idea what happens? Like what, what, how does it make sense that the optimal number is, let's say, a few dozen and not super many, but also not super many? Yeah, so great question. So yeah, there's like a few parts to this. So one, like, I think it's really just like an empirical observation right now that you know, 16 versus 64 versus, you know, 2048 versus 10,000. You know, like, it seems like the expert numbers in the middle, like, it's not from the standpoint of like, on a per step basis, more experts typically don't make things worse. But it's like, you know, it's like, you know, it's like, you know, it's like, more experts typically don't make things worse. Usually it's like better or about the same, but things start to level off. But it's very inconvenient to have a lot of experts because it's just like a huge memory footprint. The way that the models are distributed, it's not really amenable towards typically unless you have like tons of, you know, parallel cores going. So like, actually the observation where you kind of want to actually have like a middle amount of experts is a lot of the times actually driven by just the like practicality of then like training, serving these models. Yeah, in terms of like, what these models are actually learning, like intuitively. So we actually studied this in our most recent work, kind of looking at, you know, each expert, what are they specializing in? What are they learning? And interestingly, they kind of specialize in some shallow concepts, which you would think maybe there would be like only really deep things going on and it would be kind of hard to inspect them. But, you know, we noticed like, oh, there's like a punctuation expert or an expert that will, you know, talk about, you know, like proper nouns, which we thought was pretty funny and maybe not super intuitive for, you know, how. Yeah. Yeah. Actually, if you want, you could switch over to the recent paper and we actually have a figure which sort of shows some of these things. So you can kind of like follow along and see how shallow these things actually are. Yeah. Yeah. So this would be different. So you found an expert or in this case, multiple experts that focused on these sort of things. So there's conjunctions, punctuation, verb, visual description, which is interesting because that's kind of, I want to say like a higher level thing than just the punctuation, right? Counting numbers. Yeah. How do you make sense of this stuff? Like what's going on? I, yeah, I mean, I think we were sort of expecting maybe like a higher level of description, but like, or like sort of like representation. It's, I think we've just started, started to sort of like crack and like look into these models to actually see what's going on. That obviously like one big specialization that you're seeing here are these Sentinel tokens. To make sense of that, we were sort of doing pre-training where it sort of fill in the blank task and a blank is sort of represented by these like little Sentinels. So like extra ID 10 represents like, you know, the blank 10. And we often really frequently see experts sort of specializing on these blanks. So that's sort of an interesting thing. And then I think that also might segue into maybe you want to actually given this sort of like, you know, observe specialization, maybe you actually want to make some experts higher capacity or give them more compute to sort of do things that might be harder. But honestly, I mean, this is still very early. It'd be interesting for sort of like, you know, some of the interpretability lens that like Anthropic has on some of the recent Transformers to be applied to like some sparse expert models. Some questions we've kind of received are, what is the interplay of expert specialization with sort of like self-attention specialization? And that's honestly completely open. I think we were just sort of putting this table forth to the community to be like, well, we started. It's not exactly what we would have expected, but definitely kind of like a call to dig further and hopefully like, you know, further improve things. With the also, I believe that this was, oh, yeah, here already in Switch Transformers, this ability to distribute these things across devices that comes naturally with having sparse experts. So sparsity, meaning in this case, I only send stuff to one or a few experts. And there came the ability to shard this across devices. How practical is this really to like, when would I do something like this? At what point would it become practical and useful and the best thing to do to communicate across devices for my experts? Yeah, so really great question. And I actually think this is the reason why the method works so well, actually. So the standard way I would say people are doing distributed training of these models is they have, you know, either fully data parallelism, which means like, you know, each machine has the same set of weights, but different slices of data or a blend of data and model parallelism where it's like, you know, kind of a mix where certain like, you know, cores have sometimes different weights or sometimes different data. And then you communicate stuff to make it, you know, emulate like a full model. But I think experts, one really easy interpretation of this is like, let's say you have a model and, you know, you're using data parallelism and you have four different machines. A really natural way to overlay experts on this would be you just have one expert per machine. And then, yeah, so this is like a really nice interpretation because then when you have all of your, you know, local data per core, you'd have the router weights replicated, but then you just figure out what expert they need to go to. And then that's when you kind of, you know, shuffle all the tokens around to the machines, do all the computation and then shuffle them back. And this makes it really nice because then per machine, you actually never have any more parameters than you would have had just with a dense transformer, but now you have experts. So it's actually like a really nice way of kind of, you know, thinking about how to design the models would be like, oh, you know, you have this many cores for data parallelism, just have that many experts. And that's actually a paradigm that Liem and I use a lot when designing these models as well. And yeah, I mean, I think as soon as you have this sort of like distributed model where you're already going across accelerators and devices, you do already have these communication patterns, right? Like you need to get activations to a certain place, you need to like get gradients to a certain place. So you already have these sort of like all reduced communication collectives. Expert model is going to introduce all to all communication patterns. So that can be like a more expensive thing, especially based on like your topology and the bandwidth between all of your networks, or between all of your devices. But yeah, so I mean, this is something you sort of have to like, kind of empirically test like, okay, how how much does this architecture kind of buy you in terms of performance on your task, versus the additional cost of all to all communication? But you will be communicating across devices for these big models, regardless to to train them. Yeah. So this is a good, I guess, a good segue, because you can achieve these giant models, like trillions of parameters using these sparse expert models, because naturally, I can parallelize these experts, it doesn't cost me really much more compute, because any data point, or any token only goes to one single expert. There is always a bit of the, let's say, the question of how comparable this is to the dense models. It was it was often I don't know if this is a latent feeling that I get from the community, but people would rather have the 175 billion GPT-3 model compared to the switch transformer, even if it is trillions of parameters. What is there some sort of division factor where I could compare to a dense model? Or do you think that it's an entirely different nature of a function that's computed here? Yeah, so this is a really great question. And I think there's a lot of different ways you have to kind of look at this to figure out if a sparse model is right for you. So I think actually, in a lot of applications, if it's like, hey, I want to train the model with the smallest memory footprint, so I can just be using it on the smallest amount of devices as possible, a dense model will always be better. I think on a per parameter basis, dense models are going to be performing better. So for those types of applications, I'm like, yeah, I don't think it makes sense to be using sparse models. If you want to just train the best thing that you can fit onto your local 2GPU machine or like a 10GPU machine and do really kind of low throughput, feeding in data to this, like not high or anything like that. I think sparse models are good, where you're going to be training a model and you're going to be hosting it on a lot of machines and you're going to be having a lot of high throughput going through it. So a lot of queries, a lot of stuff going through it, because then things can be batched together and then the models actually become pretty efficient. So I think that's kind of one lens to look at when you would want to use a sparse versus dense model. And I think the kind of second lens is that, for a given amount of GPU or TPU hours on a compute cluster, what model will get you the best performance? And I think that's the lens that we actually would spend a lot of time looking at for pre-training models in this paper. Like, oh, you have 512 TPU chips and I give you, you know, X budget training hours, is a dense model or sparse model going to give you the best pre-training performance? And I think our assessment was that, yeah, I think actually the Pareto optimal model typically is a sparse model in that setup. Yeah. And like comparing parameters, especially between the dense and the sparse model is just, you know, totally incomparable. So using like GPT-3 and then like our largest like switch transformer model, it's just wildly different amount of computes in our case. You can't infer that from the parameter budget. So I don't know what the like the compute ratio was between the two, but far different. Our 1.6 trillion parameter model was actually only doing about as much compute as a billion parameter model. So for each token, it was doing, you know, it was doing roughly a billion parameters worth of flops. And, you know, whereas GPT-3 is doing 175 billion parameters worth of flops. So you can sort of tune this and DeepMind has sort of also tried to come up with like a characterization of scaling properties, far more like robust than we've been able to do, of sparse expert models and try to come up with like, you know, a dense model equivalent. So like that might be like an interesting word to sort of like refer to in the future. But really, it's just like, you know, practically speaking, it's like, OK, I give you these accelerators for this amount of time. What's like the best model? So that's like probably the fairest comparison. Have you seen this pathways paper? Yes, definitely. They came out like, how does it how does it play into something like this? Is it going to make is it going to make this easier? Is it going to make it superfluous? Like, how does the the sort of ability to schedule things heterogeneously across devices? Or does it does it enable new possibilities in the sparse expert world? Yeah, so great question. So, so one thing to note is like, OK, so typically, you have dense models and a dense model like every input will have the same amount of compute and parameters applied to it. And sparse models now you have the same amount of compute, but different parameters. And I think the kind of natural next step that I think makes a lot of sense to both Liam and I is that now for each input, you have a different amount of compute applied as well. And I think pathways is really exciting, again, like you kind of mentioned for like the heterogeneous compute, where we want to have inputs that might require, you know, different parameters and also different amounts of compute. Yeah, and I think, you know, a framework like this is going to really open up like a lot of really exciting research avenues along that direction. And I think it feels like a very natural interpretation for kind of where our models are headed for in the future. Yeah, like right now, it's like our experts are all sort of completely homogenous. They're all the same size. They do the same operations pathways. You could be like, oh, this is like a recurrent expert. This is a huge expert. There's a group of small experts. You could just be a lot more flexible in design. And like, you know, sort of like alluding to that a little bit with when we were still looking at the visualization, it's like, oh, wow, a really consistent thing are experts that want to specialize in these like fill in the blank tokens, these Sentinel tokens. Perhaps that might be an avenue or an area where it's like, oh, let's dramatically increase the compute here. This is like an area where we like a lot of extra compute could really be helpful. And there wasn't really an effective way to do this with the existing infrastructures before pathways. Yeah, sorry, that's lost the train of thought. Explain to me a little bit how GLAM improved upon switch transformers. Like what's new? What's exciting there? Yeah, so I think GLAM, so one also thing to note is like there's kind of a right now division of two different types of model classes in like language modeling space, I would say. So one is like these decoder only models where, you know, it's just, you know, a single set of parameters and it's like, you're just predicting the next token like auto aggressively. And this is like, you know, what GPT-3 is. And this is also the kind of architecture that GLAM studies these models in. So the other classes, these like encoder decoder models like T5, this was also G-shard. This is kind of what also we studied in switch transformer in our most recent work as well. So I think GLAM did a few things. So one, they really, I think pushed the scale of these models. So like, while, you know, our original model in switch transformer had more parameters, like GLAM had like much more compute applied per token. And they studied these very extensively with decoder only language models. And yeah, I think their main comparison point was to GPT-3 as well. So they were studying a lot in the context of few shot and like one shot evaluations. Whereas I think a lot of our work actually centered around like fine tuning the models. But yeah, I think GLAM really like pushed the scale of these, especially in these decoder only language models and showed that like, yeah, you know, you can get as good of quality as GPT-3 with like, you know, huge computational training savings as well. And that did a really, a lot of really good work in that space. Is there a functional difference between the sparse expert routing or anything around this in GLAM? Or is it mainly what you said with decoder only and applying more compute scaling it up? So actually, there is a few differences that are more nuanced than technical. But yeah, at a high level, you know, there's a routing function, and they actually route each token to two experts. And actually, there's like some of the differences in these models comes from like how much buffer you give each token, each expert, because, you know, you need to have like fixed batch sizes for all the experts ahead of time. And so what can happen is like, you can't guarantee that like, there's going to be perfect balancing among all the tokens getting sent to experts. So like experts can overflow. And there's this key parameter that we call the capacity factor. That's probably the single handedly most important parameter when designing a mixture of expert models, because it just has such a huge impact on the communication costs, computing, everything like that for how much buffer you should have. And yeah, I think a big difference from GLAM versus our models is they actually use like a much larger capacity factor than we've used in our other works. But yeah, the routing algorithm is essentially the same. That is, yeah, I want to get a bit more into the into the routing algorithm in just a bit. But just to end this with the last paper that we've previously looked at, was I right in saying that this is much more often, let's say a general, almost like a review paper? Or how would you describe it? Yeah, I mean, I think we we tried to make sure like we're contextualizing a lot of the work. So we try to make sure the related work was like, pretty inclusive, because I mean, I think the field's really adjusted and improved a lot in the last two years. But I would sort of characterize this paper as fixing the two big flaws from our first one from switch transformers. The first was these models are unstable to train. So you'd be training and then all of a sudden the loss or just diverge, which thwarted a lot of our issues. Interestingly, it doesn't seem like the instability arises from a lot of experts. We were consistently able to train models like our trillion parameter model, for instance, with thousands of experts, never really hitting any unstable sections. Really, it kind of came from like high quops or high computation expert models, even with like few experts, those were highly unstable. And then the second thing that this paper sort of fixed was the sort of like poor fine tuning quality. So we would sort of pre train a model, it would show like really significant speed ups over a dense counterpart. But then when it came time to fine tuning, say I'm like super glue or some like other task of interest, it would just be considerably worse. So I think this paper was just really trying to sort of like kind of patch up a couple of those issues. We identify them in our first work. Yeah, I'm always a bit intimidated when a paper has a table of index by itself. Can you can you go to something that Barry and I discussed? It's like, okay, should we break this up into multiple papers? Or should this be one? Because, you know, this is like, you know, a lot of work. And, you know, this is like something that we discussed, like, maybe in the future, we should probably be producing like more bite sized pieces of work. When you when you talk about fine tuning, can you go a bit into more detail? Like, what was exactly the problem? How did you how did you also go about fixing it? So I'm only interested in, you know, how did how what's the final model like? But what does the process of debugging something like this and then getting to an architecture or a solution that actually works look like? Yeah, I mean, it's sort of this like, very interesting problem of like, you want to, there's really just like fundamental trade off. And whenever you're sort of doing this sort of like large scale work, where you want to try to understand and characterize things at a smaller scale, understand scaling properties, understand, understand, like hyper parameter dependencies. But then you also want to be consistently checking yourself at the largest scales. And this sort of balance of like, okay, you have this much compute, you have this much time, where do you allocate it? Do you do a lot of small experiments? Or do you do a few big experiments? It's kind of tricky. But I'd say part of our, like findings were the first one was like, okay, well, characterization is we're not doing better on fine tuning, what's the cause? And it seemed like perhaps our cause is not that of optimization, it's that of generalization. So if you scroll down into section four, you can just click on the link, we might be Yeah, exactly. Yeah, so this is an example an example that, you know, kind of supports a lot of the trends we're seeing. On the left is a small superglue task. So this task has only 250 training sequences, so very small. And on the right is record. So this has over 100,000 training examples. We're showing sparse models versus dense models in the two things in the two plots. Blue represents the sparse train eval. And you can see it just very quickly gets to 100%. And it outpaces in both cases, the small task and the large task outpaces the dense model getting to 100% train evaluation accuracy. But when in the small task, we'll see the dense model in red actually outperforming the ultimate performance for the sparse model in orange, whereas for the bigger tasks, the sparse model does well. And so we kind of kept seeing this like, you know, overfitting issues. And a lot of this was then led us to sort of like investigate hyperparameters. And, you know, some of the hyperparameters can sort of be adjusted in a way to make the model like less susceptible to overfitting. So you can use like different dropout parameterizations, but also things like batch size and learning rate can inject more noise, which can also be sort of like a counter to some like overfitting properties. So we tried and then sort of consistent with this, like a lot of these things were sort of like, you know, more exhaustive studies at, say, a billion parameter scale. We then tried to continue to sort of like fact check this against our larger model and make sure that these conclusions were holding. So I think it was just sort of like the debugging process was, OK, what more precisely is going wrong? And then like, what are our levers that we can sort of like pull in order to try to improve it? But, you know, a bit of art and science, really. You so you is it you observed, OK, we are probably overfitting because you saw the smaller the tasks got sort of the worst, the sparse models would ultimately perform on the validation set of those tasks. Did you and you have it's not like quite like, yeah, it's not always like quite so easy as that. But it's sort of like, you know, directionally, like I think we have support of the hypothesis, but it's not like every single small task does poorly. Every large task is great. Yeah, but I'd say directionally, it seems to be a phenomenon we've observed. You have also a bunch of experiments down here where you where you investigate some of these, for example, dropout probabilities. You also have expert dropout probability, which is one of the questions I had in that you have a particular architecture right with these with these experts. And when I think about overfitting, what in regular transformers, you know, regular transformers, I have kind of handles I can use adapter layers, I can only fine tune the head and so on. Did you ever investigate maybe only fine tuning some of the experts? Like, is that the keeping others constant? Is that ever a thing? Like, would that work or or can we can we make use somehow of the fact that we have these different experts and they're actually different functions? Yeah, great question. And I think actually, if you scroll down, we did a very naive kind of version of this not where we freeze different experts. But we you know, freeze all of the experts or maybe only train all the experts and freeze all of the other parameters. I would say our findings were this were surprising in a bad way. So nothing, nothing really worked super well. So here you can see like, and this is also we only studied this on superglue, right? So it's far from exhaustive. But yeah, so one thing we tried was updating first all of the non mixture of expert parameters only. And that actually performed about the same, which was kind of interesting. It's like, hey, like actually freezing the mixture of expert weights like seem to perform about as well as just like updating the whole model. Then when we started to, you know, update only the mixture of expert weights and freeze all the other model parameters, like the performance was actually really bad. And there was some we still fully don't understand what's going on here. We have like a few kind of like half baked hypotheses, but yeah, then when we update only the attention parameters, things are worse. And we found a slight boost updating only the feed forward network parameters that weren't the mixture of expert layers. But yeah, overall, nothing worked that well. But yeah, I think there might be some potential really interesting things of like, hey, maybe allowing only, you know, a certain subset of experts to be fine tuned. We did spend a little bit of time actually studying like pruning off experts during fine tuning. So like for a specific fine tuning task, if your pre-trained model has like 64 experts, can you just take like a subset of like two, four, eight or 16 of them? Yeah. And we also didn't really get that good of signal with this as well. Also to some of your recommendations, they actually would be compatible with expert models too. So you're free to just like fine tune like the top like top logit layer or you could add in adapter layers. Yeah, we didn't do anything like really funky like you were suggesting like, oh, we're only going to expert like update experts like three, eight and 14 or something. Yeah, my intuition is that probably wouldn't work well. But I mean, I've been proven wrong many times. Yeah, we tried some like other things that didn't make it to this table or these plots. And yeah, again, we didn't really see like a significant boost. That said, if you are only updating like a fraction of the parameters that you get some memory savings. So, you know, some nice things. Cool. I guess one, you know, there's almost an infinite number of things one could try with these things like distilling experts, like distilling multiple experts into a single expert. So you have another expert that's again free to do some some new task once you know that two experts are converging something like, I think there's it's really interesting, right? A lot of a lot of we're adding a new experts on the fly. Yeah, a lot of possibilities. And that brings me a bit to this routing function that we talked about before. And at the beginning, which seems to me is a really crucial part of the system. Yet, as you said before, very often, I've just seen this being implemented quite simplistically, maybe there's a linear transform, and then a softmax or something like this, maybe not even maybe there is some some sort of a, you know, you have some fixed keys for all of the experts, and then you route according to that. Do you like my intuition would be that this could be a powerful handle on what's, you know, on my performance downstream, this this routing function, especially also making this different during inference, you know, any any number of things, doing a Monte Carlo tree search at inference time to get to be as accurate as possible, kind of like, like AlphaGo or something. Do you have an idea on what the power of the routing function in these sparse models is? And how does it work currently? Like, what's the most the latest and greatest? And how good is it? Yeah, so this is a really good question, actually, and something we've actually spent a lot of time about. So I would say actually, in this project, probably the thing I maybe spent the most time with is trying out different routing algorithms and routing parameterizations. But we ended up kind of going with the default thing, which I also think says something a little bit about the results of it. Yeah, so I would say my intuition is that the model actually works surprisingly well with a lot of different ways you can route the tokens. So like, you know, we tried a lot of other routing algorithms, we tried making like the routing network larger, we tried like, you know, some fancier ways of actually figuring out where you should send the token to. We tried, you know, using additional information of like, oh, when you're routing this current representation, you have access to whether or not like it was routed, or like where it was routed before in previous layers, using like word embedding information too. But yeah, I think overall, it seemed to be, you know, kind of insensitive. We actually did find like one or two methods that improve things, but they could only be used in certain situations, so it was a bit trickier to just like replace everything. The current routing algorithm we're using is basically what the original one was doing, I think in Shazer et al in 2017, when these kind of things were like really introduced into the LSTM language models. And I think, you know, our newer work, and then also GLAM as well, we're using these kind of routing algorithms too. Yeah, and also like one kind of like detail here, it's like, so right now we're sort of splitting out this little box and we're like, oh, this is the router. It's not really an accurate characterization. It's like, yes, okay, you're mapping some vector into a vector that has like the same like length as number of experts. But if you just don't update that matrix, it still works fine, right? Because now just the represent, like the weight matrices below you are just sort of adapting and just piping whatever activation they need, right? If you freeze the great, if you stop a gradient through that, then it's like catastrophically bad. But yeah, I mean, I've also sort of been surprised by the relative insensitivity to the routing algorithm. Like we've seen like, you know, maybe some small boost here and there, but it hasn't been super significant. I think you'd probably have a better, sort of like a bigger significance by actually just sort of fundamentally changing like the architecture. Like maybe there's like some wildly different approach for sort of sparse models that we're not considering. Maybe we're in some sort of like local min and like these small tweaks, I'm like, oh, okay, precisely how are we doing this? Maybe it doesn't matter as much. And DeepMind has also explored some other kind of interesting routing algorithms. Like you sort of alluded to fixed routing algorithms where it's just like, you're not even learning. They've also tried RL based routing algorithms. And I think it had like actually similar scaling properties. So again, kind of corroborating what Barrett is saying, it's just like a lot of these things when we're kind of doing this like per token routing, haven't really moved the needle substantially. That's been our luck. Yeah. And I think another important trend actually is that when we were experimenting with a lot of these different routing algorithms, we actually found that they did help models. And maybe when you had like a 1 billion parameter dense model-ish size, but then like as we scaled up the models, like actually a lot of the times, sometimes the differences would just like wash away as well. So it's kind of this interesting effect of when more scale is increased, like it maybe becomes a little bit less insensitive to some of these decisions. Yeah, I can totally see that essentially that the rest of the network adjusts, especially if everything is trainable. What I would be excited about maybe is to somehow at inference time doing something smarter than, because at training time, I can adjust to everything, right? But at inference time, maybe there's something that I could do, especially with regards to domain shift, domain adaptation, anything like this where I could tweak routing in some way, but I guess that's also up for future work. So there's a little bit of this, not tweaking the routing algorithm, but tweaking the capacity factor hyperparameter I mentioned a while ago. So this is basically the parameter that's going to dictate how many tokens are being dropped. And one cool thing you can do is you can have some capacity factor during training, but then at eval time, depending on if you want to use more or less compute, you can be either dropping more or less tokens and either increase or decrease the performance, which is pretty cool. And the model's actually pretty robust to having that train from train and evaluation time. So that's actually kind of like a good lever for depending on if you want to use more or less compute during evaluation. I think we have a pretty good overview now. I want to get a little bit into just the future prospects maybe also of this. We already talked about with pathways, we could have heterogeneous things. Could this be pushed to some sort of limit? Whenever I see a distributed system, I immediately think distributed, maybe not even in a data center, but across users, across networks. Is there applications to maybe, what was it called? Federated, some kind of federated computing, some kind of federated learning where I could somehow contribute with my maybe confidential data, but I could still contribute to a whole compute process. Is there, I'm going to say the B word, is there an application for blockchain distribution, something like this? Do you think about the higher degrees of distribution here? Do you want me to go for it? Yeah, go for it. Me personally, I haven't spent a ton of time thinking about this, but I do think it's very interesting. And yeah, there definitely seems to be a lot of really open problems around this, especially given the growing amount of fragmented compute, fragmented devices. There's so much compute around here. How can you effectively utilize all of this, utilize different data and stuff? I think it's a super cool, and I think it was going to require a lot of really interesting research because right now the way we're currently training these models is it's all synchronized lockstep typically. You're doing like, oh, after each batch you do these gradients, you send the gradients around and everything. But I think actually maybe the future of these models when you're really allowing them to be distributed across very different types of computing, everything might actually now introduce asynchronous training as kind of like the new paradigm. So I think that's a really exciting space, but yeah, I haven't spent too much time thinking about it personally. Yeah. And I think as it pertains to say blockchain or something, I think one problem with these expert models as designed in this way are these all to all communications. So over this sort of decentralized peer-to-peer network where nodes are really far apart, inconsistent bandwidth and stuff, that could be really tough if sort of your experts were sort of distributed among many different nodes in this sort of like unreliable network where nodes are kind of coming and going. Like right now, all our systems are in this sort of like very constrained fault intolerant area where it's like, oh, all highly internet work chips that are highly reliable. And then so like blockchain would just have like a whole different set of like kind of problems that you'd have to sort of address like unreliability and some of these other areas. Not to say, I think it just like requires some like additional kind of research, like just sort of adopting the model as is, I think would pretty poorly map on that kind of computing infrastructure. But I think there's something there that could be done. Is there work on because I see these works mostly here in NLP, yet transformers kind of taking over the rest of the world. Is there work on how these experts, sparse expert transformers behave in vision, in reinforcement learning, speech, whatever? Yeah, yeah, great question. So absolutely, actually, there's been some really good work applying these models to like the IT based, like image classification and stuff. And there, it's actually really nice, because then you can leverage all of the, you know, niceties around like people figuring out how to get these working really well in transformers and kind of, you know, nicely map it over as well. I've all Yeah, there's also been some good work using these in speech as well. Liam, any other things to add on top of that? Some I used to do reinforcement learning, more full time and some colleagues kind of reached out about doing like sparse expert models for RL. I haven't seen I'm not familiar with some work. But, you know, that might be sort of like another interesting avenue. But like, for sure. So language, vision, speech. I don't know if there's been any video work yet. Yeah, but like high data, a lot of throughput, those would be like, you know, really good areas. I think video would be also really promising. Yeah, I really like also the I feel like it feels very natural in these high dimensionality spaces that you really might want different parameters to be applied like when you have a video, like one, I think you don't want to be applying all the same amount of compute to every frame. But then on top of that, I can see like, actually, you really want to have different parameters applying the different, you know, things going on in the video, because it's just gonna be like, wildly different stuff happening. So yeah, I think I think I'm very excited about these models for video as well. Do you imagine that these models will just essentially right now they're competition to dense models, they are they're competing, you're tracking Pareto front tiers, how much compute, how well are they doing, tackling very much the same tasks? Do you think this will go on? Like, do you do you think these models might overtake dense models if we figure out how to handle them correctly? Or is it is it more like, there's a killer app for each one of them? Yeah, I think in Oh, go ahead. I Yeah, I mean, I honestly think that the future is going to be adaptive. Like, I don't think there's any way that like, in 10 years, our models are treating all examples coming in with like the same parameters over and over again, and the same amount of compute. It may not be this precise sort of like sparsity regime, or it may not be the precise sort of adaptive computation, kind of like paradigms that have been put forth. But I view this sort of kind of work of like sparsity, adaptive computation, as kind of like inevitable, like I don't think it's going to be considered like competition, it's just going to be sort of like, integrated into a lot of like leading models. That's, that's my expectation. I'd be really shocked in like 10 years, we're training like a 100 trillion parameter dense model. And it's just kind of doing the same thing, like, over and over again, for no matter what comes in, just seems really strange to me. What's the future for your particular research? Like, where do you see where do you see yourself going in the next, maybe not the next paper that you haven't published yet, but maybe a bit broader timescale? Like what excites you? And what are your next plans here? Yeah, great question. I mean, I think the thing that really excites me is like what we were kind of talking about earlier of each input getting a different amount of compute applied. Like, I think right now, the models are working well for each input getting different parameters. And I think, you know, coupling this with like, adaptive amounts of computation is like, I think, really where I want to be spending time thinking about in the next, you know, upcoming years. Is there? Yeah, I don't know, is you have something like Ponder, there's PonderNet, and so on. There's these recursive architectures, or recurrent architectures that, that sort of decide themselves when to when to exit? Would that be one thing? Or do you simply imagine that each expert is kind of one is the buff expert, and one is the lean expert, and then the routing function essentially takes care of the different amount of compute? Yeah, I don't know. This is a great question. I think, I don't know, I can see either approach potentially working, or maybe you actually want combinations or potentially something completely new. Yeah, it feels like the space is still, you know, very exciting. And there's like a lot of really interesting different verticals being pushed. So the space still feels like, you know, pretty young to me. Okay, last question from my side, what's the connection of this to something like capsules? I don't know if you've ever thought about the the connection there. But with capsules, I always think this is these abstract, very abstract things, very high level ideas flying around. And you here have something like very practical, you know, very on the metal thing. Yeah, there seems to be quite some commonalities. Is there is that something that ever came up to you? Or in the two years of doing sparsity research, this is literally the first time. I actually should be going back to that work. I feel like capsules like had a lot of like really interesting conceptions. But maybe like you're kind of alluding to it didn't like map super well to the metal. So maybe that sort of like hindered its like its use, whereas this is just like highly motivated from like an engineering perspective. We've had like some questions like, Oh, what is like the neuroscientific kind of motivation of our work? And it's like, it's really engineering kind of driven. So it's like, okay, what what will be fast on our existing hardware? But yeah, I will revisit capsules and kind of see like, Oh, okay, how could we actually map this a little bit better to the hardware? And like, you know, I think that could be like, you know, an interesting source of ideas. Is there any any last thing you want to get out to viewers that they should take away from this work? Any any way that a regular person can get into this type of research? Anything like this? Yes, a great question. So actually, one thing we tried to show in our switch transformer work is that these models work pretty well, even if you only have two experts. So I definitely I don't want people to think that you know, you really need a supercomputer to run the models or to, you know, get benefits from having experts, even having I think, as little as two experts and running models could lead to developing really interesting research ideas, improving the performance and everything like that. So yeah, I definitely hope that you know, more people can continue to experiment and push for these models. Yeah, and then I would say like, another interesting trend that I've been following is sort of in parallel to sparsity in these like, you know, really large models is the idea of like, well, what if we just sort of like have the model sort of offload and like, sort of do lookups or, you know, look at documents and retrieval type methods. I think this is sort of like a very interesting area. And I'd love to see like, kind of head to head comparisons of like, okay, do we want to try to encapsulate the knowledge into parameters? Or do we want to just like, keep it sort of like, you know, parametric, non parametric type thing? And we keep the information kind of written in docs or like, what does the interplay look like? I think that's sort of like another really interesting avenue, like, kind of comparing these things. Awesome. Yeah, it sounds really cool. I'm excited to see what the future of these models bring. Yeah, Barrett and William, thank you so much for being here. This was a lot of fun. I hope to see you again soon. Yeah, cool. Thanks for having us. Yeah, thanks for having us.
[{"start": 0.0, "end": 5.28, "text": " Hello, today I'm having an interview about the topic of sparse experts. Now, ironically,"}, {"start": 5.28, "end": 11.200000000000001, "text": " the people are absolute experts in this type of models. These models, they are huge, they're"}, {"start": 11.200000000000001, "end": 15.76, "text": " usually language models, but they don't have to be they're usually transformers, but they don't"}, {"start": 15.76, "end": 21.2, "text": " have to be what they do have in common is this notion of sparse experts. These models go up to"}, {"start": 21.2, "end": 27.04, "text": " the trillions of parameters, and they achieve this via sparsity. Now I want to do a very,"}, {"start": 27.04, "end": 31.919999999999998, "text": " very brief introduction of what sparse expert models are. And then we'll dive into the interview"}, {"start": 31.919999999999998, "end": 37.2, "text": " right away, because I don't want to keep it from you. So let's look at a transformer model. Usually,"}, {"start": 37.2, "end": 42.879999999999995, "text": " I have some sort of an input that is tokens, a sequence of tokens, which are represented here"}, {"start": 42.879999999999995, "end": 48.239999999999995, "text": " by circles. And what I'm going to do with these tokens is I'm going to alternatingly push them"}, {"start": 48.239999999999995, "end": 54.56, "text": " through different layers. Now one big layer type that is common in transformers is the attention"}, {"start": 54.56, "end": 59.92, "text": " layer. We're not going to talk about the attention layer today, all you have to know is that it takes"}, {"start": 59.92, "end": 66.72, "text": " in a sequence of tokens, and it outputs a sequence of tokens again, ideally the same amount as went"}, {"start": 66.72, "end": 72.64, "text": " in, which I failed to draw here. The other very common big type of layer in these transformers is"}, {"start": 72.64, "end": 78.0, "text": " what's called the feed forward layer. Now the feed forward layer is just a linear layer and every"}, {"start": 78.0, "end": 85.36, "text": " token goes through this linear layer by itself. So every token individually goes through the same"}, {"start": 85.36, "end": 90.72, "text": " transformation. And thus as we do this with all tokens, again, we end up with a sequence of as"}, {"start": 90.72, "end": 96.72, "text": " many tokens as we input. Now a sparse expert model isn't very different than this, the attention"}, {"start": 96.72, "end": 102.24000000000001, "text": " layers commonly aren't really touched. So that works just the same. However, in the feed forward"}, {"start": 102.24, "end": 108.24, "text": " layer, we see a big difference. Notably, we don't only have one feed forward layer, we have many."}, {"start": 108.24, "end": 114.72, "text": " So here is feed forward one, here is feed forward two, here is feed forward three, and here is feed"}, {"start": 114.72, "end": 120.96, "text": " forward four, each one representing a different individual linear transformation of a token. Now"}, {"start": 120.96, "end": 126.64, "text": " when we talk about sparse experts, these things here are called the experts. They're called the"}, {"start": 126.64, "end": 132.8, "text": " experts because they're thought to specialize in very specific tasks. And the goal in sparse expert"}, {"start": 132.8, "end": 139.12, "text": " models is to route the tokens to the corresponding correct experts. So every token goes through"}, {"start": 139.12, "end": 143.52, "text": " what's known as a routing function. We're going to talk about this routing function in the interview,"}, {"start": 143.52, "end": 148.64, "text": " but in essence, it is a very simple, usually something like a linear function or a simple"}, {"start": 148.64, "end": 155.44, "text": " transformation that decides to which of the experts any given token is routed. So sometimes"}, {"start": 155.44, "end": 160.96, "text": " even in sparse expert models, a token is routed to multiple experts. But in the newest iterations,"}, {"start": 160.96, "end": 166.96, "text": " the tokens are simply routed to one single experts and none of the other. Usually this is done, as I"}, {"start": 166.96, "end": 173.76, "text": " said, by some sort of a linear transformation, followed by a softmax to decide where the token"}, {"start": 173.76, "end": 179.92, "text": " goes. So every token would be assigned to one expert. And that gives the possibility of scaling"}, {"start": 179.92, "end": 185.36, "text": " these models up dramatically. Not only do you save a lot of compute, because the tokens only go to"}, {"start": 185.36, "end": 190.96, "text": " one place, ergo, you only need to compute that one thing for that particular token. But also,"}, {"start": 190.96, "end": 196.32000000000002, "text": " there's the opportunity to massively shard and parallelize these different experts across"}, {"start": 196.32000000000002, "end": 201.76000000000002, "text": " different machines, as you only need to route the token to one place, that means you dramatically"}, {"start": 201.76000000000002, "end": 207.60000000000002, "text": " reduce these big all to all reductions, they still happen, but not as much. So as I already said,"}, {"start": 207.60000000000002, "end": 212.48000000000002, "text": " the biggest models have trillions of parameters, you need to take a little bit of care of how you"}, {"start": 212.48, "end": 217.67999999999998, "text": " then aggregate the tokens once they come out of the experts. So essentially, what you want to do"}, {"start": 217.67999999999998, "end": 223.83999999999997, "text": " is you want to carry over the likelihood from the routing function up here. But this is a minor"}, {"start": 223.83999999999997, "end": 228.95999999999998, "text": " detail, a minor details are important, but you know, so I know it doesn't look like much, but"}, {"start": 228.95999999999998, "end": 234.72, "text": " these sparse expert models really have the potential to massively scale up our current"}, {"start": 234.72, "end": 240.07999999999998, "text": " efforts in AI. And I have no doubt that they're going to play a role in the near future, when"}, {"start": 240.08, "end": 245.68, "text": " we're looking at bigger and bigger models, because at some point, the purely dense models will reach"}, {"start": 245.68, "end": 251.76000000000002, "text": " sort of the limit of what's physically doable. And then it's a good opportunity that we have models"}, {"start": 251.76000000000002, "end": 256.96000000000004, "text": " that can go even larger. Alright, so without further ado, let's jump into the interview. I hope"}, {"start": 256.96000000000004, "end": 261.52000000000004, "text": " you're enjoying yourself. If you do have any sort of comments, please leave a comment, share the"}, {"start": 261.52000000000004, "end": 268.40000000000003, "text": " video around if you like it, and I'll see you around. Bye bye. Hello, everyone, my guests today"}, {"start": 268.4, "end": 275.2, "text": " are William Fedis and Barrett Zoff, who are engineers and researchers at Google, Google Brain,"}, {"start": 275.2, "end": 282.15999999999997, "text": " and have been diving into large models, specifically sparse expert models, which are"}, {"start": 282.15999999999997, "end": 289.52, "text": " models that, well, feature this notion of experts, and also have a notion of sparsity. And hopefully"}, {"start": 289.52, "end": 295.03999999999996, "text": " today, we'll discover what this is all about. Specifically, we'll talk broadly about three"}, {"start": 295.04, "end": 301.52000000000004, "text": " papers in a long line of work. One is the Switch Transformers paper, which was really, I believe,"}, {"start": 301.52000000000004, "end": 307.92, "text": " one of the first papers that just had like massive amounts of parameter. Was that like trillion,"}, {"start": 307.92, "end": 313.92, "text": " probably trillion parameters? It was big. 1.6 trillion parameters. That's right. Yeah, yeah,"}, {"start": 313.92, "end": 322.08000000000004, "text": " insane. And then there's there's GLAM, which demonstrated really nice scaling laws with these"}, {"start": 322.08, "end": 328.47999999999996, "text": " sparse experts. And more recently, there is designing effective sparse expert models,"}, {"start": 328.47999999999996, "end": 336.64, "text": " which as far as I can see, is also a bit of a of a maybe a summary recommendations, more of a what"}, {"start": 336.64, "end": 344.15999999999997, "text": " we learned type of thing. So William and Barrett, welcome to the channel. Thanks so much for being"}, {"start": 344.15999999999997, "end": 351.68, "text": " here. Yeah, thanks for having us. So can you give us just a little bit of context what"}, {"start": 351.68, "end": 358.88, "text": " you mean when you say sparse expert models? Yeah, sure. So this is a great question, especially"}, {"start": 358.88, "end": 363.44, "text": " since the word sparsity crops up in like many different aspects of deep learning, whether it's"}, {"start": 363.44, "end": 370.48, "text": " like sparse attention or various other sparse paradigms. So yeah, sparsity in our case means"}, {"start": 370.48, "end": 376.4, "text": " that each input can get different subsets of parameters. So that's kind of like the main"}, {"start": 376.4, "end": 381.84, "text": " sparsity that we're talking about here. And it's like, you know, it's a very natural concept,"}, {"start": 381.84, "end": 386.96, "text": " right? Like normally in like a dense transformer, for example, you have, you know, a word embedding"}, {"start": 387.52, "end": 392.47999999999996, "text": " and, you know, any word will have the same parameters and compute applied to it."}, {"start": 393.44, "end": 396.88, "text": " And in sparse models, typically what happens is you have the same amount of compute,"}, {"start": 396.88, "end": 402.15999999999997, "text": " but you can have different subsets of the model parameters be like, you know, acting on the model"}, {"start": 402.16, "end": 408.32000000000005, "text": " inputs. And what does that mean in practice? So we're talking mainly about, let's say,"}, {"start": 408.32000000000005, "end": 414.48, "text": " transformer models here. No, is that a good characterization of things? Or do you see sparse"}, {"start": 414.48, "end": 419.76000000000005, "text": " expert models in a more general sense? Yeah, I mean, these things actually almost sort of like"}, {"start": 419.76000000000005, "end": 423.92, "text": " cropped up originally as almost like in the context of like ensemble type methods where you"}, {"start": 423.92, "end": 429.76000000000005, "text": " have a bunch of like almost like fully independent models. And then you're sort of using these as"}, {"start": 429.76, "end": 436.8, "text": " like, you know, each model is an expert. But the common paradigm as of like 2022 is sort of experts"}, {"start": 436.8, "end": 443.52, "text": " as a layer. So this was like really popularized by Noam Chazir's work in 2017, outrageously large"}, {"start": 443.52, "end": 448.08, "text": " models. And in that context, they were actually inserting it in between LSTM layers, which is like"}, {"start": 448.08, "end": 452.96, "text": " the prevailing like recurrent architecture at the time. Most of the things just because like the"}, {"start": 452.96, "end": 457.03999999999996, "text": " world has sort of shifted towards transformers in it seems like almost all modalities now,"}, {"start": 457.04, "end": 463.6, "text": " we're often thinking about experts as a layer inside transformers. Typically, we're sort of"}, {"start": 463.6, "end": 468.32, "text": " doing this at the feed forward. So these blocks that just sort of independently apply on the"}, {"start": 468.32, "end": 474.24, "text": " different like tokens, but we've also kind of considered it in self attention layers. It's just"}, {"start": 474.24, "end": 479.76, "text": " sort of like a very general concept. But yeah, typically in transformers. So you have this"}, {"start": 479.76, "end": 487.68, "text": " notion of an expert, which you say is is sort of a specialized function or something like this. And"}, {"start": 487.68, "end": 495.44, "text": " then there's often this thing called a router. How does information find its way through these"}, {"start": 495.44, "end": 501.76, "text": " experts? What are the general principles in that? And why would I even consider doing something like"}, {"start": 501.76, "end": 509.92, "text": " this? Yeah, so great question. So yeah, so you have this figure up here. And so one thing to notice"}, {"start": 509.92, "end": 515.2, "text": " that basically, if you only have a single expert, it essentially reduces to just a normal dense"}, {"start": 515.2, "end": 521.2, "text": " transformer. So the interpretation is pretty natural. And in almost all of the ways people are"}, {"start": 521.2, "end": 527.36, "text": " doing sparse expert model nowadays, there's some notion of a learned mechanism that for you know,"}, {"start": 527.36, "end": 532.64, "text": " embedding at the current layer, you figure out what expert you should send this representation to."}, {"start": 533.76, "end": 539.28, "text": " And this can be ranging from very simple to just like a simple softmax function over the total"}, {"start": 539.28, "end": 544.8000000000001, "text": " number of experts to very complicated, linear programming type solutions that have a more"}, {"start": 544.8000000000001, "end": 552.0, "text": " like globally optimal solution. So yeah, so this is kind of like the paradigm and I think it's a"}, {"start": 552.0, "end": 559.6, "text": " pretty natural one. So even if you want to only, you know, yeah, apply one set of weights per"}, {"start": 559.6, "end": 564.48, "text": " representation, now you have the option of just instead of always applying the same weight matrix,"}, {"start": 564.48, "end": 570.0, "text": " now you can, you know, maybe have a selection of in this figure for different weight matrices. And"}, {"start": 570.0, "end": 574.0, "text": " the way that, you know, we've done this in our work, and I think is the most common is just as"}, {"start": 574.0, "end": 578.96, "text": " a single feed forward network. So you take your input representation, and then you just, you know,"}, {"start": 578.96, "end": 582.88, "text": " apply it with something that's going to be like, you know, the model dimension by the number of"}, {"start": 582.88, "end": 586.72, "text": " experts. And then you apply like a softmax function to get like a probability over all of the"}, {"start": 586.72, "end": 591.36, "text": " different experts. In our switch transformer work, the routing was extremely simple where it's just"}, {"start": 591.36, "end": 596.4000000000001, "text": " like you just send it to the highest, like the highest expert with the highest probability."}, {"start": 597.2800000000001, "end": 601.84, "text": " And then, you know, you just simply route it to that expert, then the output of that computation"}, {"start": 601.84, "end": 608.1600000000001, "text": " gets scaled by the router probability. So if it was like, oh, with 0.9 send it to experts,"}, {"start": 608.16, "end": 614.4, "text": " the expert too, then when you have the output of that computation, you scale it all by 0.9."}, {"start": 615.28, "end": 620.48, "text": " Do I remember correctly that there was some paper that it was this an older paper,"}, {"start": 620.48, "end": 626.24, "text": " and this might be getting very technical for a second. But was there an older paper that said"}, {"start": 626.24, "end": 630.16, "text": " something like you always needed to send it to at least two of these experts, otherwise,"}, {"start": 630.16, "end": 635.28, "text": " it's kind of unstable. Is that an older paper or a newer than yours?"}, {"start": 635.28, "end": 641.68, "text": " It actually wasn't instability that they're clashing against. It was more this idea that"}, {"start": 642.48, "end": 647.12, "text": " we're doing this like weird discretized operations. So instead of using like reinforcement"}, {"start": 647.12, "end": 651.76, "text": " learning to sort of like update on the experts, we're kind of doing this like kind of hacky back"}, {"start": 651.76, "end": 659.36, "text": " propagation through these like softmax operations, which had been masked. And the idea that top two"}, {"start": 659.36, "end": 664.24, "text": " or greater was necessary because they were thinking, well, I'm creating a probability"}, {"start": 664.24, "end": 670.0, "text": " distribution for this token for this word over the available experts. If I don't have at least two,"}, {"start": 670.0, "end": 677.12, "text": " I can't tell whether expert I or J was sort of better for this one. So it's like, in order to"}, {"start": 677.12, "end": 682.64, "text": " have, you know, the hypothesis was sort of like a useful gradient signal for the router, it has to"}, {"start": 682.64, "end": 688.72, "text": " know, well, should I have sent it to I or J? And then we just sort of didn't follow convention and"}, {"start": 688.72, "end": 694.24, "text": " did one. And it also seems to work just fine. I think in part because you're sort of doing this"}, {"start": 694.24, "end": 700.0, "text": " sort of normalization. So you can still get an up weighting or a down weighting if you select"}, {"start": 700.48, "end": 705.52, "text": " an expert. So it's like, oh, if that expert selection worked out well for you or worked out"}, {"start": 705.52, "end": 710.8000000000001, "text": " poorly for you, you can then sort of adjust the embedding for that expert. And then you at the"}, {"start": 710.8000000000001, "end": 715.2, "text": " next pass, if you saw that same token, you're still doing this like softmax distribution. So"}, {"start": 715.2, "end": 718.5600000000001, "text": " you're kind of like up weighting or down weighting it. So I think that's sort of like the gist of the"}, {"start": 718.5600000000001, "end": 726.24, "text": " mechanism. And this, I think this idea was at least from 2017, it may have predated it."}, {"start": 726.88, "end": 733.44, "text": " Could you maybe now that we're talking about history, trace the evolution of this line of"}, {"start": 733.44, "end": 740.4000000000001, "text": " research a little bit? You already mentioned this existed as sort of ensemble methods inside of it."}, {"start": 740.4, "end": 747.04, "text": " I'm talking specifically about sparse experts within transformers, which are the things that"}, {"start": 747.04, "end": 752.3199999999999, "text": " allow us to really scale up to these these giant models. What's the what's sort of the line of"}, {"start": 752.3199999999999, "end": 758.0799999999999, "text": " research? What are the original things? I'm going to guess this this work is among them. And what"}, {"start": 758.0799999999999, "end": 761.6, "text": " were the improvements that happened since then in this field?"}, {"start": 761.6, "end": 762.72, "text": " Do you want me to go or you?"}, {"start": 762.72, "end": 763.68, "text": " Go for it, Liam."}, {"start": 763.68, "end": 770.0, "text": " Yeah, so I mean, like, going back 30 years, like you have like Jordans and Jacob. This obviously"}, {"start": 770.0, "end": 775.68, "text": " predates transformer because transformer was a 2017 development. So I mean, the concept is very,"}, {"start": 775.68, "end": 783.5999999999999, "text": " very old. I think it just kind of like resurged in popularity, I'd say the first. Yeah, the very"}, {"start": 783.5999999999999, "end": 789.5999999999999, "text": " first sort of use of mixture of experts and transformer was left in it all in 2020. So this"}, {"start": 789.6, "end": 796.4, "text": " is G shard. And it just showed really remarkable improvements in translation. What they were doing"}, {"start": 796.4, "end": 800.64, "text": " was, you know, analogous to switch transforming these other works is they just sort of substitute"}, {"start": 800.64, "end": 805.84, "text": " these feedforward blocks with experts. And in that case, sort of also similar with switch"}, {"start": 805.84, "end": 810.4, "text": " transformer, they had many, many experts, I think, in that case, it was thousands. And they were"}, {"start": 810.4, "end": 813.9200000000001, "text": " showing really significant improvements over state of the art translation models."}, {"start": 813.92, "end": 818.7199999999999, "text": " I think as the field has sort of evolved, as we've sort of like learned a bit more about it,"}, {"start": 819.76, "end": 824.8, "text": " there seem to be this like kind of general trend of like, okay, cool, we can pre train"}, {"start": 824.8, "end": 829.36, "text": " these models or like in the case of translation, there's no big distribution shift. When you're"}, {"start": 829.36, "end": 835.04, "text": " training to translate, you're also doing inference to translate. But in switch transformer, we found"}, {"start": 835.04, "end": 835.4599999999999, "text": " okay, we'll pre train to, you know, improve the perplexity improve the"}, {"start": 835.46, "end": 844.2800000000001, "text": " prediction of next token. And we were getting significant improvements. But then when we took"}, {"start": 844.2800000000001, "end": 851.1600000000001, "text": " it under a data distribution shift to fine tuning, it was performing quite badly with many experts."}, {"start": 851.1600000000001, "end": 856.36, "text": " So I think there's been this trend to try to balance the computation and the parameters a bit"}, {"start": 856.36, "end": 861.08, "text": " more. So I think some of the prevailing models have actually in transformers have actually gone"}, {"start": 861.08, "end": 869.32, "text": " towards fewer experts. So 1632 64 experts, not thousands of experts. So that's kind of like the"}, {"start": 869.32, "end": 874.36, "text": " lineage of mixture of experts and then like mixture of experts in the context of transformers."}, {"start": 875.88, "end": 885.1600000000001, "text": " And what is so in that context? If one expert is the classic transformer model, and that seems to"}, {"start": 885.16, "end": 892.76, "text": " not work as well as many experts, but too many don't work? What is the abstraction that I can"}, {"start": 892.76, "end": 898.52, "text": " think of for an expert? Like what does an expert learn? What is an expert responsible for?"}, {"start": 898.52, "end": 905.24, "text": " Approximately, do you have any idea what happens? Like what, what, how does it make sense that the"}, {"start": 905.24, "end": 911.48, "text": " optimal number is, let's say, a few dozen and not super many, but also not super many?"}, {"start": 911.48, "end": 917.88, "text": " Yeah, so great question. So yeah, there's like a few parts to this. So one, like, I think it's"}, {"start": 917.88, "end": 923.72, "text": " really just like an empirical observation right now that you know, 16 versus 64 versus, you know,"}, {"start": 923.72, "end": 929.88, "text": " 2048 versus 10,000. You know, like, it seems like the expert numbers in the middle, like,"}, {"start": 929.88, "end": 934.6, "text": " it's not from the standpoint of like, on a per step basis, more experts typically don't make"}, {"start": 934.6, "end": 939.32, "text": " things worse. But it's like, you know, it's like, you know, it's like, you know, it's like,"}, {"start": 939.32, "end": 943.32, "text": " more experts typically don't make things worse. Usually it's like better or about the same,"}, {"start": 943.32, "end": 948.84, "text": " but things start to level off. But it's very inconvenient to have a lot of experts because"}, {"start": 948.84, "end": 953.4000000000001, "text": " it's just like a huge memory footprint. The way that the models are distributed, it's not really"}, {"start": 953.4000000000001, "end": 958.7600000000001, "text": " amenable towards typically unless you have like tons of, you know, parallel cores going. So like,"}, {"start": 958.7600000000001, "end": 964.12, "text": " actually the observation where you kind of want to actually have like a middle amount of experts is"}, {"start": 964.12, "end": 969.64, "text": " a lot of the times actually driven by just the like practicality of then like training, serving"}, {"start": 969.64, "end": 976.76, "text": " these models. Yeah, in terms of like, what these models are actually learning, like intuitively. So"}, {"start": 976.76, "end": 982.28, "text": " we actually studied this in our most recent work, kind of looking at, you know, each expert, what"}, {"start": 982.28, "end": 987.96, "text": " are they specializing in? What are they learning? And interestingly, they kind of specialize in some"}, {"start": 987.96, "end": 992.04, "text": " shallow concepts, which you would think maybe there would be like only really deep things going"}, {"start": 992.04, "end": 996.4399999999999, "text": " on and it would be kind of hard to inspect them. But, you know, we noticed like, oh, there's like"}, {"start": 996.4399999999999, "end": 1002.28, "text": " a punctuation expert or an expert that will, you know, talk about, you know, like proper nouns,"}, {"start": 1002.28, "end": 1007.0, "text": " which we thought was pretty funny and maybe not super intuitive for, you know, how. Yeah."}, {"start": 1007.0, "end": 1010.92, "text": " Yeah. Actually, if you want, you could switch over to the recent paper and we actually have a"}, {"start": 1011.56, "end": 1015.24, "text": " figure which sort of shows some of these things. So you can kind of like follow along and"}, {"start": 1015.24, "end": 1025.4, "text": " see how shallow these things actually are. Yeah. Yeah. So this would be different. So you found an"}, {"start": 1025.4, "end": 1033.08, "text": " expert or in this case, multiple experts that focused on these sort of things."}, {"start": 1034.68, "end": 1041.08, "text": " So there's conjunctions, punctuation, verb, visual description, which is interesting because that's"}, {"start": 1041.08, "end": 1050.4399999999998, "text": " kind of, I want to say like a higher level thing than just the punctuation, right? Counting numbers."}, {"start": 1050.4399999999998, "end": 1054.6799999999998, "text": " Yeah. How do you make sense of this stuff? Like what's going on?"}, {"start": 1058.36, "end": 1063.96, "text": " I, yeah, I mean, I think we were sort of expecting maybe like a higher level of description,"}, {"start": 1063.96, "end": 1071.96, "text": " but like, or like sort of like representation. It's, I think we've just started, started to sort"}, {"start": 1071.96, "end": 1077.48, "text": " of like crack and like look into these models to actually see what's going on. That obviously like"}, {"start": 1077.48, "end": 1082.44, "text": " one big specialization that you're seeing here are these Sentinel tokens. To make sense of that,"}, {"start": 1082.44, "end": 1086.8400000000001, "text": " we were sort of doing pre-training where it sort of fill in the blank task and a blank is sort of"}, {"start": 1086.8400000000001, "end": 1091.48, "text": " represented by these like little Sentinels. So like extra ID 10 represents like, you know,"}, {"start": 1091.48, "end": 1098.76, "text": " the blank 10. And we often really frequently see experts sort of specializing on these blanks."}, {"start": 1100.3600000000001, "end": 1105.08, "text": " So that's sort of an interesting thing. And then I think that also might segue into maybe you want"}, {"start": 1105.08, "end": 1109.8, "text": " to actually given this sort of like, you know, observe specialization, maybe you actually want to"}, {"start": 1109.8, "end": 1115.56, "text": " make some experts higher capacity or give them more compute to sort of do things that might be"}, {"start": 1115.56, "end": 1122.28, "text": " harder. But honestly, I mean, this is still very early. It'd be interesting for sort of like, you"}, {"start": 1122.28, "end": 1127.1599999999999, "text": " know, some of the interpretability lens that like Anthropic has on some of the recent Transformers"}, {"start": 1127.1599999999999, "end": 1132.36, "text": " to be applied to like some sparse expert models. Some questions we've kind of received are,"}, {"start": 1132.36, "end": 1137.32, "text": " what is the interplay of expert specialization with sort of like self-attention specialization?"}, {"start": 1137.32, "end": 1143.32, "text": " And that's honestly completely open. I think we were just sort of putting this table forth to the"}, {"start": 1143.32, "end": 1149.8, "text": " community to be like, well, we started. It's not exactly what we would have expected, but definitely"}, {"start": 1149.8, "end": 1155.1599999999999, "text": " kind of like a call to dig further and hopefully like, you know, further improve things."}, {"start": 1156.84, "end": 1162.04, "text": " With the also, I believe that this was, oh, yeah, here already in Switch Transformers,"}, {"start": 1162.04, "end": 1171.3999999999999, "text": " this ability to distribute these things across devices that comes naturally with having sparse"}, {"start": 1171.4, "end": 1177.48, "text": " experts. So sparsity, meaning in this case, I only send stuff to one or a few experts. And"}, {"start": 1178.3600000000001, "end": 1187.88, "text": " there came the ability to shard this across devices. How practical is this really to like,"}, {"start": 1190.0400000000002, "end": 1196.92, "text": " when would I do something like this? At what point would it become practical and useful and the best"}, {"start": 1196.92, "end": 1205.0800000000002, "text": " thing to do to communicate across devices for my experts? Yeah, so really great question. And I"}, {"start": 1205.0800000000002, "end": 1211.0800000000002, "text": " actually think this is the reason why the method works so well, actually. So the standard way I"}, {"start": 1211.0800000000002, "end": 1214.76, "text": " would say people are doing distributed training of these models is they have, you know, either"}, {"start": 1214.76, "end": 1218.8400000000001, "text": " fully data parallelism, which means like, you know, each machine has the same set of weights,"}, {"start": 1218.8400000000001, "end": 1222.8400000000001, "text": " but different slices of data or a blend of data and model parallelism where it's like, you know,"}, {"start": 1222.84, "end": 1227.6399999999999, "text": " kind of a mix where certain like, you know, cores have sometimes different weights or sometimes"}, {"start": 1227.6399999999999, "end": 1231.08, "text": " different data. And then you communicate stuff to make it, you know, emulate like a full model."}, {"start": 1231.9599999999998, "end": 1238.28, "text": " But I think experts, one really easy interpretation of this is like, let's say you have a model and,"}, {"start": 1238.28, "end": 1241.72, "text": " you know, you're using data parallelism and you have four different machines."}, {"start": 1242.6, "end": 1247.9599999999998, "text": " A really natural way to overlay experts on this would be you just have one expert per machine."}, {"start": 1247.96, "end": 1252.44, "text": " And then, yeah, so this is like a really nice interpretation because then when you have all"}, {"start": 1252.44, "end": 1257.32, "text": " of your, you know, local data per core, you'd have the router weights replicated,"}, {"start": 1257.72, "end": 1261.88, "text": " but then you just figure out what expert they need to go to. And then that's when you kind of,"}, {"start": 1261.88, "end": 1266.6000000000001, "text": " you know, shuffle all the tokens around to the machines, do all the computation and then shuffle"}, {"start": 1266.6000000000001, "end": 1274.28, "text": " them back. And this makes it really nice because then per machine, you actually never have any more"}, {"start": 1274.28, "end": 1278.28, "text": " parameters than you would have had just with a dense transformer, but now you have experts."}, {"start": 1279.0, "end": 1283.6399999999999, "text": " So it's actually like a really nice way of kind of, you know, thinking about how to design the"}, {"start": 1283.6399999999999, "end": 1287.56, "text": " models would be like, oh, you know, you have this many cores for data parallelism, just have that"}, {"start": 1287.56, "end": 1292.44, "text": " many experts. And that's actually a paradigm that Liem and I use a lot when designing these models"}, {"start": 1292.44, "end": 1299.08, "text": " as well. And yeah, I mean, I think as soon as you have this sort of like distributed model where"}, {"start": 1299.08, "end": 1304.4399999999998, "text": " you're already going across accelerators and devices, you do already have these communication"}, {"start": 1304.4399999999998, "end": 1309.3999999999999, "text": " patterns, right? Like you need to get activations to a certain place, you need to like get gradients"}, {"start": 1309.3999999999999, "end": 1314.9199999999998, "text": " to a certain place. So you already have these sort of like all reduced communication collectives."}, {"start": 1316.9199999999998, "end": 1321.8799999999999, "text": " Expert model is going to introduce all to all communication patterns. So that can be like a more"}, {"start": 1321.8799999999999, "end": 1326.84, "text": " expensive thing, especially based on like your topology and the bandwidth between all of your"}, {"start": 1326.84, "end": 1332.84, "text": " networks, or between all of your devices. But yeah, so I mean, this is something you sort of"}, {"start": 1332.84, "end": 1341.08, "text": " have to like, kind of empirically test like, okay, how how much does this architecture kind of buy"}, {"start": 1341.08, "end": 1346.84, "text": " you in terms of performance on your task, versus the additional cost of all to all communication?"}, {"start": 1347.56, "end": 1352.6, "text": " But you will be communicating across devices for these big models, regardless to to train them."}, {"start": 1352.6, "end": 1361.0, "text": " Yeah. So this is a good, I guess, a good segue, because you can achieve these giant models,"}, {"start": 1361.0, "end": 1367.48, "text": " like trillions of parameters using these sparse expert models, because naturally, I can"}, {"start": 1367.48, "end": 1372.9199999999998, "text": " parallelize these experts, it doesn't cost me really much more compute, because any data point,"}, {"start": 1372.9199999999998, "end": 1379.7199999999998, "text": " or any token only goes to one single expert. There is always a bit of the, let's say,"}, {"start": 1379.72, "end": 1385.08, "text": " the question of how comparable this is to the dense models. It was it was often I don't know if"}, {"start": 1385.08, "end": 1390.28, "text": " this is a latent feeling that I get from the community, but people would rather have the"}, {"start": 1390.28, "end": 1397.96, "text": " 175 billion GPT-3 model compared to the switch transformer, even if it is trillions of parameters."}, {"start": 1398.6000000000001, "end": 1406.3600000000001, "text": " What is there some sort of division factor where I could compare to a dense model? Or do you think"}, {"start": 1406.36, "end": 1410.28, "text": " that it's an entirely different nature of a function that's computed here?"}, {"start": 1411.0, "end": 1416.12, "text": " Yeah, so this is a really great question. And I think there's a lot of different ways you have to"}, {"start": 1416.12, "end": 1420.36, "text": " kind of look at this to figure out if a sparse model is right for you. So I think actually,"}, {"start": 1420.36, "end": 1424.84, "text": " in a lot of applications, if it's like, hey, I want to train the model with the smallest memory"}, {"start": 1424.84, "end": 1430.9199999999998, "text": " footprint, so I can just be using it on the smallest amount of devices as possible, a dense"}, {"start": 1430.92, "end": 1435.88, "text": " model will always be better. I think on a per parameter basis, dense models are going to be"}, {"start": 1435.88, "end": 1438.8400000000001, "text": " performing better. So for those types of applications, I'm like, yeah, I don't think it"}, {"start": 1438.8400000000001, "end": 1442.68, "text": " makes sense to be using sparse models. If you want to just train the best thing that you can"}, {"start": 1442.68, "end": 1448.68, "text": " fit onto your local 2GPU machine or like a 10GPU machine and do really kind of low throughput,"}, {"start": 1450.28, "end": 1454.68, "text": " feeding in data to this, like not high or anything like that. I think sparse models are good,"}, {"start": 1455.3200000000002, "end": 1459.48, "text": " where you're going to be training a model and you're going to be hosting it on a lot of machines"}, {"start": 1459.48, "end": 1463.64, "text": " and you're going to be having a lot of high throughput going through it. So a lot of queries,"}, {"start": 1463.64, "end": 1466.52, "text": " a lot of stuff going through it, because then things can be batched together and then the"}, {"start": 1466.52, "end": 1471.72, "text": " models actually become pretty efficient. So I think that's kind of one lens to look at when"}, {"start": 1471.72, "end": 1477.16, "text": " you would want to use a sparse versus dense model. And I think the kind of second lens is that,"}, {"start": 1477.88, "end": 1484.3600000000001, "text": " for a given amount of GPU or TPU hours on a compute cluster, what model will get you the"}, {"start": 1484.3600000000001, "end": 1488.52, "text": " best performance? And I think that's the lens that we actually would spend a lot of time looking at"}, {"start": 1488.52, "end": 1493.96, "text": " for pre-training models in this paper. Like, oh, you have 512 TPU chips and I give you,"}, {"start": 1493.96, "end": 1498.28, "text": " you know, X budget training hours, is a dense model or sparse model going to give you the best"}, {"start": 1498.28, "end": 1502.52, "text": " pre-training performance? And I think our assessment was that, yeah, I think actually"}, {"start": 1502.52, "end": 1506.68, "text": " the Pareto optimal model typically is a sparse model in that setup."}, {"start": 1508.68, "end": 1513.6399999999999, "text": " Yeah. And like comparing parameters, especially between the dense and the sparse model is just,"}, {"start": 1513.64, "end": 1518.76, "text": " you know, totally incomparable. So using like GPT-3 and then like our largest like"}, {"start": 1518.76, "end": 1523.64, "text": " switch transformer model, it's just wildly different amount of computes in our case."}, {"start": 1523.64, "end": 1530.2800000000002, "text": " You can't infer that from the parameter budget. So I don't know what the like the compute ratio"}, {"start": 1530.2800000000002, "end": 1536.76, "text": " was between the two, but far different. Our 1.6 trillion parameter model was actually only doing"}, {"start": 1536.76, "end": 1542.2800000000002, "text": " about as much compute as a billion parameter model. So for each token, it was doing, you know,"}, {"start": 1542.28, "end": 1547.32, "text": " it was doing roughly a billion parameters worth of flops. And, you know, whereas GPT-3 is doing"}, {"start": 1547.32, "end": 1553.72, "text": " 175 billion parameters worth of flops. So you can sort of tune this and DeepMind has sort of"}, {"start": 1553.72, "end": 1559.72, "text": " also tried to come up with like a characterization of scaling properties, far more like robust than"}, {"start": 1559.72, "end": 1565.6399999999999, "text": " we've been able to do, of sparse expert models and try to come up with like, you know, a dense"}, {"start": 1565.6399999999999, "end": 1570.36, "text": " model equivalent. So like that might be like an interesting word to sort of like refer to in the"}, {"start": 1570.36, "end": 1574.9199999999998, "text": " future. But really, it's just like, you know, practically speaking, it's like, OK, I give you"}, {"start": 1574.9199999999998, "end": 1581.4799999999998, "text": " these accelerators for this amount of time. What's like the best model? So that's like probably the"}, {"start": 1581.4799999999998, "end": 1591.9599999999998, "text": " fairest comparison. Have you seen this pathways paper? Yes, definitely. They came out like,"}, {"start": 1591.9599999999998, "end": 1597.1599999999999, "text": " how does it how does it play into something like this? Is it going to make is it going to make this"}, {"start": 1597.16, "end": 1604.6000000000001, "text": " easier? Is it going to make it superfluous? Like, how does the the sort of ability to schedule"}, {"start": 1604.6000000000001, "end": 1611.64, "text": " things heterogeneously across devices? Or does it does it enable new possibilities in the sparse"}, {"start": 1611.64, "end": 1618.1200000000001, "text": " expert world? Yeah, so great question. So, so one thing to note is like, OK, so typically, you have"}, {"start": 1618.1200000000001, "end": 1622.2, "text": " dense models and a dense model like every input will have the same amount of compute and parameters"}, {"start": 1622.2, "end": 1627.24, "text": " applied to it. And sparse models now you have the same amount of compute, but different parameters."}, {"start": 1627.24, "end": 1631.4, "text": " And I think the kind of natural next step that I think makes a lot of sense to both Liam and I is"}, {"start": 1631.4, "end": 1635.96, "text": " that now for each input, you have a different amount of compute applied as well. And I think"}, {"start": 1635.96, "end": 1640.68, "text": " pathways is really exciting, again, like you kind of mentioned for like the heterogeneous compute,"}, {"start": 1640.68, "end": 1644.04, "text": " where we want to have inputs that might require, you know, different parameters and also different"}, {"start": 1644.04, "end": 1647.96, "text": " amounts of compute. Yeah, and I think, you know, a framework like this is going to really open up"}, {"start": 1647.96, "end": 1652.6000000000001, "text": " like a lot of really exciting research avenues along that direction. And I think it feels like"}, {"start": 1652.6000000000001, "end": 1656.44, "text": " a very natural interpretation for kind of where our models are headed for in the future."}, {"start": 1658.52, "end": 1662.92, "text": " Yeah, like right now, it's like our experts are all sort of completely homogenous. They're all"}, {"start": 1662.92, "end": 1668.04, "text": " the same size. They do the same operations pathways. You could be like, oh, this is like"}, {"start": 1668.04, "end": 1674.28, "text": " a recurrent expert. This is a huge expert. There's a group of small experts. You could just be a lot"}, {"start": 1674.28, "end": 1680.36, "text": " more flexible in design. And like, you know, sort of like alluding to that a little bit with when"}, {"start": 1680.36, "end": 1684.52, "text": " we were still looking at the visualization, it's like, oh, wow, a really consistent thing are"}, {"start": 1684.52, "end": 1688.76, "text": " experts that want to specialize in these like fill in the blank tokens, these Sentinel tokens."}, {"start": 1689.8799999999999, "end": 1694.44, "text": " Perhaps that might be an avenue or an area where it's like, oh, let's dramatically increase the"}, {"start": 1694.44, "end": 1706.04, "text": " compute here. This is like an area where we like a lot of extra compute could really be helpful."}, {"start": 1706.04, "end": 1711.0, "text": " And there wasn't really an effective way to do this with the existing infrastructures before"}, {"start": 1711.0, "end": 1725.56, "text": " pathways. Yeah, sorry, that's lost the train of thought. Explain to me a little bit how GLAM"}, {"start": 1725.56, "end": 1730.68, "text": " improved upon switch transformers. Like what's new? What's exciting there?"}, {"start": 1732.52, "end": 1737.8, "text": " Yeah, so I think GLAM, so one also thing to note is like there's kind of a right now division of"}, {"start": 1737.8, "end": 1742.52, "text": " two different types of model classes in like language modeling space, I would say. So one is"}, {"start": 1742.52, "end": 1746.9199999999998, "text": " like these decoder only models where, you know, it's just, you know, a single set of parameters"}, {"start": 1746.9199999999998, "end": 1752.12, "text": " and it's like, you're just predicting the next token like auto aggressively. And this is like,"}, {"start": 1752.12, "end": 1756.76, "text": " you know, what GPT-3 is. And this is also the kind of architecture that GLAM studies these models in."}, {"start": 1757.32, "end": 1762.2, "text": " So the other classes, these like encoder decoder models like T5, this was also G-shard. This is"}, {"start": 1762.2, "end": 1767.56, "text": " kind of what also we studied in switch transformer in our most recent work as well. So I think"}, {"start": 1767.56, "end": 1773.08, "text": " GLAM did a few things. So one, they really, I think pushed the scale of these models. So like,"}, {"start": 1773.08, "end": 1776.6799999999998, "text": " while, you know, our original model in switch transformer had more parameters, like GLAM had"}, {"start": 1776.6799999999998, "end": 1781.8, "text": " like much more compute applied per token. And they studied these very extensively with decoder"}, {"start": 1781.8, "end": 1787.1599999999999, "text": " only language models. And yeah, I think their main comparison point was to GPT-3 as well. So"}, {"start": 1787.1599999999999, "end": 1792.2, "text": " they were studying a lot in the context of few shot and like one shot evaluations. Whereas I"}, {"start": 1792.2, "end": 1796.44, "text": " think a lot of our work actually centered around like fine tuning the models. But yeah, I think"}, {"start": 1796.44, "end": 1800.6000000000001, "text": " GLAM really like pushed the scale of these, especially in these decoder only language"}, {"start": 1800.6000000000001, "end": 1805.72, "text": " models and showed that like, yeah, you know, you can get as good of quality as GPT-3 with like,"}, {"start": 1805.72, "end": 1810.52, "text": " you know, huge computational training savings as well. And that did a really, a lot of really good"}, {"start": 1810.52, "end": 1818.1200000000001, "text": " work in that space. Is there a functional difference between the sparse expert routing"}, {"start": 1818.12, "end": 1826.9199999999998, "text": " or anything around this in GLAM? Or is it mainly what you said with decoder only and"}, {"start": 1826.9199999999998, "end": 1834.6, "text": " applying more compute scaling it up? So actually, there is a few differences that are more nuanced"}, {"start": 1834.6, "end": 1838.84, "text": " than technical. But yeah, at a high level, you know, there's a routing function, and they actually"}, {"start": 1838.84, "end": 1844.12, "text": " route each token to two experts. And actually, there's like some of the differences in these"}, {"start": 1844.12, "end": 1848.36, "text": " models comes from like how much buffer you give each token, each expert, because, you know,"}, {"start": 1848.36, "end": 1853.8, "text": " you need to have like fixed batch sizes for all the experts ahead of time. And so what can happen"}, {"start": 1853.8, "end": 1858.52, "text": " is like, you can't guarantee that like, there's going to be perfect balancing among all the tokens"}, {"start": 1858.52, "end": 1862.9199999999998, "text": " getting sent to experts. So like experts can overflow. And there's this key parameter that"}, {"start": 1862.9199999999998, "end": 1867.6399999999999, "text": " we call the capacity factor. That's probably the single handedly most important parameter when"}, {"start": 1867.6399999999999, "end": 1871.4799999999998, "text": " designing a mixture of expert models, because it just has such a huge impact on the communication"}, {"start": 1871.48, "end": 1876.28, "text": " costs, computing, everything like that for how much buffer you should have. And yeah, I think"}, {"start": 1876.28, "end": 1880.44, "text": " a big difference from GLAM versus our models is they actually use like a much larger capacity"}, {"start": 1880.44, "end": 1886.3600000000001, "text": " factor than we've used in our other works. But yeah, the routing algorithm is essentially the same."}, {"start": 1888.84, "end": 1894.3600000000001, "text": " That is, yeah, I want to get a bit more into the into the routing algorithm in just a bit. But just"}, {"start": 1894.36, "end": 1901.4799999999998, "text": " to end this with the last paper that we've previously looked at, was I right in saying that"}, {"start": 1901.4799999999998, "end": 1909.8, "text": " this is much more often, let's say a general, almost like a review paper? Or how would you describe it?"}, {"start": 1912.1999999999998, "end": 1916.76, "text": " Yeah, I mean, I think we we tried to make sure like we're contextualizing a lot of the work. So"}, {"start": 1916.76, "end": 1921.08, "text": " we try to make sure the related work was like, pretty inclusive, because I mean, I think the"}, {"start": 1921.08, "end": 1927.8, "text": " field's really adjusted and improved a lot in the last two years. But I would sort of characterize"}, {"start": 1927.8, "end": 1933.96, "text": " this paper as fixing the two big flaws from our first one from switch transformers. The first was"}, {"start": 1933.96, "end": 1937.6399999999999, "text": " these models are unstable to train. So you'd be training and then all of a sudden the loss or just"}, {"start": 1937.6399999999999, "end": 1943.72, "text": " diverge, which thwarted a lot of our issues. Interestingly, it doesn't seem like the instability"}, {"start": 1943.72, "end": 1948.6799999999998, "text": " arises from a lot of experts. We were consistently able to train models like our trillion parameter"}, {"start": 1948.68, "end": 1953.96, "text": " model, for instance, with thousands of experts, never really hitting any unstable sections."}, {"start": 1954.52, "end": 1959.5600000000002, "text": " Really, it kind of came from like high quops or high computation expert models, even with like few"}, {"start": 1959.5600000000002, "end": 1964.68, "text": " experts, those were highly unstable. And then the second thing that this paper sort of fixed was"}, {"start": 1964.68, "end": 1970.6000000000001, "text": " the sort of like poor fine tuning quality. So we would sort of pre train a model, it would show like"}, {"start": 1970.6000000000001, "end": 1974.6000000000001, "text": " really significant speed ups over a dense counterpart. But then when it came time to fine"}, {"start": 1974.6, "end": 1979.8799999999999, "text": " tuning, say I'm like super glue or some like other task of interest, it would just be considerably"}, {"start": 1979.8799999999999, "end": 1985.9599999999998, "text": " worse. So I think this paper was just really trying to sort of like kind of patch up a couple"}, {"start": 1985.9599999999998, "end": 1991.3999999999999, "text": " of those issues. We identify them in our first work. Yeah, I'm always a bit intimidated when a"}, {"start": 1991.3999999999999, "end": 2000.36, "text": " paper has a table of index by itself. Can you can you go to something that Barry and I discussed?"}, {"start": 2000.36, "end": 2005.6399999999999, "text": " It's like, okay, should we break this up into multiple papers? Or should this be one? Because,"}, {"start": 2005.6399999999999, "end": 2009.4799999999998, "text": " you know, this is like, you know, a lot of work. And, you know, this is like something that we"}, {"start": 2009.4799999999998, "end": 2014.36, "text": " discussed, like, maybe in the future, we should probably be producing like more bite sized pieces"}, {"start": 2014.36, "end": 2021.32, "text": " of work. When you when you talk about fine tuning, can you go a bit into more detail? Like, what was"}, {"start": 2021.32, "end": 2026.76, "text": " exactly the problem? How did you how did you also go about fixing it? So I'm only interested in,"}, {"start": 2026.76, "end": 2033.8799999999999, "text": " you know, how did how what's the final model like? But what does the process of debugging"}, {"start": 2033.8799999999999, "end": 2038.84, "text": " something like this and then getting to an architecture or a solution that actually works"}, {"start": 2038.84, "end": 2049.16, "text": " look like? Yeah, I mean, it's sort of this like, very interesting problem of like, you want to,"}, {"start": 2050.2, "end": 2053.56, "text": " there's really just like fundamental trade off. And whenever you're sort of doing this sort of"}, {"start": 2053.56, "end": 2058.6, "text": " like large scale work, where you want to try to understand and characterize things at a smaller"}, {"start": 2058.6, "end": 2064.68, "text": " scale, understand scaling properties, understand, understand, like hyper parameter dependencies."}, {"start": 2065.72, "end": 2071.4, "text": " But then you also want to be consistently checking yourself at the largest scales. And this sort of"}, {"start": 2071.4, "end": 2075.7999999999997, "text": " balance of like, okay, you have this much compute, you have this much time, where do you allocate it?"}, {"start": 2075.7999999999997, "end": 2080.92, "text": " Do you do a lot of small experiments? Or do you do a few big experiments? It's kind of tricky."}, {"start": 2080.92, "end": 2088.2000000000003, "text": " But I'd say part of our, like findings were the first one was like, okay, well, characterization"}, {"start": 2088.2000000000003, "end": 2094.52, "text": " is we're not doing better on fine tuning, what's the cause? And it seemed like perhaps our cause"}, {"start": 2094.52, "end": 2099.8, "text": " is not that of optimization, it's that of generalization. So if you scroll down into"}, {"start": 2099.8, "end": 2107.96, "text": " section four, you can just click on the link, we might be Yeah, exactly. Yeah, so this is an example"}, {"start": 2107.96, "end": 2114.28, "text": " an example that, you know, kind of supports a lot of the trends we're seeing. On the left is a small"}, {"start": 2114.28, "end": 2121.08, "text": " superglue task. So this task has only 250 training sequences, so very small. And on the right is"}, {"start": 2121.08, "end": 2128.76, "text": " record. So this has over 100,000 training examples. We're showing sparse models versus dense models"}, {"start": 2130.12, "end": 2136.2, "text": " in the two things in the two plots. Blue represents the sparse train eval. And you can see it just"}, {"start": 2136.2, "end": 2141.3999999999996, "text": " very quickly gets to 100%. And it outpaces in both cases, the small task and the large task"}, {"start": 2142.04, "end": 2148.2799999999997, "text": " outpaces the dense model getting to 100% train evaluation accuracy. But when in the small task,"}, {"start": 2148.2799999999997, "end": 2152.6, "text": " we'll see the dense model in red actually outperforming the ultimate performance for the"}, {"start": 2152.6, "end": 2158.04, "text": " sparse model in orange, whereas for the bigger tasks, the sparse model does well. And so we kind"}, {"start": 2158.04, "end": 2164.2, "text": " of kept seeing this like, you know, overfitting issues. And a lot of this was then led us to sort"}, {"start": 2164.2, "end": 2168.6, "text": " of like investigate hyperparameters. And, you know, some of the hyperparameters can sort of be"}, {"start": 2168.6, "end": 2175.16, "text": " adjusted in a way to make the model like less susceptible to overfitting. So you can use like"}, {"start": 2175.16, "end": 2180.9199999999996, "text": " different dropout parameterizations, but also things like batch size and learning rate can"}, {"start": 2180.9199999999996, "end": 2186.4399999999996, "text": " inject more noise, which can also be sort of like a counter to some like overfitting properties."}, {"start": 2187.56, "end": 2192.3599999999997, "text": " So we tried and then sort of consistent with this, like a lot of these things were sort of like,"}, {"start": 2192.36, "end": 2197.6400000000003, "text": " you know, more exhaustive studies at, say, a billion parameter scale. We then tried to"}, {"start": 2197.6400000000003, "end": 2202.36, "text": " continue to sort of like fact check this against our larger model and make sure that these"}, {"start": 2202.36, "end": 2207.32, "text": " conclusions were holding. So I think it was just sort of like the debugging process was,"}, {"start": 2207.32, "end": 2212.92, "text": " OK, what more precisely is going wrong? And then like, what are our levers that we can sort of like"}, {"start": 2212.92, "end": 2217.88, "text": " pull in order to try to improve it? But, you know, a bit of art and science, really."}, {"start": 2217.88, "end": 2226.12, "text": " You so you is it you observed, OK, we are probably overfitting because you saw the smaller the tasks"}, {"start": 2226.12, "end": 2232.36, "text": " got sort of the worst, the sparse models would ultimately perform on the validation set of those"}, {"start": 2232.36, "end": 2239.0, "text": " tasks. Did you and you have it's not like quite like, yeah, it's not always like quite so easy"}, {"start": 2239.0, "end": 2242.52, "text": " as that. But it's sort of like, you know, directionally, like I think we have support of"}, {"start": 2242.52, "end": 2246.2000000000003, "text": " the hypothesis, but it's not like every single small task does poorly."}, {"start": 2246.2, "end": 2251.3999999999996, "text": " Every large task is great. Yeah, but I'd say directionally, it seems to be a phenomenon we've"}, {"start": 2251.3999999999996, "end": 2258.9199999999996, "text": " observed. You have also a bunch of experiments down here where you where you investigate some of"}, {"start": 2258.9199999999996, "end": 2263.72, "text": " these, for example, dropout probabilities. You also have expert dropout probability, which is"}, {"start": 2263.72, "end": 2269.96, "text": " one of the questions I had in that you have a particular architecture right with these with"}, {"start": 2269.96, "end": 2275.24, "text": " these experts. And when I think about overfitting, what in regular transformers, you know,"}, {"start": 2275.24, "end": 2281.3199999999997, "text": " regular transformers, I have kind of handles I can use adapter layers, I can only fine tune the"}, {"start": 2281.3199999999997, "end": 2288.9199999999996, "text": " head and so on. Did you ever investigate maybe only fine tuning some of the experts? Like, is"}, {"start": 2288.9199999999996, "end": 2297.24, "text": " that the keeping others constant? Is that ever a thing? Like, would that work or or can we can we"}, {"start": 2297.24, "end": 2301.24, "text": " make use somehow of the fact that we have these different experts and they're actually different"}, {"start": 2301.24, "end": 2306.7599999999998, "text": " functions? Yeah, great question. And I think actually, if you scroll down, we did a very naive"}, {"start": 2306.7599999999998, "end": 2311.08, "text": " kind of version of this not where we freeze different experts. But we you know, freeze all"}, {"start": 2311.08, "end": 2316.2, "text": " of the experts or maybe only train all the experts and freeze all of the other parameters. I would"}, {"start": 2316.2, "end": 2324.3599999999997, "text": " say our findings were this were surprising in a bad way. So nothing, nothing really worked super"}, {"start": 2324.3599999999997, "end": 2329.24, "text": " well. So here you can see like, and this is also we only studied this on superglue, right? So it's"}, {"start": 2329.24, "end": 2335.16, "text": " far from exhaustive. But yeah, so one thing we tried was updating first all of the non mixture"}, {"start": 2335.16, "end": 2338.7599999999998, "text": " of expert parameters only. And that actually performed about the same, which was kind of"}, {"start": 2338.7599999999998, "end": 2342.68, "text": " interesting. It's like, hey, like actually freezing the mixture of expert weights like seem to"}, {"start": 2342.68, "end": 2348.3599999999997, "text": " perform about as well as just like updating the whole model. Then when we started to, you know,"}, {"start": 2348.3599999999997, "end": 2351.4799999999996, "text": " update only the mixture of expert weights and freeze all the other model parameters, like the"}, {"start": 2351.4799999999996, "end": 2355.9599999999996, "text": " performance was actually really bad. And there was some we still fully don't understand what's"}, {"start": 2355.96, "end": 2361.7200000000003, "text": " going on here. We have like a few kind of like half baked hypotheses, but yeah, then when we update"}, {"start": 2361.7200000000003, "end": 2366.6, "text": " only the attention parameters, things are worse. And we found a slight boost updating only the"}, {"start": 2366.6, "end": 2371.2400000000002, "text": " feed forward network parameters that weren't the mixture of expert layers. But yeah, overall,"}, {"start": 2371.2400000000002, "end": 2376.12, "text": " nothing worked that well. But yeah, I think there might be some potential really interesting things"}, {"start": 2376.12, "end": 2380.28, "text": " of like, hey, maybe allowing only, you know, a certain subset of experts to be fine tuned."}, {"start": 2380.28, "end": 2386.1200000000003, "text": " We did spend a little bit of time actually studying like pruning off experts during fine tuning."}, {"start": 2387.0, "end": 2391.5600000000004, "text": " So like for a specific fine tuning task, if your pre-trained model has like 64 experts,"}, {"start": 2391.5600000000004, "end": 2394.92, "text": " can you just take like a subset of like two, four, eight or 16 of them?"}, {"start": 2396.1200000000003, "end": 2398.92, "text": " Yeah. And we also didn't really get that good of signal with this as well."}, {"start": 2401.2400000000002, "end": 2404.84, "text": " Also to some of your recommendations, they actually would be compatible with expert"}, {"start": 2404.84, "end": 2411.4, "text": " models too. So you're free to just like fine tune like the top like top logit layer or you could"}, {"start": 2411.4, "end": 2415.56, "text": " add in adapter layers. Yeah, we didn't do anything like really funky like you were suggesting like,"}, {"start": 2415.56, "end": 2419.56, "text": " oh, we're only going to expert like update experts like three, eight and 14 or something."}, {"start": 2421.48, "end": 2425.08, "text": " Yeah, my intuition is that probably wouldn't work well. But I mean,"}, {"start": 2425.08, "end": 2431.48, "text": " I've been proven wrong many times. Yeah, we tried some like other things that didn't make it to"}, {"start": 2431.48, "end": 2438.04, "text": " this table or these plots. And yeah, again, we didn't really see like a significant boost."}, {"start": 2438.04, "end": 2442.04, "text": " That said, if you are only updating like a fraction of the parameters that you get some"}, {"start": 2442.04, "end": 2452.84, "text": " memory savings. So, you know, some nice things. Cool. I guess one, you know, there's almost an"}, {"start": 2452.84, "end": 2458.36, "text": " infinite number of things one could try with these things like distilling experts, like distilling"}, {"start": 2458.36, "end": 2465.48, "text": " multiple experts into a single expert. So you have another expert that's again free to do some"}, {"start": 2465.48, "end": 2472.04, "text": " some new task once you know that two experts are converging something like, I think there's it's"}, {"start": 2472.04, "end": 2476.36, "text": " really interesting, right? A lot of a lot of we're adding a new experts on the fly. Yeah,"}, {"start": 2476.36, "end": 2483.0, "text": " a lot of possibilities. And that brings me a bit to this routing function that we talked about"}, {"start": 2483.0, "end": 2490.76, "text": " before. And at the beginning, which seems to me is a really crucial part of the system. Yet,"}, {"start": 2490.76, "end": 2497.96, "text": " as you said before, very often, I've just seen this being implemented quite simplistically,"}, {"start": 2497.96, "end": 2503.96, "text": " maybe there's a linear transform, and then a softmax or something like this, maybe not even"}, {"start": 2503.96, "end": 2510.2, "text": " maybe there is some some sort of a, you know, you have some fixed keys for all of the experts,"}, {"start": 2510.2, "end": 2518.12, "text": " and then you route according to that. Do you like my intuition would be that this could be a powerful"}, {"start": 2518.12, "end": 2524.9199999999996, "text": " handle on what's, you know, on my performance downstream, this this routing function, especially"}, {"start": 2524.9199999999996, "end": 2532.4399999999996, "text": " also making this different during inference, you know, any any number of things, doing a Monte"}, {"start": 2532.4399999999996, "end": 2537.64, "text": " Carlo tree search at inference time to get to be as accurate as possible, kind of like, like"}, {"start": 2537.64, "end": 2544.6, "text": " AlphaGo or something. Do you have an idea on what the power of the routing function in these sparse"}, {"start": 2544.6, "end": 2549.56, "text": " models is? And how does it work currently? Like, what's the most the latest and greatest? And"}, {"start": 2550.68, "end": 2556.2799999999997, "text": " how good is it? Yeah, so this is a really good question, actually, and something we've actually"}, {"start": 2556.2799999999997, "end": 2560.04, "text": " spent a lot of time about. So I would say actually, in this project, probably the thing I"}, {"start": 2560.04, "end": 2563.08, "text": " maybe spent the most time with is trying out different routing algorithms and routing"}, {"start": 2563.08, "end": 2568.44, "text": " parameterizations. But we ended up kind of going with the default thing, which I also think says"}, {"start": 2568.44, "end": 2576.2799999999997, "text": " something a little bit about the results of it. Yeah, so I would say my intuition is that the"}, {"start": 2576.2799999999997, "end": 2582.2799999999997, "text": " model actually works surprisingly well with a lot of different ways you can route the tokens. So"}, {"start": 2582.2799999999997, "end": 2585.72, "text": " like, you know, we tried a lot of other routing algorithms, we tried making like the routing"}, {"start": 2585.72, "end": 2590.44, "text": " network larger, we tried like, you know, some fancier ways of actually figuring out where you"}, {"start": 2590.44, "end": 2595.08, "text": " should send the token to. We tried, you know, using additional information of like, oh, when you're"}, {"start": 2595.08, "end": 2599.8, "text": " routing this current representation, you have access to whether or not like it was routed,"}, {"start": 2599.8, "end": 2604.44, "text": " or like where it was routed before in previous layers, using like word embedding information too."}, {"start": 2606.92, "end": 2611.7200000000003, "text": " But yeah, I think overall, it seemed to be, you know, kind of insensitive. We actually did find"}, {"start": 2611.7200000000003, "end": 2616.84, "text": " like one or two methods that improve things, but they could only be used in certain situations,"}, {"start": 2616.84, "end": 2623.0, "text": " so it was a bit trickier to just like replace everything. The current routing algorithm we're"}, {"start": 2623.0, "end": 2628.92, "text": " using is basically what the original one was doing, I think in Shazer et al in 2017, when"}, {"start": 2628.92, "end": 2634.04, "text": " these kind of things were like really introduced into the LSTM language models. And I think, you"}, {"start": 2634.04, "end": 2639.4, "text": " know, our newer work, and then also GLAM as well, we're using these kind of routing algorithms too."}, {"start": 2641.88, "end": 2645.8, "text": " Yeah, and also like one kind of like detail here, it's like, so right now we're sort of"}, {"start": 2645.8, "end": 2652.84, "text": " splitting out this little box and we're like, oh, this is the router. It's not really an accurate"}, {"start": 2652.84, "end": 2657.5600000000004, "text": " characterization. It's like, yes, okay, you're mapping some vector into a vector that has like"}, {"start": 2657.5600000000004, "end": 2663.5600000000004, "text": " the same like length as number of experts. But if you just don't update that matrix,"}, {"start": 2664.44, "end": 2669.32, "text": " it still works fine, right? Because now just the represent, like the weight matrices below you"}, {"start": 2669.32, "end": 2673.6400000000003, "text": " are just sort of adapting and just piping whatever activation they need, right? If you freeze the"}, {"start": 2673.64, "end": 2678.92, "text": " great, if you stop a gradient through that, then it's like catastrophically bad. But yeah, I mean,"}, {"start": 2678.92, "end": 2685.0, "text": " I've also sort of been surprised by the relative insensitivity to the routing algorithm. Like"}, {"start": 2685.0, "end": 2689.56, "text": " we've seen like, you know, maybe some small boost here and there, but it hasn't been super"}, {"start": 2689.56, "end": 2694.8399999999997, "text": " significant. I think you'd probably have a better, sort of like a bigger significance"}, {"start": 2694.8399999999997, "end": 2700.04, "text": " by actually just sort of fundamentally changing like the architecture. Like maybe there's like"}, {"start": 2700.04, "end": 2705.56, "text": " some wildly different approach for sort of sparse models that we're not considering. Maybe we're in"}, {"start": 2705.56, "end": 2710.7599999999998, "text": " some sort of like local min and like these small tweaks, I'm like, oh, okay, precisely how are we"}, {"start": 2710.7599999999998, "end": 2715.56, "text": " doing this? Maybe it doesn't matter as much. And DeepMind has also explored some other kind of"}, {"start": 2715.56, "end": 2719.88, "text": " interesting routing algorithms. Like you sort of alluded to fixed routing algorithms where it's"}, {"start": 2719.88, "end": 2724.7599999999998, "text": " just like, you're not even learning. They've also tried RL based routing algorithms. And I think it"}, {"start": 2724.7599999999998, "end": 2728.84, "text": " had like actually similar scaling properties. So again, kind of corroborating what Barrett is saying,"}, {"start": 2728.84, "end": 2732.84, "text": " it's just like a lot of these things when we're kind of doing this like per token routing,"}, {"start": 2733.48, "end": 2738.44, "text": " haven't really moved the needle substantially. That's been our luck."}, {"start": 2738.44, "end": 2743.08, "text": " Yeah. And I think another important trend actually is that when we were experimenting with a lot of"}, {"start": 2743.08, "end": 2746.92, "text": " these different routing algorithms, we actually found that they did help models. And maybe when"}, {"start": 2746.92, "end": 2752.1200000000003, "text": " you had like a 1 billion parameter dense model-ish size, but then like as we scaled up the models,"}, {"start": 2752.1200000000003, "end": 2755.8, "text": " like actually a lot of the times, sometimes the differences would just like wash away as well. So"}, {"start": 2755.8, "end": 2759.96, "text": " it's kind of this interesting effect of when more scale is increased, like it maybe becomes a little"}, {"start": 2759.96, "end": 2769.6400000000003, "text": " bit less insensitive to some of these decisions. Yeah, I can totally see that essentially that"}, {"start": 2769.6400000000003, "end": 2775.0800000000004, "text": " the rest of the network adjusts, especially if everything is trainable. What I would be excited"}, {"start": 2775.0800000000004, "end": 2780.6800000000003, "text": " about maybe is to somehow at inference time doing something smarter than, because at training time,"}, {"start": 2780.6800000000003, "end": 2785.6400000000003, "text": " I can adjust to everything, right? But at inference time, maybe there's something that I could"}, {"start": 2785.64, "end": 2791.64, "text": " do, especially with regards to domain shift, domain adaptation, anything like this where"}, {"start": 2793.16, "end": 2799.0, "text": " I could tweak routing in some way, but I guess that's also up for future work."}, {"start": 2800.12, "end": 2804.2, "text": " So there's a little bit of this, not tweaking the routing algorithm, but tweaking the capacity factor"}, {"start": 2804.2, "end": 2808.3599999999997, "text": " hyperparameter I mentioned a while ago. So this is basically the parameter that's going to dictate"}, {"start": 2808.3599999999997, "end": 2813.0, "text": " how many tokens are being dropped. And one cool thing you can do is you can have some capacity"}, {"start": 2813.0, "end": 2816.68, "text": " factor during training, but then at eval time, depending on if you want to use more or less"}, {"start": 2816.68, "end": 2820.84, "text": " compute, you can be either dropping more or less tokens and either increase or decrease"}, {"start": 2820.84, "end": 2824.6, "text": " the performance, which is pretty cool. And the model's actually pretty robust to having that"}, {"start": 2824.6, "end": 2829.96, "text": " train from train and evaluation time. So that's actually kind of like a good lever for depending"}, {"start": 2829.96, "end": 2836.68, "text": " on if you want to use more or less compute during evaluation. I think we have a pretty good overview"}, {"start": 2836.68, "end": 2845.0, "text": " now. I want to get a little bit into just the future prospects maybe also of this. We already"}, {"start": 2845.0, "end": 2851.56, "text": " talked about with pathways, we could have heterogeneous things. Could this be pushed to"}, {"start": 2852.52, "end": 2858.7599999999998, "text": " some sort of limit? Whenever I see a distributed system, I immediately think distributed, maybe not"}, {"start": 2858.76, "end": 2867.88, "text": " even in a data center, but across users, across networks. Is there applications to maybe, what was"}, {"start": 2867.88, "end": 2873.7200000000003, "text": " it called? Federated, some kind of federated computing, some kind of federated learning where"}, {"start": 2873.7200000000003, "end": 2881.48, "text": " I could somehow contribute with my maybe confidential data, but I could still contribute to a whole"}, {"start": 2881.48, "end": 2887.7200000000003, "text": " compute process. Is there, I'm going to say the B word, is there an application for blockchain"}, {"start": 2887.72, "end": 2895.08, "text": " distribution, something like this? Do you think about the higher degrees of distribution here?"}, {"start": 2895.7999999999997, "end": 2896.68, "text": " Do you want me to go for it?"}, {"start": 2896.68, "end": 2901.9599999999996, "text": " Yeah, go for it. Me personally, I haven't spent a ton of time thinking about this,"}, {"start": 2901.9599999999996, "end": 2907.3999999999996, "text": " but I do think it's very interesting. And yeah, there definitely seems to be a lot of really"}, {"start": 2908.6, "end": 2912.04, "text": " open problems around this, especially given the growing amount of fragmented compute,"}, {"start": 2912.04, "end": 2917.08, "text": " fragmented devices. There's so much compute around here. How can you effectively utilize"}, {"start": 2917.08, "end": 2921.88, "text": " all of this, utilize different data and stuff? I think it's a super cool, and I think it was going"}, {"start": 2921.88, "end": 2926.92, "text": " to require a lot of really interesting research because right now the way we're currently training"}, {"start": 2926.92, "end": 2932.52, "text": " these models is it's all synchronized lockstep typically. You're doing like, oh, after each"}, {"start": 2932.52, "end": 2936.2799999999997, "text": " batch you do these gradients, you send the gradients around and everything. But I think"}, {"start": 2936.2799999999997, "end": 2940.68, "text": " actually maybe the future of these models when you're really allowing them to be distributed"}, {"start": 2940.68, "end": 2944.92, "text": " across very different types of computing, everything might actually now introduce asynchronous"}, {"start": 2944.92, "end": 2950.04, "text": " training as kind of like the new paradigm. So I think that's a really exciting space,"}, {"start": 2950.04, "end": 2952.12, "text": " but yeah, I haven't spent too much time thinking about it personally."}, {"start": 2953.64, "end": 2959.0, "text": " Yeah. And I think as it pertains to say blockchain or something, I think one problem with these"}, {"start": 2959.0, "end": 2965.64, "text": " expert models as designed in this way are these all to all communications. So over this sort of"}, {"start": 2966.2000000000003, "end": 2972.6, "text": " decentralized peer-to-peer network where nodes are really far apart, inconsistent bandwidth and"}, {"start": 2972.6, "end": 2980.2799999999997, "text": " stuff, that could be really tough if sort of your experts were sort of distributed among many"}, {"start": 2980.2799999999997, "end": 2985.64, "text": " different nodes in this sort of like unreliable network where nodes are kind of coming and going."}, {"start": 2985.64, "end": 2992.7599999999998, "text": " Like right now, all our systems are in this sort of like very constrained fault intolerant area"}, {"start": 2992.7599999999998, "end": 2998.92, "text": " where it's like, oh, all highly internet work chips that are highly reliable. And then so"}, {"start": 2998.92, "end": 3002.52, "text": " like blockchain would just have like a whole different set of like kind of problems that you'd"}, {"start": 3002.52, "end": 3008.6800000000003, "text": " have to sort of address like unreliability and some of these other areas. Not to say, I think"}, {"start": 3008.6800000000003, "end": 3012.92, "text": " it just like requires some like additional kind of research, like just sort of adopting the model as"}, {"start": 3012.92, "end": 3019.08, "text": " is, I think would pretty poorly map on that kind of computing infrastructure. But I think there's"}, {"start": 3019.08, "end": 3026.92, "text": " something there that could be done. Is there work on because I see these works mostly here in NLP,"}, {"start": 3026.92, "end": 3032.84, "text": " yet transformers kind of taking over the rest of the world. Is there work on how these"}, {"start": 3033.4, "end": 3039.56, "text": " experts, sparse expert transformers behave in vision, in reinforcement learning, speech, whatever?"}, {"start": 3040.6, "end": 3044.76, "text": " Yeah, yeah, great question. So absolutely, actually, there's been some really good work"}, {"start": 3044.76, "end": 3049.4, "text": " applying these models to like the IT based, like image classification and stuff. And there,"}, {"start": 3049.4, "end": 3053.48, "text": " it's actually really nice, because then you can leverage all of the, you know, niceties around"}, {"start": 3053.48, "end": 3056.84, "text": " like people figuring out how to get these working really well in transformers and kind of, you know,"}, {"start": 3056.84, "end": 3062.76, "text": " nicely map it over as well. I've all Yeah, there's also been some good work using these in speech as"}, {"start": 3062.76, "end": 3069.88, "text": " well. Liam, any other things to add on top of that? Some I used to do reinforcement learning,"}, {"start": 3070.6000000000004, "end": 3075.7200000000003, "text": " more full time and some colleagues kind of reached out about doing like sparse expert models for RL."}, {"start": 3075.7200000000003, "end": 3081.7200000000003, "text": " I haven't seen I'm not familiar with some work. But, you know, that might be sort of like another"}, {"start": 3081.72, "end": 3088.2, "text": " interesting avenue. But like, for sure. So language, vision, speech. I don't know if there's"}, {"start": 3088.2, "end": 3095.64, "text": " been any video work yet. Yeah, but like high data, a lot of throughput, those would be like,"}, {"start": 3095.64, "end": 3098.52, "text": " you know, really good areas. I think video would be also really promising."}, {"start": 3099.16, "end": 3103.7999999999997, "text": " Yeah, I really like also the I feel like it feels very natural in these high dimensionality"}, {"start": 3103.7999999999997, "end": 3107.08, "text": " spaces that you really might want different parameters to be applied like when you have a"}, {"start": 3107.08, "end": 3110.8399999999997, "text": " video, like one, I think you don't want to be applying all the same amount of compute to every"}, {"start": 3110.84, "end": 3114.44, "text": " frame. But then on top of that, I can see like, actually, you really want to have different"}, {"start": 3114.44, "end": 3117.96, "text": " parameters applying the different, you know, things going on in the video, because it's just"}, {"start": 3117.96, "end": 3122.52, "text": " gonna be like, wildly different stuff happening. So yeah, I think I think I'm very excited about"}, {"start": 3122.52, "end": 3130.76, "text": " these models for video as well. Do you imagine that these models will just essentially right now"}, {"start": 3130.76, "end": 3137.32, "text": " they're competition to dense models, they are they're competing, you're tracking Pareto front"}, {"start": 3137.32, "end": 3144.84, "text": " tiers, how much compute, how well are they doing, tackling very much the same tasks? Do you think"}, {"start": 3144.84, "end": 3150.44, "text": " this will go on? Like, do you do you think these models might overtake dense models if we figure"}, {"start": 3150.44, "end": 3157.32, "text": " out how to handle them correctly? Or is it is it more like, there's a killer app for each one of"}, {"start": 3157.32, "end": 3166.1200000000003, "text": " them? Yeah, I think in Oh, go ahead. I Yeah, I mean, I honestly think that the future is going"}, {"start": 3166.12, "end": 3171.16, "text": " to be adaptive. Like, I don't think there's any way that like, in 10 years, our models are treating"}, {"start": 3171.16, "end": 3175.08, "text": " all examples coming in with like the same parameters over and over again, and the same"}, {"start": 3175.08, "end": 3181.64, "text": " amount of compute. It may not be this precise sort of like sparsity regime, or it may not be"}, {"start": 3181.64, "end": 3187.3199999999997, "text": " the precise sort of adaptive computation, kind of like paradigms that have been put forth. But I"}, {"start": 3187.3199999999997, "end": 3193.16, "text": " view this sort of kind of work of like sparsity, adaptive computation, as kind of like inevitable,"}, {"start": 3193.16, "end": 3196.2799999999997, "text": " like I don't think it's going to be considered like competition, it's just going to be sort of"}, {"start": 3196.2799999999997, "end": 3202.3599999999997, "text": " like, integrated into a lot of like leading models. That's, that's my expectation. I'd be"}, {"start": 3202.3599999999997, "end": 3207.24, "text": " really shocked in like 10 years, we're training like a 100 trillion parameter dense model. And"}, {"start": 3207.24, "end": 3211.64, "text": " it's just kind of doing the same thing, like, over and over again, for no matter what comes in,"}, {"start": 3212.44, "end": 3220.68, "text": " just seems really strange to me. What's the future for your particular research? Like,"}, {"start": 3220.68, "end": 3226.68, "text": " where do you see where do you see yourself going in the next, maybe not the next paper that you"}, {"start": 3226.68, "end": 3233.08, "text": " haven't published yet, but maybe a bit broader timescale? Like what excites you? And what are"}, {"start": 3233.08, "end": 3239.08, "text": " your next plans here? Yeah, great question. I mean, I think the thing that really excites me is"}, {"start": 3239.08, "end": 3243.16, "text": " like what we were kind of talking about earlier of each input getting a different amount of compute"}, {"start": 3243.16, "end": 3246.44, "text": " applied. Like, I think right now, the models are working well for each input getting different"}, {"start": 3246.44, "end": 3251.16, "text": " parameters. And I think, you know, coupling this with like, adaptive amounts of computation is"}, {"start": 3251.16, "end": 3255.96, "text": " like, I think, really where I want to be spending time thinking about in the next, you know, upcoming"}, {"start": 3255.96, "end": 3264.76, "text": " years. Is there? Yeah, I don't know, is you have something like Ponder, there's PonderNet, and so"}, {"start": 3264.76, "end": 3270.12, "text": " on. There's these recursive architectures, or recurrent architectures that, that sort of decide"}, {"start": 3270.12, "end": 3275.7200000000003, "text": " themselves when to when to exit? Would that be one thing? Or do you simply imagine that each expert"}, {"start": 3275.72, "end": 3279.9599999999996, "text": " is kind of one is the buff expert, and one is the lean expert, and then the routing function"}, {"start": 3279.9599999999996, "end": 3285.3199999999997, "text": " essentially takes care of the different amount of compute? Yeah, I don't know. This is a great"}, {"start": 3285.3199999999997, "end": 3290.2, "text": " question. I think, I don't know, I can see either approach potentially working, or maybe you"}, {"start": 3290.2, "end": 3295.3199999999997, "text": " actually want combinations or potentially something completely new. Yeah, it feels like the space is"}, {"start": 3295.3199999999997, "end": 3299.7999999999997, "text": " still, you know, very exciting. And there's like a lot of really interesting different verticals"}, {"start": 3299.8, "end": 3306.76, "text": " being pushed. So the space still feels like, you know, pretty young to me. Okay, last question from"}, {"start": 3306.76, "end": 3312.36, "text": " my side, what's the connection of this to something like capsules? I don't know if you've ever thought"}, {"start": 3312.36, "end": 3317.6400000000003, "text": " about the the connection there. But with capsules, I always think this is these abstract, very"}, {"start": 3317.6400000000003, "end": 3322.84, "text": " abstract things, very high level ideas flying around. And you here have something like very"}, {"start": 3322.84, "end": 3328.84, "text": " practical, you know, very on the metal thing. Yeah, there seems to be quite some commonalities."}, {"start": 3328.84, "end": 3336.6800000000003, "text": " Is there is that something that ever came up to you? Or in the two years of doing sparsity"}, {"start": 3336.6800000000003, "end": 3343.8, "text": " research, this is literally the first time. I actually should be going back to that work. I feel"}, {"start": 3343.8, "end": 3349.4, "text": " like capsules like had a lot of like really interesting conceptions. But maybe like you're"}, {"start": 3349.4, "end": 3354.52, "text": " kind of alluding to it didn't like map super well to the metal. So maybe that sort of like hindered"}, {"start": 3354.52, "end": 3359.88, "text": " its like its use, whereas this is just like highly motivated from like an engineering perspective."}, {"start": 3360.68, "end": 3364.6, "text": " We've had like some questions like, Oh, what is like the neuroscientific kind of motivation of"}, {"start": 3364.6, "end": 3370.28, "text": " our work? And it's like, it's really engineering kind of driven. So it's like, okay, what what will"}, {"start": 3370.28, "end": 3377.64, "text": " be fast on our existing hardware? But yeah, I will revisit capsules and kind of see like, Oh, okay,"}, {"start": 3377.64, "end": 3381.8, "text": " how could we actually map this a little bit better to the hardware? And like, you know, I think that"}, {"start": 3381.8, "end": 3387.1600000000003, "text": " could be like, you know, an interesting source of ideas. Is there any any last thing you want to get"}, {"start": 3387.1600000000003, "end": 3393.8, "text": " out to viewers that they should take away from this work? Any any way that a regular person can"}, {"start": 3393.8, "end": 3399.4, "text": " get into this type of research? Anything like this? Yes, a great question. So actually, one thing we"}, {"start": 3399.4, "end": 3402.84, "text": " tried to show in our switch transformer work is that these models work pretty well, even if you"}, {"start": 3402.84, "end": 3406.92, "text": " only have two experts. So I definitely I don't want people to think that you know, you really"}, {"start": 3406.92, "end": 3412.2000000000003, "text": " need a supercomputer to run the models or to, you know, get benefits from having experts, even having"}, {"start": 3412.2000000000003, "end": 3416.2000000000003, "text": " I think, as little as two experts and running models could lead to developing really interesting"}, {"start": 3416.2000000000003, "end": 3420.84, "text": " research ideas, improving the performance and everything like that. So yeah, I definitely hope"}, {"start": 3420.84, "end": 3426.12, "text": " that you know, more people can continue to experiment and push for these models. Yeah,"}, {"start": 3426.12, "end": 3431.64, "text": " and then I would say like, another interesting trend that I've been following is sort of in"}, {"start": 3431.64, "end": 3436.8399999999997, "text": " parallel to sparsity in these like, you know, really large models is the idea of like, well, what if we"}, {"start": 3436.8399999999997, "end": 3442.2, "text": " just sort of like have the model sort of offload and like, sort of do lookups or, you know, look at"}, {"start": 3442.2, "end": 3448.2, "text": " documents and retrieval type methods. I think this is sort of like a very interesting area. And I'd"}, {"start": 3448.2, "end": 3453.16, "text": " love to see like, kind of head to head comparisons of like, okay, do we want to try to encapsulate"}, {"start": 3453.16, "end": 3458.7599999999998, "text": " the knowledge into parameters? Or do we want to just like, keep it sort of like, you know, parametric,"}, {"start": 3458.76, "end": 3464.44, "text": " non parametric type thing? And we keep the information kind of written in docs or like,"}, {"start": 3464.44, "end": 3469.0, "text": " what does the interplay look like? I think that's sort of like another really interesting avenue,"}, {"start": 3469.0, "end": 3475.32, "text": " like, kind of comparing these things. Awesome. Yeah, it sounds really cool. I'm excited to see"}, {"start": 3475.32, "end": 3481.32, "text": " what the future of these models bring. Yeah, Barrett and William, thank you so much for being"}, {"start": 3481.32, "end": 3487.0, "text": " here. This was a lot of fun. I hope to see you again soon. Yeah, cool. Thanks for having us."}, {"start": 3487.0, "end": 3489.0, "text": " Yeah, thanks for having us."}]