How to make your CPU as fast as a GPU - Advances in Sparsity w/ Nir Shavit
#ai #sparsity #gpu 

Sparsity is awesome, but only recently has it become possible to properly handle sparse models at good performance. Neural Magic does exactly this, using a plain CPU. No specialized hardware needed, just clever algorithms for pruning and forward-propagation of neural networks. Nir Shavit and I talk about how this is possible, what it means in terms of applications, and why sparsity should play a much larger role in the Deep Learning community.

Sponsor: AssemblyAI
Link: https://www.assemblyai.com/?utm_source=youtube&utm_medium=social&utm_campaign=yannic_autochapters

Check out Neural Magic: https://neuralmagic.com/
and DeepSparse: https://github.com/neuralmagic/deepsparse

OUTLINE:
0:00 Introduction
1:08 Sponsor: AssemblyAI
2:50 Start of Interview
4:15 How the NIR company was founded? 
5:10 What is Sparsity about? 
9:30 Link between the human brain and sparsity
12:10 Where should the extra resource that the human brain doesn't have go?
14:40 Analogy for Sparse Architecture
16:48 Possible future for Sparse Architecture as standard architure for Neural Networks
20:08 Pruning & Sparsification
22:57 What keeps us from building sparse models?
25:34 Why are GPUs so unsuited for sparse models?
28:47 CPU and GPU in connection with memory
30:14 What Neural Magic does?
32:54 How do you deal with overlaps in tensor columns?
33:41 The best type of sparsity to execute tons of CPU
37:24 What kind of architecture would make the best use out of a combined system of CPUs and GPUs?
41:04 Graph Neural Networks in connection to sparsity
43:04 Intrinsic connection between the Sparsification of Neural Networks, Non Layer-Wise Computation, Blockchain Technology, Smart Contracts and Distributed Computing
45:23 Neural Magic's target audience
48:16 Is there a type of model where it works particularly well and the type where it doesn't?

Links:
Homepage: https://ykilcher.com
Merch: https://ykilcher.com/merch
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
Discord: https://ykilcher.com/discord
LinkedIn: https://www.linkedin.com/in/ykilcher

If you want to support me, the best thing to do is to share out the content :)

If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this):
SubscribeStar: https://www.subscribestar.com/yannickilcher
Patreon: https://www.patreon.com/yannickilcher
Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq
Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2
Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m
Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n
 Today I'm talking to Nir Shavit about sparsity. Nir has been long time active in the field as a professor at Technion and MIT and has also been awarded with various prizes such as the Gödel Prize in 2004 and the Dijkstra Prize in 2012. He's also founder of a company called Neuralmagic that questions one of the fundamental core principles of current machine learning, namely you need GPUs. Neuralmagic uses various techniques such as sparsity which we're going to talk about today, but also other optimization techniques to make inference on models like BERT to be as fast as a GPU on a regular CPU. This is pretty huge and can have vast implications on where you can deploy these models and just how expensive it gets to roll them out to many people in many places. So today we'll talk about the biological foundations for sparsity, why we shouldn't attempt to replicate the brain and just what it takes to make something go really fast on just the CPU. I hope you enjoyed this conversation. If you do give Nir and his company a follow and I'll see you around. Bye bye. Hi, this video is sponsored by assembly AI. Assembly AI does real time and batch audio transcription of audio and video files powered by the latest advances in artificial intelligence. So if you are a developer or work for a company that's looking to get more out of your audio or video data through transcription and audio intelligence, assembly AI is the best place to go. Not only do they have a user interface where you can just upload stuff, but they do have a very powerful API. But transcription isn't all they do. Once your audio is described, they actually post process it in many different optional ways. So they can do things like speaker classification or annotations of various forms inside of your audio. One feature I'd like to particularly highlight today are the auto chapters for this simply provide auto chapters equals true on your upload and assembly AI will after it's transcribed your audio automatically recognize chunks of audio where you talk about the same thing give you a summary of those chunks and a neat single description headline of what you were talking about there. This is absolutely ideal for anyone who does any sort of long form podcasting or videos like mine where viewers are very, very helped by the fact that there are chapter annotations and to have these be done automatically is just absolutely great. So if you're interested, head on over to assembly AI use the link in the description to let them know that I sent you there the single API to transcribe and understand audio they do so in batch and in real time via web socket, they accept all kinds of audio and video formats and they do so in over 15 languages, give it a try. And thank you very much to assembly AI for sponsoring this video. And now let's get into the video. The topic of sparsity is a big thing in neural networks right now, mostly because we have no idea really how to do it. And I think that's exciting times for the future. So welcome what what brings you into the sparse world? Actually, I, you know, I've been a professor of computer science for many years, and I worked on multi course for more than 30 years, and got involved in computational neurobiology in the last 10 years. And one of the things that you really see in the brain is really how sparse its computation is, it really is very, very sparse. And so, you know, looking at neural networks, we see that there are there's a similar phenomenon to what happens in brains happening in neural networks, right, where you can actually reduce the number of parameters through pruning by huge amounts and preserve accuracy of the performance of the network. And that kind of says, okay, if we really want to have brain like performance, you know, sparsity is probably one of the tools that we want to use to get there. So that's kind of how I kind of got into this. And you founded a company that also works into this direction, right? You want to talk about that? Yeah, a little bit. Yes, I founded Neural Magic. Neural Magic was founded because what we were seeing in my lab, I was busy with doing machine learning at a large scale for biology projects. And what we realized was that we could get CPUs to run at GPU speeds like at the time, it was a Pascal GPU. And you can make just a regular CPU do what the Pascal GPU was doing through the use of sparsity and other similar techniques. And so we said, okay, well, there's a real commercial value here for people because you don't need an accelerator, you can just do it on your commodity CPU. And that's that's Neural Magic. So what we do is we deliver, you know, through sparsity and similar optimization techniques, GPU performance on CPUs. That is, it's quite a promise. Maybe let's first dive into a little bit about sparsity itself. What is it about sparsity? You mentioned the brain is very sparse. Yet our current or at least the way we train neural network is very dense, we can accelerate the dense neural networks much better. What is it about sparsity? Is it just the saving of parameters? Or is there something more to sparse connections than to dense connections? What do we know? That's a good question. So clearly, what we're doing today is not the sparsity that we will be doing in the future. And what I mean by that is your brain is sparse way beyond the levels of what we see in neural networks today. So your typical brain in terms of the compute, right, you know, your cortex is like a cell phone of compute, right? But the graph is enormous. It's like, you know, the graph is the size and you need petabytes to basically hold it. So a cell phone of compute on a petabyte or more of memory, right? But the accelerators that we build, you know, are designed to deliver petaflops of compute. But on a cell phone size memory, the memory is very limited because we use this high bandwidth memory. So in a sense, we're building the opposite of what we want, right? So if we want to mimic the brain, we should not busy ourselves so much with the amount of compute and rather worry about how it is that we implement this very large graph. It's a very large graph, but it's extremely sparse. That's the point, right? And as you asked, the sparsity is not necessarily the same sparsity that we do today through pruning techniques, but it's a combination of a very sparse architecture together with, you know, a sparsity in what we call in machine learning the kernel, right? So it's not just that the kernels are sparse, but everything in the design is very, very sparse. Okay. And we don't know yet how to design very sparse architectures. Part of that has to do with the fact that machine learning grew up in the GPU world where sparsity is not an advantage actually, because you're doing lockstep computation. So you win nothing by being very sparse. And therefore, you know, we don't see those architectural sparsity things yet. But I'm expecting that to happen. We should be, this should come along, you know. And even more than that, what I expect is things are starting to show up like the pathways from models from Google and so on where even if you have a very large model, you don't execute the full model layer after layer, but rather you execute small regions of the model at any given time. Per input. That's another form of sparsification of your computation. Right. And that is what the brain really does. So your brain typically, you know, when you see an input or so on, uses a very small fraction of its total graph to do the computation. And so that's where we're headed. We're not there yet. We don't know how to do it. But this is the goal. And that's the old, you only use 10 percent of the brain at any given time. Right. Yeah, that's right. I mean, really, from energy considerations, it really is like a cell phone. OK, it really isn't, you know, this massive monster multi GPU thing that we use today. And so my expectation is that, you know, that as we learn more and more about how to design sparse networks, we're going to see them become the standard. They're not the standard right now because we started the whole journey, right, by applying flops and still applying flops is the main paradigm. But we will see it appear both in hardware and accelerators and in CPUs. This idea that we can utilize sparsity, you know, to get really great performance gains. Yeah, that's coming. Now is the question is a little bit the chicken and the egg problem. Is the brain sparse because it has the limitations of the cell phone power or does the brain only need cell phone power because sparsity is such a good architecture? Right. Like which causes which? Yeah. So I would say that, you know, the whole notion of parallelism in the brain, right. If you think about it, imagine that you need to do a billion operations per second. OK, and what you have are these very slow chemical devices, neurons, right, that can do that. Right. So you need a billion operations, a billion firings of neurons in a second. How are you going to do that? Well, what you need is massive parallelism. Right. You've got to get massive parallelism. If you can do the massive parallelism, you can get the billion operations. Right. And so our brains are parallel, if you will, because we have this special medium. Right. Now on a modern multiprocessor, right, you can get a billion or 10 billion instructions executed per second sequentially. You don't really need parallelism for it. Right. And so what I'm trying to say is, you know, the whole idea of kind of how brains evolve is clearly because of the way, you know, they're implemented. But we should not think of going and implementing this in some way. Right. And we should not think of going and implementing this in silicon in the same way. Right. Because we really what we really should think about just is that both of these things are train complete. Right. You can do you can implement the algorithm. You just need to know what the algorithm is. And then on silicon will implement the best algorithm we can. Right. You know, of course, we need a lot of time to do that. OK. Does that make sense? That's what I'm trying to say. You know, let's implement the algorithm, but not necessarily the architecture. OK. So when I say sparsity, I really mean sparsity algorithmic sparsity. Right. And it doesn't mean that you have to have a very sparse kind of, you know, silicon VLSI circuit to do this. That's not the case. Right. Given that we that's a good segue given that we do have the flops right that we don't have in the brain. It naturally it is a different a different system. We do have teraflops petaflops even in these giant compute clusters. Where should we put them in your opinion? Like where where should that extra resource that the brain doesn't have go? Should it go into sequentially executing what the brain executes in parallel or, you know, where should we put that? So first I want to say is that we have those flops, but they're costing us a lot. And you just have to open the papers to see what the cost of the flops is. It's enormous, an enormous energy drain. And it's also an enormous architectural drain on what we're doing. And so I would say we want to get rid of the flops because probably we don't need them. OK. And especially as you go from the data center down to the edge, you get the capability of delivering flops comes directly at the you know, if at the edge you can put the sorry in the data center, you can put, you know, your Google data warehouse right next to a waterfall or whatever you want, right, to a source of energy. Right. When you're doing this on your cell phone or on a tiny device at the edge, every little bit of energy that you waste is critical for you. Right. And so what we really want to do is move away from the flops and move more towards the very energy efficient way the brains work because this adding more flops is a momentary thing for us. Right. So, yes, we can do this, but at a very high cost. And no, we don't want to do this forever. We want to find ways to cut the cost, reduce the compute. And there's a little other thing that I want to say, and that is architecturally we generate the flops by running right now, at least by running many, many, many tiny cores, thousands of tiny cores typically. Right. And in architectures that require a lot of connections to the memory, this high bandwidth memory. And this thing doesn't scale. So in a sense, we're trading flops for memory. If you use the CPU today, you could get a terabyte on your desktop, but go get a terabyte on a GPU. Right. And so using the flops is going to enable us changing the architecture. If we don't need so many flops, then we can actually increase the size of our memory, which will make us able to hold these giant models that we want to do very cheaply, if you will. If I explain a deep neural network to someone, I usually, you know, you start with a fully connected layer, you say, you know, here is a layer of neurons, and here is a layer of neurons, and they have their connections, right, and each connection has a little weight, and so on, you usually describe like a dense, fully connected architecture. And that is conceptually, I want to say, easy to grasp for people and so on. Do you have an analogy for sparse architectures? Like, what is the conceptual, like, could you conceptualize to someone who doesn't know what like a sparse architecture is and how to think about it? What is different? Yeah, the way we do sparsity today, I don't know what it'll look like in the future. But today, sparsity looks like, imagine that the two layers of the neural network are these kind of, there are cords from one layer to the next, right, there are strings attached, and these are, of course, these are the connections, the weights that we're using in the computation, right. And sparsity means I take scissors, and I chop, chop, chop, chop, chop, you know, till I have five or 10% of those cords left, right. And those cords, it turns out, right, if I do this right, if I do this kind of pruning, right, are good enough to capture, right, the accuracy of the model as it was before, because a lot of the connections are not important for this process. That's kind of the big discovery. And modern research in techniques for sparsification, right, you know, play along this kind of game. So you can do this kind of unstructured thing that I just described, where you arbitrarily cut in many places based on the effectiveness, or you can also structurally take things out. So in a lot of the modern models, right, we're removing pieces that are not necessary. We do architecture search to find these places to cut things, right. So that's where the whole game right now of efficiency and neural networks, right, is the game of how do I cut this thing down, right. In the brain, there are certainly some systems like the visual system, where that is clearly organized into layers, but there are many other systems that have no resemblance to layers, there are connections going up and down and left and right. And, you know, between the halves of the brain and all, is there a possible future where this could become into like a standard architectures for neural networks, that the notion of layers and things like this isn't even really a, you know, a thing anymore? Or is there, you know, some some fundamental way where we say, no, there's probably always going to be layers, but it's just going to be a sparsity between those layers. So when we look at, you know, we have a full connectome of essentially only a couple of animals, a worm and a fruit fly, that's it. And that's it, you don't see a lot of layering there, it looks more like a mess, very sparse mess, okay. And I wouldn't venture to think about what a cortex looks like, right. We don't have that yet, we're working very hard, to these are very hard computational problems to be able to go and get a model. We just want to do a mouse, even a mouse is just too big for us to do right now, like a small mammal, right. But my, I would venture to guess that yes, the answer is that, you know, it's extremely, it's an extremely sparse architecture, and that it wouldn't, it will not look like layers, okay. You can impose a layer structure on any graph, okay. It's not, so the idea that I say there are layers, sure, okay, I can take the graph and I can layer it, yeah, I could do a BFS on it and layer it. But the point is not so much that it's more that by design, when I think about it, right, I'm not going to think about it as a sequence of layers where the change that I make is the change in the layer, one layer is different from the other. But rather, it'll be a combination of thinking about paths, different paths, and I'll do different things along different paths. That's kind of the idea, you know. If you think about, you know, there's recent research from MIT, you know, you can detect, people can detect an image in 0.13, set 0.013 seconds, in 13 milliseconds, okay. In 13 milliseconds, you can detect, you can say what an image is, okay. This is, there's no time for neurons to fire. This thing is extremely kind of parallel, right, and uses very little compute and gets you an answer. And a large part of that is prediction, because you're already expecting something. So we need to learn how to do those things. And so machine learning right now is in a very naive early stage. And so given that, and given the things that we are doing right now, it's not a surprise that we're doing the brute force kind of massive compute kind of thing. That's always what you do. And with time, we're going to get better and better at it, right. So that's kind of how I see this progressing. Speaking of becoming better, if you know, the flatworm is sparse, the mouse is sparse, the human is certainly sparse, yet our best models today are all big, dense, you know, computation hungry things, there is not really a case, every time I prune, I sparse if I and so on, I get savings in like, you know, savings in CPU or GPU, I get savings in, you know, my storage, but I also get like a little bit worse, right? That's the common thing today in pruning is that I get like just a tiny bit worse than the dense model I prune from. Why do you do you think that is just the fact that we prune from a dense model? Or what's holding back the sparse models? How about if I turn this around? Let me turn this around for you. Okay, you can take BERT base, which is a common model that people use, okay. And you can sparsify BERT base. Neural magic, we sparsify at 95%. So a 95% sparse BERT base, one over 20th of the compute, okay, way beyond anything a GPU does, even if you run it with full throttle, okay, it's just cutting the compute so much that there's really almost nothing to compute there. It's just moving data, okay. Now I'm exaggerating, of course, but, you know, it's really becomes a data movement problem rather than a compute problem when you, and you lose 1%, less than 1% accuracy, okay. And I say, okay, great. So you've done that, you know, and you've gotten all this speed up, but you've lost, you say, oh, near, but you lost less than 1% accuracy. But what I say instead is forget that. Take BERT large, a much more accurate model, several points more accurate than BERT base, okay, and prune it so that it actually, right, with 20x less compute, it's actually faster than BERT base, okay. And so now you have the accuracy, right, and you have great compute, and this is through sparsity. So by sparsifying the larger model, I actually delivered you the best of both worlds, little compute and great accuracy. And that's how I want you to think about sparsity, right. It's a way of enabling us to run much larger, more accurate dense models, but because we sparsified them, we are, you know, we're getting great performance. That's how to think about it. What's the limit currently that keeps us from, we always need the dense model first in this model in the pruning, in a pruning setup, we first need the dense model, then we go to the sparse model, we get huge savings at inference time. What keeps us from just building the sparse model in the first place? Great. So this is kind of the lottery ticket kind of question, if you will. There is research, actually, Dan Alistair, one of our consultants, Neural Magic works exactly on this kind of stuff. We know how to run a training session right now for four models where you start out and you need to do only a certain fraction of the, you know, of the forward passes, backward passes, dense, and then immediately you can already start pruning while training. So there is research going in that direction. But you are right that right now, at least, right in the standard, if you look at what's going on there out there, standardly, you're right, we do most of the time take a standard model and from dense we sparse it and so on. But the thing to remember, and this now I'm not talking about the research, because the research is going to get there. Janek, I don't know if to what extent we will, how fast this will happen and so on, but we will learn how to build sparse architectures that start sparse and continues, you know, it's really a matter, nature does this. And so there's no reason why we wouldn't be able to do it. But I want to say something about today's machine learning where you kind of start with the dense and then you have to sparsify. This is really not the common paradigm for most users of neural networks. For most users, a model is given to them that, you know, from a known architecture, right, and then they transfer learn onto it. And most people do that rather than train from scratch. They really use the model that somebody already worked very hard to build for their specific use case, and then they transfer learn onto it. So this is what you can do with sparsity. You can take a sparse model and sparse transfer learn onto it. It's extremely efficient because you're running at the speed of the sparse network, right? So you can sparse transfer and then you don't need all of this kind of start with dense. And we're seeing more and more sparse networks appear, you know, in the literature and in the database collections of machine learning models. And as we have more and more of these initial good sparse models, right, people are going to learn to start with the sparse already. That's kind of commercially, I think that's what we're going to see more and more of. Why, you mentioned this a bit already, but why are GPUs so unsuited for sparse models? And what makes CPUs in the way you do it, really suited for sparse models? Or are they even suited? Or are you simply, you know, seeing that they're better? Yeah, I mean, look, the the GPU architecture, you know, is designed for this very, you know, small cores, tiny caches. You're not going to go and throw all that away just because, you know, you found, you discovered sparsity. So you're trying to do sparsity while keeping this kind of lockstep execution structure, right? And this is difficult to do sparse. You need, you need, you need, you need really a different kind of setup to get an advantage out of sparsity. Now, now I'm not, it's not like you can't do that, right? It's not like you can't do that. People can design and have designed hardware that utilizes sparsity efficiently, okay? There is such hardware. It's just not, it's not GPU-like. It's not like the accelerators that we have today. But all of these, again, all of these accelerators have a different problem that has just to do with the memory. Because of the way they're designed, right, they typically have very small memories. So we're talking even, even ones that can run sparse, right, still have the limitation of their memory size. So the reason that CPUs are attractive is not so much that, you know, that they, that you have a natural way of running sparsity because you can run asynchronous with large cores, but rather that the large cores, you know, they're not going to run all the time. So the large cores enable you very easy access to very large memory pools, right? So the advantage of having strong, powerful cores, right, is really that I can put several terabytes of memory next to them, right, and run easily. And that's where the big advantage is going to be. As we understand more and more about how to build giant models that don't run all the time, right, then the compute will be less important, but actually the ability to hold that model in one place and run it rather than break it apart on eight or 16 GPUs, that's going to be your advantage. And so this is, so I'm kind of saying it's not so much that you can't build a hard piece of hardware to run sparsity, you can, right, but you should build it looking like a CPU in the sense of you can access a lot of memory because you're not doing tiny cores. That's kind of, that's my two cents. So the the CPUs are good because they have, you know, fast connect to large memory, but also over the years, we've put more and more levels of cache onto the CPU. How much do you have to have to take this into account when you're building? I mean, you're maybe you can explain a little bit what your company does in terms of software, you build compilers, or can I just run TensorFlow or something? Yeah, so let me explain. So first of all, the connection between the CPU and the memory is slow. GPU has a faster memory and faster access to it, right, smaller, but fast, right? CPU memory is slow, but large, very large. But CPUs have a cache hierarchy, as you said. And so if you know how to utilize your cache hierarchy, then, you know, if you're running in the L1 cache of the CPU, okay, you're running as fast as the GPU. There's nothing there that the GPU does that the CPU can't do once you're in cache. Okay, in fact, CPU caches are much faster than GPU caches, and the performance is better. So the question then, right, and this is what NeuralMagic does is, okay, so what we do is we sparsify the model. Now, you know, if the pro, you know, machine learning is about, okay, I need to meet a certain latency. And because I couldn't meet that latency with the CPU, then we added the GPU and boom, there's machine learning with GPUs. Now I can meet the latency. But there's two ways to deal with latency. One is to add more flops, and the other is to reduce the flops, right? And so sparsity, instead of adding more flops in hardware, reduces the number of flops needed in software. But now that you have this very sparse model, because the CPU memory is slow, okay, then what happens is you hit a bottleneck, and it's very hard to move. If you do this layer after layer, it's very hard to move the data in and out, okay? So what NeuralMagic invented is a way of running neural networks depth-wide. So we have this technology, which we call tensor columns, where essentially you can run, okay, you know, you can break the model lengthwise and run, you know, each one of these kind of columns, you know, in cache, okay? And because you're not leaving L2 really, or rarely leaving L2, you know, you actually get great performance. So in a sense, right, what we're doing is we're using the natural ability of CPUs to prefetch things from memory and then run in cache. And because this, you know, this cache hierarchy on CPUs has evolved over 70 years, or maybe I'm exaggerating, 60 years of hardware design, it's a very, very well understood thing where people know how to optimize it, right? Especially the big, you know, chip makers, they really know how to make these caches work really well. And so with these really good cache hierarchies, you really get great performance by running the model depth-wise. So that's NeuralMagic, you know, we take the model, sparsify it, now it doesn't need the compute, and now we run it on the CPU and get speed because we're running in cache, okay? And if you look at the numbers, I mean, you know, we are, you know, at the speed of, I mean, some numbers we have in PubG, we're at the speed of an A100, even faster, in terms of how long it takes. A four-core CPU can, in terms of latency, do what an A100 does on a common model like BERT, okay? So it's really the emphasis... Given that it's sparse or... Yes, yes, yes, by sparsifying it and running it, you can make a four-core do what A100 does. So it's really now a matter of throughput, and the A100 has a lot of throughput, okay? So now the question is, you know, how many cores do you want on your CPU to meet the throughput of the A100? And again, the story is that, you know, the big providers are adding more and more and more cores, so you're going to be able to compete better with the GPUs down the road. So that's kind of the story of NeuralMagic. So the way I can imagine these tensor columns is that because I execute depth-wise, the sort of values that I need for the next step in the computation are the results of the very last step, therefore are already going to be in cache. And since everything's sparse, I don't need all of the last layer for the current step, and therefore, you know, I have it already. Okay. Right, and of course, you know, when you think about neural networks, there are overlaps between these columns. And the question is, how do you deal with the overlaps in a way that doesn't kill your computation? And that's the magic. That's the magic of it. There's an algorithm that allows you to do that. And because you can do it, you manage to run this way, and you don't hit this memory bottleneck, and boom, you're in business. Yeah. So for GPU, it's almost like, you know, GPUs enable us to do dense models. But I think also models have almost co-evolved with the GPU. So people have started building models to fit the GPU architectures better, right? Especially something like a transformer is like, that's it, that's like made for GPUs. Is there a type of sparse model? Like if you if you could wish for the best possible sparse, but you know, there's different kinds of sparsity, like, what is the best type of sparsity to let's say, execute on a CPU, if we want to look forward, and we want to especially build architectures? Yeah, this goes back to your original, one of the first questions you asked, right? It's about a different structure for the neural network execution. So we should forget the synchronous layer after layer execution. And think about the fact that, you know, we can run through a model, right, in multiple paths with multiple computing units, use the same weight structure, and so on, of the model, right, but run at different speeds. And by running at different speeds, and going through the model in different paths, I can get from the same model, multiple answers to my question, which is kind of what I believe what your brain does. So what happens there is, you have this network, but it's not like, you know, it's all firing like this layer after layer, it's rather, you have these asynchronous flows going through it, right, even going through matching paths, and CPUs are naturally built for this thing. Now, I'm not saying that somebody can't build a beautiful FPGA that will perhaps have a better, closer structure to what a brain does. Maybe so. But, you know, but there is an advantage to being commodity, okay? The fact that the CPU can do other things is a big win. If I can make, if I can move everything to software is really, is the thing, then I can really get all the advantages of modern software. So I'm not poo-pooing hardware accelerators, I'm saying, great, you know, they have a role, and so on and so forth, but they come at a price, right? And the price for any organization is that you, instead of just downloading or shipping your product with the machine learning piece, you have to ask the client to buy a certain accelerator, or run it with a certain accelerator. And this all goes away if we can figure out how to make the CPUs do what the GPUs do, right? Then we have, then we're back into this beautiful world of containerized, movable software, and that's really kind of where I would love machine learning to move to, rather, right? That we would have, and maybe down the road, right? There is this, you know, you know, CPUs have a history of absorbing the key components of any new paradigm that shows up. You know, virtualization started out with tricks on a GPU, on a CPU, and then later on added the features. Networking had special accelerators, and then they moved into the CPU. And I'm expecting that whatever features are necessary for machine learning to run well, will move into the CPU, and we won't need an outside accelerator to make this thing work. If you could, so I think that's, by the way, also the story of GPUs themselves, right? They were already kind of consumerish available, and then they can, they absorbed machine learning. It's not necessarily the best architecture for machine learning. Let's say there's already all this hardware out there, right? There is very good CPUs next to very good GPUs. How do we get the best out of a machine like this? Right now we've advocated for, let's move things to the CPU, right? We have some advantages there. But what if I have a box with both? Like currently, I just use my CPU to ship data to the GPU, right? That's what my CPU does. But is there a way where I could potentially, you know, what kind of architecture would make the best use out of a combined system of CPUs and GPUs? No, I think this is really the vision that Nvidia has at least today for their Grace Hopper architecture. It's essentially, there will be a CPU and a GPU connected to one another, and the CPU will do all the things that are memory intense, and the GPU will do all the data intense things. The thing about the problem with this kind of a model is it's a beautiful model, by the way. I'm not saying anything bad about this. If you really want to build a GPU world, that's a great thing to do. But again, how much you utilize your GPU, your attached GPU, has to do with how you write your application. Because you need to move the data into the GPU in and out, and that's slow. Remember, it's exactly like going to memory, right? The GPU is not sitting in your caches. So if you're on the CPU and you're computing something on a cache and suddenly you get a page fault and you have to go and get something from memory, that's the latency that the GPU introduces here, right? And so if you're going to design it with that, you have to create really good software to pipeline things. And this is at the level of the application. So the application programmer has a big programming task. And so this is a great solution for large scale, big projects where, okay, Facebook is going to get a thousand of these or 10,000 of these, whatever it is. Or Google, 10,000, 100,000 of these and put them together. Then it's worthwhile to write this kind of complex software. But if you're Joe company, right, and you have your little thing, I don't think you want to be writing that interface, right? So I'm saying it's great for large things, right? Data center things, big things. But I'm very doubtful if this is going to be effective at the edge, if you can actually utilize the CPU for it. Okay. And I will say one more thing and that is that, you know, that the modern way that the designers of hardware think about it is that it's built in modules. If you look at the AMD latest architecture, right, essentially you have the CCXs. So the machine, even though it has, you know, maybe 40 or 50 or 60 cores, right, they're grouped into groups of eight, right? And each group of eight like this is a little piece of the die. Okay. And I think Intel is shifting in that direction too. So nothing's to prevent you from making pieces of that die be specialized pieces of hardware, like a GPU. You don't have to have outside device. So if you ask me what the future is going to look like, it's probably going to look like, you know, these large cores, right, that have or large machines with multiple dies and on these dies, we might have a GPU die, we might have accelerated. And that's more like what I expect to happen rather than having a massive, you know, accelerator on the side. If we, if we hear sparsity, and things not being in layers and so on, naturally, the topic of I think graph neural networks is very close to that, at least in the imagination of people, do you have anything to say about, you know, where current graph neural networks stand with respect to sparsity? Yeah, I would think of graph neural networks as a, as a, you know, graph neural networks as a, as a, as a different kind of, okay, so, so graph neural networks, I use some some graph neural networks in my research and the, and the idea there, you know, is that, you know, we can use graph neural networks to solve graph problems that otherwise would be very complicated to solve if we tried to solve them brute force. Okay, now, it's not generally true. There are quite a few limitations. But, but as a tool, I would say that, you know, rather than think about the neural network itself as being looking like a graph neural network, right, I could use graph neural networks, right, to define what we call motifs in the neural network. So for example, when we try to look at, at how brain structure, brains are structured, right, graphs of brains, and we try to understand, you know, is there a motif that is repeating itself in this graph, right, then using a graph neural network for that is a really nice way to try to find these motifs, okay, efficiently, right, because the problem itself is, is PSPACE complete, or we don't know, it's graph isomorphism. So, so clearly, we don't know, right, how to do the brute force algorithm well, but, but the graph neural network can come to our aid here. And so, so I would say that right now, I don't really see a real network design, neural network design, that is specific to that or a way that it helps, but, but in research, it definitely helps. And we really want to use these networks to help us in research. This might be a bit of a tech bro question. But if I hear, you know, I can do sparse computation, very, I can reduce the flops and so on. Is there any intrinsic connection between the sparsification of neural networks, the non layer wise computation, and blockchain technology and smart contracts and distributed computing and things like this? Have you ever given this any thought or, or, yeah, is that completely off? Yeah, look, I think nothing is completely off with respect to machine that in the sense that I am sure that machine learning will find its way into into all of those areas. Right, it's a matter of time. And, and right now, right, the all the work there doesn't need the efficiency of right, of what machine learning offers, because machine learning in the end is an optimization technique. And so when I think when all these blockchain algorithms and all, you know, become more commonplace, and we need to provide them with things like security, further security or analysis, and so on, and so on, I think then we're going to see applications of machine learning there. And with that, I think, all these things of sparsity and so on are going to appear. But, you know, but, but for me, right, it really is the whole story of sparsity, right, is the story of a phenomenon that is very prevalent in nature, right, that may you can say surprisingly or not surprisingly shows up in machine learning. And it kind of it makes me feel like it's strengthening my belief, right, that even though the exact computations that we're doing are not the same as spiking neural networks in brains, right, that there is a lot of commonality there. And the emergence of these similar phenomena, like sparsity, like, you know, pruning and so on, and the fact that we can get benefits from it, this tells me, oh, okay, these are related. I think that's a very important point to keep in mind. With neural magic, who is your main target audience? Like who who is listening to this? Do you want to let know like, we are exactly for you. So we span the gamut from the data center to the edge. I would like to say, I mean, we just now are moving into providing the same properties for ARM architectures. And so I would say the exciting new thing in neural magic is we're moving from doing this, you know, for AMD and Intel architectures to doing it for ARM, which means that we're going to span the gamut all the way to the very bottom of the of the food chain, if you will. And I think this is very exciting. Because as you know, because because sparsity has a dual role as you go down the food chain, right, because for the large accelerating, you know, the fact that the memory footprint is large as small is not that important. But as I go down, sparsity gives me two things speed with neural magic gives you speed, but it also makes the model extremely small. So you're getting a small, accurate model, right running on a very small device. And this, you know, typically is an ARM device. And so that's, that's, that's the audience that I'd like to say, hey, we're coming, you know, we're coming in, we're going to deliver the same things that we can deliver for Intel and AMD. We're now going to deliver it for ARM at the very end. If you say edge, do you mean smartphones? Do you mean security cameras? Do you mean robots? Everything? Okay, I mean everything I not like I'm going to do everything to start with. But but yes, yes, we're aiming in that direction. Yes. And with the danger that this has become going to become like a marketing opportunity question, but how easy is it to get started with what you're doing? Like, let's say I'm a I'm like, I've done, you know, my TensorFlow tutorials, I know how to build a model and train it and so on. Like, how much does it take for me to do that? And then you know, how much does it take for me to transition? Or to apply what you're doing? Yeah, so you just go to our website, go to get go to get download, deep sparse, or, you know, our engine download our ML tooling. And, you know, immediately, you just either pick a sparse model and transfer learn on to it with our tool. So we have recipes, you have a model, you have a recipe, exactly what you would do if you went to Hugging Face and downloaded a model and downloaded a recipe, you do the same kind of thing. And you sparse transfer learn onto it, and you're in business. So it's not very hard. So I think this is really and we're working on making it even even easier. This is one of our goals, right is to make it really, really easy to do this. And the advantage, of course, is that, you know, people are already busy, you know, quantizing their models to get more performance. So this is like quantize, in some sense, right, you're going to do the same kind of thing and get a lot more performance. Is there a type of model where it works particularly well and the type of model where it doesn't like I'm thinking, you know, convnets, recursive networks, autoregressive, maybe, you know, the big language models, like what is it best at? Yeah, so right now, you know, it's best at at BERT, YOLO models, we do, we do computer vision, and we do, and we do the language models, but not the large language models, we haven't done the large language models yet. So for those types of things, like the BERTs and the YOLOs and the, you know, the, whatever the variants of efficient nets and all these guys, this is, you know, visual transformers, these are the things that that we do right now. And every all our technology is right now, you know, available for those. I'd love to do the large models, a CPU is a natural environment for running the knowledge models. You know, these giant models, these trillion or whatever parameter models that people talk about splitting across 16 GPUs, they fit on your desktop. Okay, so clearly a CPU is a natural place to run a very large model. Okay. And so that's that will be a target but not right now. Okay, very exciting. Is there any last things you want to get out maybe about neural magic or sparsity in general? Well, you know, our whole machine learning software stack is open source. And we'd love people to comment and help us build, you know, better sparsity, use sparsity in their models and, and tell us about what they're doing. And, you know, that it would we have a community and we'd love you to join us. Excellent. Nir, thank you so much for being here today. So it was very pleasant. Thank you very much. Bye bye. Bye bye.
[{"start": 0.0, "end": 5.6000000000000005, "text": " Today I'm talking to Nir Shavit about sparsity. Nir has been long time active in the field as"}, {"start": 5.6000000000000005, "end": 11.200000000000001, "text": " a professor at Technion and MIT and has also been awarded with various prizes such as the"}, {"start": 11.200000000000001, "end": 17.28, "text": " G\u00f6del Prize in 2004 and the Dijkstra Prize in 2012. He's also founder of a company called"}, {"start": 17.28, "end": 23.68, "text": " Neuralmagic that questions one of the fundamental core principles of current machine learning,"}, {"start": 23.68, "end": 29.92, "text": " namely you need GPUs. Neuralmagic uses various techniques such as sparsity which we're going to"}, {"start": 29.92, "end": 35.84, "text": " talk about today, but also other optimization techniques to make inference on models like BERT"}, {"start": 35.84, "end": 44.24, "text": " to be as fast as a GPU on a regular CPU. This is pretty huge and can have vast implications on"}, {"start": 44.24, "end": 49.68000000000001, "text": " where you can deploy these models and just how expensive it gets to roll them out to many people"}, {"start": 49.68000000000001, "end": 55.760000000000005, "text": " in many places. So today we'll talk about the biological foundations for sparsity, why we"}, {"start": 55.76, "end": 61.44, "text": " shouldn't attempt to replicate the brain and just what it takes to make something go really fast on"}, {"start": 61.44, "end": 67.28, "text": " just the CPU. I hope you enjoyed this conversation. If you do give Nir and his company a follow and"}, {"start": 67.28, "end": 73.44, "text": " I'll see you around. Bye bye. Hi, this video is sponsored by assembly AI. Assembly AI does real"}, {"start": 73.44, "end": 80.0, "text": " time and batch audio transcription of audio and video files powered by the latest advances in"}, {"start": 80.0, "end": 85.28, "text": " artificial intelligence. So if you are a developer or work for a company that's looking to get more"}, {"start": 85.28, "end": 89.52, "text": " out of your audio or video data through transcription and audio intelligence,"}, {"start": 89.52, "end": 94.8, "text": " assembly AI is the best place to go. Not only do they have a user interface where you can just"}, {"start": 94.8, "end": 100.56, "text": " upload stuff, but they do have a very powerful API. But transcription isn't all they do. Once"}, {"start": 100.56, "end": 105.84, "text": " your audio is described, they actually post process it in many different optional ways. So"}, {"start": 105.84, "end": 111.44, "text": " they can do things like speaker classification or annotations of various forms inside of your audio."}, {"start": 111.44, "end": 116.8, "text": " One feature I'd like to particularly highlight today are the auto chapters for this simply"}, {"start": 116.8, "end": 122.64, "text": " provide auto chapters equals true on your upload and assembly AI will after it's transcribed your"}, {"start": 122.64, "end": 128.0, "text": " audio automatically recognize chunks of audio where you talk about the same thing give you a"}, {"start": 128.0, "end": 132.72, "text": " summary of those chunks and a neat single description headline of what you were talking"}, {"start": 132.72, "end": 138.4, "text": " about there. This is absolutely ideal for anyone who does any sort of long form podcasting or"}, {"start": 138.4, "end": 143.6, "text": " videos like mine where viewers are very, very helped by the fact that there are chapter"}, {"start": 143.6, "end": 149.12, "text": " annotations and to have these be done automatically is just absolutely great. So if you're interested,"}, {"start": 149.12, "end": 153.84, "text": " head on over to assembly AI use the link in the description to let them know that I sent you"}, {"start": 153.84, "end": 159.6, "text": " there the single API to transcribe and understand audio they do so in batch and in real time via"}, {"start": 159.6, "end": 165.20000000000002, "text": " web socket, they accept all kinds of audio and video formats and they do so in over 15 languages,"}, {"start": 165.2, "end": 169.51999999999998, "text": " give it a try. And thank you very much to assembly AI for sponsoring this video. And now let's get"}, {"start": 169.51999999999998, "end": 176.88, "text": " into the video. The topic of sparsity is a big thing in neural networks right now, mostly because"}, {"start": 176.88, "end": 184.23999999999998, "text": " we have no idea really how to do it. And I think that's exciting times for the future. So welcome"}, {"start": 184.23999999999998, "end": 192.23999999999998, "text": " what what brings you into the sparse world? Actually, I, you know, I've been a professor of"}, {"start": 192.24, "end": 200.0, "text": " computer science for many years, and I worked on multi course for more than 30 years, and"}, {"start": 201.28, "end": 209.12, "text": " got involved in computational neurobiology in the last 10 years. And one of the things that"}, {"start": 209.12, "end": 215.76000000000002, "text": " you really see in the brain is really how sparse its computation is, it really is very, very sparse."}, {"start": 215.76, "end": 222.79999999999998, "text": " And so, you know, looking at neural networks, we see that there are there's a similar phenomenon"}, {"start": 222.79999999999998, "end": 229.84, "text": " to what happens in brains happening in neural networks, right, where you can actually reduce"}, {"start": 229.84, "end": 236.64, "text": " the number of parameters through pruning by huge amounts and preserve accuracy of the performance"}, {"start": 236.64, "end": 243.35999999999999, "text": " of the network. And that kind of says, okay, if we really want to have brain like performance,"}, {"start": 243.36, "end": 249.36, "text": " you know, sparsity is probably one of the tools that we want to use to get there. So that's kind"}, {"start": 249.36, "end": 259.28000000000003, "text": " of how I kind of got into this. And you founded a company that also works into this direction,"}, {"start": 259.28000000000003, "end": 266.16, "text": " right? You want to talk about that? Yeah, a little bit. Yes, I founded Neural Magic. Neural Magic was"}, {"start": 266.16, "end": 272.24, "text": " founded because what we were seeing in my lab, I was busy with doing machine learning at a large"}, {"start": 272.24, "end": 279.52, "text": " scale for biology projects. And what we realized was that we could get CPUs to run at GPU speeds"}, {"start": 279.52, "end": 286.0, "text": " like at the time, it was a Pascal GPU. And you can make just a regular CPU do what the Pascal GPU"}, {"start": 286.0, "end": 293.36, "text": " was doing through the use of sparsity and other similar techniques. And so we said, okay, well,"}, {"start": 293.36, "end": 298.08, "text": " there's a real commercial value here for people because you don't need an accelerator, you can"}, {"start": 298.08, "end": 304.15999999999997, "text": " just do it on your commodity CPU. And that's that's Neural Magic. So what we do is we deliver,"}, {"start": 304.15999999999997, "end": 310.15999999999997, "text": " you know, through sparsity and similar optimization techniques, GPU performance on CPUs."}, {"start": 310.71999999999997, "end": 316.24, "text": " That is, it's quite a promise. Maybe let's first dive into a little bit about sparsity itself. What"}, {"start": 316.24, "end": 322.56, "text": " is it about sparsity? You mentioned the brain is very sparse. Yet our current or at least the way"}, {"start": 322.56, "end": 328.16, "text": " we train neural network is very dense, we can accelerate the dense neural networks much better."}, {"start": 328.16, "end": 337.36, "text": " What is it about sparsity? Is it just the saving of parameters? Or is there something more to sparse"}, {"start": 337.36, "end": 343.84000000000003, "text": " connections than to dense connections? What do we know? That's a good question. So clearly,"}, {"start": 343.84000000000003, "end": 348.96, "text": " what we're doing today is not the sparsity that we will be doing in the future. And what I mean by"}, {"start": 348.96, "end": 356.96, "text": " that is your brain is sparse way beyond the levels of what we see in neural networks today. So your"}, {"start": 356.96, "end": 363.91999999999996, "text": " typical brain in terms of the compute, right, you know, your cortex is like a cell phone of compute,"}, {"start": 363.91999999999996, "end": 370.15999999999997, "text": " right? But the graph is enormous. It's like, you know, the graph is the size and you need petabytes"}, {"start": 370.15999999999997, "end": 377.52, "text": " to basically hold it. So a cell phone of compute on a petabyte or more of memory, right? But the"}, {"start": 377.52, "end": 384.47999999999996, "text": " accelerators that we build, you know, are designed to deliver petaflops of compute. But on a cell"}, {"start": 384.47999999999996, "end": 389.03999999999996, "text": " phone size memory, the memory is very limited because we use this high bandwidth memory. So"}, {"start": 389.03999999999996, "end": 395.44, "text": " in a sense, we're building the opposite of what we want, right? So if we want to mimic the brain,"}, {"start": 395.44, "end": 400.47999999999996, "text": " we should not busy ourselves so much with the amount of compute and rather worry about"}, {"start": 400.47999999999996, "end": 406.56, "text": " how it is that we implement this very large graph. It's a very large graph, but it's extremely sparse."}, {"start": 406.56, "end": 412.08, "text": " That's the point, right? And as you asked, the sparsity is not necessarily the same sparsity"}, {"start": 412.08, "end": 417.6, "text": " that we do today through pruning techniques, but it's a combination of a very sparse architecture"}, {"start": 418.16, "end": 424.8, "text": " together with, you know, a sparsity in what we call in machine learning the kernel, right? So"}, {"start": 424.8, "end": 430.8, "text": " it's not just that the kernels are sparse, but everything in the design is very, very sparse."}, {"start": 430.8, "end": 439.76, "text": " Okay. And we don't know yet how to design very sparse architectures. Part of that has to do"}, {"start": 439.76, "end": 447.68, "text": " with the fact that machine learning grew up in the GPU world where sparsity is not an advantage"}, {"start": 447.68, "end": 453.28000000000003, "text": " actually, because you're doing lockstep computation. So you win nothing by being very sparse."}, {"start": 453.28000000000003, "end": 458.56, "text": " And therefore, you know, we don't see those architectural sparsity things yet."}, {"start": 458.56, "end": 466.16, "text": " But I'm expecting that to happen. We should be, this should come along, you know. And even more"}, {"start": 466.16, "end": 473.92, "text": " than that, what I expect is things are starting to show up like the pathways from models from"}, {"start": 473.92, "end": 481.12, "text": " Google and so on where even if you have a very large model, you don't execute the full model"}, {"start": 481.12, "end": 487.12, "text": " layer after layer, but rather you execute small regions of the model at any given time."}, {"start": 487.12, "end": 493.52, "text": " Per input. That's another form of sparsification of your computation. Right. And that is what the"}, {"start": 493.52, "end": 499.6, "text": " brain really does. So your brain typically, you know, when you see an input or so on, uses a very"}, {"start": 499.6, "end": 506.64, "text": " small fraction of its total graph to do the computation. And so that's where we're headed."}, {"start": 506.64, "end": 513.44, "text": " We're not there yet. We don't know how to do it. But this is the goal. And that's the old, you only"}, {"start": 513.44, "end": 519.84, "text": " use 10 percent of the brain at any given time. Right. Yeah, that's right. I mean, really,"}, {"start": 519.84, "end": 525.6800000000001, "text": " from energy considerations, it really is like a cell phone. OK, it really isn't, you know,"}, {"start": 525.6800000000001, "end": 534.48, "text": " this massive monster multi GPU thing that we use today. And so my expectation is that, you know,"}, {"start": 535.0400000000001, "end": 541.44, "text": " that as we learn more and more about how to design sparse networks, we're going to see them become"}, {"start": 541.44, "end": 546.48, "text": " the standard. They're not the standard right now because we started the whole journey,"}, {"start": 546.48, "end": 554.72, "text": " right, by applying flops and still applying flops is the main paradigm. But we will see it appear"}, {"start": 554.72, "end": 563.6, "text": " both in hardware and accelerators and in CPUs. This idea that we can utilize sparsity, you know,"}, {"start": 563.6, "end": 572.08, "text": " to get really great performance gains. Yeah, that's coming. Now is the question is a little bit"}, {"start": 572.08, "end": 579.36, "text": " the chicken and the egg problem. Is the brain sparse because it has the limitations of the cell"}, {"start": 579.36, "end": 587.0400000000001, "text": " phone power or does the brain only need cell phone power because sparsity is such a good architecture?"}, {"start": 587.04, "end": 599.4399999999999, "text": " Right. Like which causes which? Yeah. So I would say that, you know, the whole notion of parallelism"}, {"start": 599.4399999999999, "end": 606.24, "text": " in the brain, right. If you think about it, imagine that you need to do a billion operations per"}, {"start": 606.24, "end": 613.92, "text": " second. OK, and what you have are these very slow chemical devices, neurons, right, that can do that."}, {"start": 613.92, "end": 619.4399999999999, "text": " Right. So you need a billion operations, a billion firings of neurons in a second. How are you going"}, {"start": 619.4399999999999, "end": 623.92, "text": " to do that? Well, what you need is massive parallelism. Right. You've got to get massive"}, {"start": 623.92, "end": 631.5999999999999, "text": " parallelism. If you can do the massive parallelism, you can get the billion operations. Right. And so"}, {"start": 631.5999999999999, "end": 639.1999999999999, "text": " our brains are parallel, if you will, because we have this special medium. Right. Now on a modern"}, {"start": 639.2, "end": 645.9200000000001, "text": " multiprocessor, right, you can get a billion or 10 billion instructions executed per second"}, {"start": 645.9200000000001, "end": 652.0, "text": " sequentially. You don't really need parallelism for it. Right. And so what I'm trying to say is, you"}, {"start": 652.0, "end": 660.32, "text": " know, the whole idea of kind of how brains evolve is clearly because of the way, you know, they're"}, {"start": 660.32, "end": 668.4000000000001, "text": " implemented. But we should not think of going and implementing this in some way."}, {"start": 668.4, "end": 674.88, "text": " Right. And we should not think of going and implementing this in silicon in the same way."}, {"start": 674.88, "end": 681.36, "text": " Right. Because we really what we really should think about just is that both of these things are"}, {"start": 681.36, "end": 687.04, "text": " train complete. Right. You can do you can implement the algorithm. You just need to know what the"}, {"start": 687.04, "end": 694.3199999999999, "text": " algorithm is. And then on silicon will implement the best algorithm we can. Right. You know, of"}, {"start": 694.32, "end": 700.48, "text": " course, we need a lot of time to do that. OK. Does that make sense? That's what I'm trying to say."}, {"start": 700.48, "end": 706.6400000000001, "text": " You know, let's implement the algorithm, but not necessarily the architecture. OK. So when I say"}, {"start": 706.6400000000001, "end": 712.88, "text": " sparsity, I really mean sparsity algorithmic sparsity. Right. And it doesn't mean that you have"}, {"start": 712.88, "end": 719.5200000000001, "text": " to have a very sparse kind of, you know, silicon VLSI circuit to do this. That's not the case."}, {"start": 719.52, "end": 726.48, "text": " Right. Given that we that's a good segue given that we do have the flops right that we don't"}, {"start": 726.48, "end": 731.36, "text": " have in the brain. It naturally it is a different a different system. We do have"}, {"start": 731.36, "end": 738.8, "text": " teraflops petaflops even in these giant compute clusters. Where should we put them in your opinion?"}, {"start": 738.8, "end": 745.28, "text": " Like where where should that extra resource that the brain doesn't have go? Should it go into"}, {"start": 745.28, "end": 750.48, "text": " sequentially executing what the brain executes in parallel or, you know, where should we put that?"}, {"start": 751.4399999999999, "end": 759.12, "text": " So first I want to say is that we have those flops, but they're costing us a lot. And you just"}, {"start": 759.12, "end": 765.12, "text": " have to open the papers to see what the cost of the flops is. It's enormous, an enormous energy"}, {"start": 765.12, "end": 772.88, "text": " drain. And it's also an enormous architectural drain on what we're doing. And so I would say"}, {"start": 772.88, "end": 779.36, "text": " we want to get rid of the flops because probably we don't need them. OK. And especially as you go"}, {"start": 779.36, "end": 786.88, "text": " from the data center down to the edge, you get the capability of delivering flops comes directly at"}, {"start": 786.88, "end": 791.92, "text": " the you know, if at the edge you can put the sorry in the data center, you can put, you know,"}, {"start": 791.92, "end": 797.76, "text": " your Google data warehouse right next to a waterfall or whatever you want, right,"}, {"start": 797.76, "end": 802.32, "text": " to a source of energy. Right. When you're doing this on your cell phone or on a tiny device at"}, {"start": 802.32, "end": 809.6800000000001, "text": " the edge, every little bit of energy that you waste is critical for you. Right. And so what"}, {"start": 809.6800000000001, "end": 815.44, "text": " we really want to do is move away from the flops and move more towards the very energy efficient"}, {"start": 815.44, "end": 823.36, "text": " way the brains work because this adding more flops is a momentary thing for us. Right. So, yes,"}, {"start": 823.36, "end": 829.5200000000001, "text": " we can do this, but at a very high cost. And no, we don't want to do this forever. We want to find"}, {"start": 829.52, "end": 836.96, "text": " ways to cut the cost, reduce the compute. And there's a little other thing that I want to say,"}, {"start": 836.96, "end": 843.92, "text": " and that is architecturally we generate the flops by running right now, at least by running many,"}, {"start": 843.92, "end": 851.04, "text": " many, many tiny cores, thousands of tiny cores typically. Right. And in architectures that"}, {"start": 851.04, "end": 856.48, "text": " require a lot of connections to the memory, this high bandwidth memory. And this thing doesn't"}, {"start": 856.48, "end": 863.04, "text": " scale. So in a sense, we're trading flops for memory. If you use the CPU today, you could get"}, {"start": 863.04, "end": 871.36, "text": " a terabyte on your desktop, but go get a terabyte on a GPU. Right. And so using the flops is going"}, {"start": 871.36, "end": 875.76, "text": " to enable us changing the architecture. If we don't need so many flops, then we can actually"}, {"start": 875.76, "end": 881.36, "text": " increase the size of our memory, which will make us able to hold these giant models that we want to"}, {"start": 881.36, "end": 889.36, "text": " do very cheaply, if you will. If I explain a deep neural network to someone, I usually, you know,"}, {"start": 889.36, "end": 894.64, "text": " you start with a fully connected layer, you say, you know, here is a layer of neurons, and here is"}, {"start": 894.64, "end": 898.24, "text": " a layer of neurons, and they have their connections, right, and each connection has a"}, {"start": 898.24, "end": 903.76, "text": " little weight, and so on, you usually describe like a dense, fully connected architecture. And"}, {"start": 903.76, "end": 911.2, "text": " that is conceptually, I want to say, easy to grasp for people and so on. Do you have an analogy for"}, {"start": 911.2, "end": 918.6400000000001, "text": " sparse architectures? Like, what is the conceptual, like, could you conceptualize to someone who"}, {"start": 918.6400000000001, "end": 923.84, "text": " doesn't know what like a sparse architecture is and how to think about it? What is different?"}, {"start": 924.8000000000001, "end": 930.24, "text": " Yeah, the way we do sparsity today, I don't know what it'll look like in the future. But today,"}, {"start": 930.24, "end": 935.2, "text": " sparsity looks like, imagine that the two layers of the neural network are these kind of,"}, {"start": 935.2, "end": 941.12, "text": " there are cords from one layer to the next, right, there are strings attached, and these are, of"}, {"start": 941.12, "end": 945.5200000000001, "text": " course, these are the connections, the weights that we're using in the computation, right. And"}, {"start": 946.08, "end": 951.0400000000001, "text": " sparsity means I take scissors, and I chop, chop, chop, chop, chop, you know, till I have"}, {"start": 951.0400000000001, "end": 957.36, "text": " five or 10% of those cords left, right. And those cords, it turns out, right, if I do this right,"}, {"start": 957.36, "end": 965.36, "text": " if I do this kind of pruning, right, are good enough to capture, right, the accuracy of the"}, {"start": 965.36, "end": 970.72, "text": " model as it was before, because a lot of the connections are not important for this process."}, {"start": 970.72, "end": 978.5600000000001, "text": " That's kind of the big discovery. And modern research in techniques for sparsification, right,"}, {"start": 979.6800000000001, "end": 984.5600000000001, "text": " you know, play along this kind of game. So you can do this kind of unstructured thing that I just"}, {"start": 984.56, "end": 989.4399999999999, "text": " described, where you arbitrarily cut in many places based on the effectiveness, or you can"}, {"start": 989.4399999999999, "end": 995.5999999999999, "text": " also structurally take things out. So in a lot of the modern models, right, we're removing pieces"}, {"start": 995.5999999999999, "end": 1003.92, "text": " that are not necessary. We do architecture search to find these places to cut things, right. So"}, {"start": 1003.92, "end": 1009.76, "text": " that's where the whole game right now of efficiency and neural networks, right, is the game of how do"}, {"start": 1009.76, "end": 1017.2, "text": " I cut this thing down, right. In the brain, there are certainly some systems like the visual system,"}, {"start": 1017.2, "end": 1022.3199999999999, "text": " where that is clearly organized into layers, but there are many other systems that have no"}, {"start": 1022.88, "end": 1028.72, "text": " resemblance to layers, there are connections going up and down and left and right. And, you know,"}, {"start": 1028.72, "end": 1036.64, "text": " between the halves of the brain and all, is there a possible future where this could become into"}, {"start": 1036.64, "end": 1042.3200000000002, "text": " like a standard architectures for neural networks, that the notion of layers and things like this"}, {"start": 1042.3200000000002, "end": 1048.8000000000002, "text": " isn't even really a, you know, a thing anymore? Or is there, you know, some some fundamental way"}, {"start": 1048.8000000000002, "end": 1054.0, "text": " where we say, no, there's probably always going to be layers, but it's just going to be a sparsity"}, {"start": 1054.0, "end": 1060.4, "text": " between those layers. So when we look at, you know, we have a full connectome of essentially"}, {"start": 1060.4, "end": 1066.88, "text": " only a couple of animals, a worm and a fruit fly, that's it. And that's it, you don't see a lot of"}, {"start": 1066.88, "end": 1077.0400000000002, "text": " layering there, it looks more like a mess, very sparse mess, okay. And I wouldn't venture to"}, {"start": 1077.0400000000002, "end": 1084.0, "text": " think about what a cortex looks like, right. We don't have that yet, we're working very hard,"}, {"start": 1084.0, "end": 1090.64, "text": " to these are very hard computational problems to be able to go and get a model. We just want to do"}, {"start": 1090.64, "end": 1097.12, "text": " a mouse, even a mouse is just too big for us to do right now, like a small mammal, right. But my,"}, {"start": 1097.12, "end": 1102.88, "text": " I would venture to guess that yes, the answer is that, you know, it's extremely, it's an extremely"}, {"start": 1102.88, "end": 1110.64, "text": " sparse architecture, and that it wouldn't, it will not look like layers, okay. You can impose a layer"}, {"start": 1110.64, "end": 1117.44, "text": " structure on any graph, okay. It's not, so the idea that I say there are layers, sure, okay,"}, {"start": 1117.44, "end": 1122.8000000000002, "text": " I can take the graph and I can layer it, yeah, I could do a BFS on it and layer it. But the point"}, {"start": 1122.8000000000002, "end": 1129.44, "text": " is not so much that it's more that by design, when I think about it, right, I'm not going to think"}, {"start": 1129.44, "end": 1135.2, "text": " about it as a sequence of layers where the change that I make is the change in the layer, one layer"}, {"start": 1135.2, "end": 1140.72, "text": " is different from the other. But rather, it'll be a combination of thinking about paths, different"}, {"start": 1140.72, "end": 1147.3600000000001, "text": " paths, and I'll do different things along different paths. That's kind of the idea, you know. If you"}, {"start": 1147.3600000000001, "end": 1155.1200000000001, "text": " think about, you know, there's recent research from MIT, you know, you can detect, people can"}, {"start": 1155.12, "end": 1165.52, "text": " detect an image in 0.13, set 0.013 seconds, in 13 milliseconds, okay. In 13 milliseconds, you can detect,"}, {"start": 1165.52, "end": 1172.32, "text": " you can say what an image is, okay. This is, there's no time for neurons to fire. This thing is"}, {"start": 1172.32, "end": 1178.8, "text": " extremely kind of parallel, right, and uses very little compute and gets you an answer. And a"}, {"start": 1178.8, "end": 1185.6, "text": " large part of that is prediction, because you're already expecting something. So we need to learn"}, {"start": 1185.6, "end": 1193.04, "text": " how to do those things. And so machine learning right now is in a very naive early stage. And so"}, {"start": 1193.04, "end": 1198.1599999999999, "text": " given that, and given the things that we are doing right now, it's not a surprise that"}, {"start": 1198.1599999999999, "end": 1203.9199999999998, "text": " we're doing the brute force kind of massive compute kind of thing. That's always what you do. And with"}, {"start": 1203.92, "end": 1210.4, "text": " time, we're going to get better and better at it, right. So that's kind of how I see this progressing."}, {"start": 1211.52, "end": 1218.48, "text": " Speaking of becoming better, if you know, the flatworm is sparse, the mouse is sparse, the human"}, {"start": 1218.48, "end": 1226.8000000000002, "text": " is certainly sparse, yet our best models today are all big, dense, you know, computation hungry"}, {"start": 1226.8, "end": 1234.3999999999999, "text": " things, there is not really a case, every time I prune, I sparse if I and so on, I get savings in"}, {"start": 1234.3999999999999, "end": 1241.52, "text": " like, you know, savings in CPU or GPU, I get savings in, you know, my storage, but I also get"}, {"start": 1241.52, "end": 1247.6, "text": " like a little bit worse, right? That's the common thing today in pruning is that I get like just a"}, {"start": 1247.6, "end": 1253.9199999999998, "text": " tiny bit worse than the dense model I prune from. Why do you do you think that is just the fact that"}, {"start": 1253.92, "end": 1260.8000000000002, "text": " we prune from a dense model? Or what's holding back the sparse models? How about if I turn this"}, {"start": 1260.8000000000002, "end": 1268.96, "text": " around? Let me turn this around for you. Okay, you can take BERT base, which is a common model"}, {"start": 1268.96, "end": 1279.6000000000001, "text": " that people use, okay. And you can sparsify BERT base. Neural magic, we sparsify at 95%. So a 95%"}, {"start": 1279.6, "end": 1286.8799999999999, "text": " sparse BERT base, one over 20th of the compute, okay, way beyond anything a GPU does, even if you"}, {"start": 1286.8799999999999, "end": 1291.76, "text": " run it with full throttle, okay, it's just cutting the compute so much that there's really almost"}, {"start": 1291.76, "end": 1297.1999999999998, "text": " nothing to compute there. It's just moving data, okay. Now I'm exaggerating, of course, but, you"}, {"start": 1297.1999999999998, "end": 1302.24, "text": " know, it's really becomes a data movement problem rather than a compute problem when you, and you"}, {"start": 1302.24, "end": 1310.72, "text": " lose 1%, less than 1% accuracy, okay. And I say, okay, great. So you've done that, you know, and"}, {"start": 1310.72, "end": 1315.6, "text": " you've gotten all this speed up, but you've lost, you say, oh, near, but you lost less than 1%"}, {"start": 1315.6, "end": 1322.08, "text": " accuracy. But what I say instead is forget that. Take BERT large, a much more accurate model,"}, {"start": 1322.08, "end": 1328.64, "text": " several points more accurate than BERT base, okay, and prune it so that it actually, right,"}, {"start": 1328.64, "end": 1336.48, "text": " with 20x less compute, it's actually faster than BERT base, okay. And so now you have the accuracy,"}, {"start": 1337.0400000000002, "end": 1343.5200000000002, "text": " right, and you have great compute, and this is through sparsity. So by sparsifying the larger"}, {"start": 1343.5200000000002, "end": 1349.76, "text": " model, I actually delivered you the best of both worlds, little compute and great accuracy. And"}, {"start": 1349.76, "end": 1355.0400000000002, "text": " that's how I want you to think about sparsity, right. It's a way of enabling us to run much"}, {"start": 1355.04, "end": 1362.8799999999999, "text": " larger, more accurate dense models, but because we sparsified them, we are, you know, we're getting"}, {"start": 1362.8799999999999, "end": 1365.76, "text": " great performance. That's how to think about it."}, {"start": 1367.04, "end": 1372.96, "text": " What's the limit currently that keeps us from, we always need the dense model first in this model"}, {"start": 1372.96, "end": 1377.44, "text": " in the pruning, in a pruning setup, we first need the dense model, then we go to the sparse model,"}, {"start": 1377.44, "end": 1383.12, "text": " we get huge savings at inference time. What keeps us from just building the sparse model in the"}, {"start": 1383.12, "end": 1391.04, "text": " first place? Great. So this is kind of the lottery ticket kind of question, if you will. There is"}, {"start": 1391.04, "end": 1396.8799999999999, "text": " research, actually, Dan Alistair, one of our consultants, Neural Magic works exactly on this"}, {"start": 1396.8799999999999, "end": 1407.12, "text": " kind of stuff. We know how to run a training session right now for four models where you start"}, {"start": 1407.12, "end": 1414.2399999999998, "text": " out and you need to do only a certain fraction of the, you know, of the forward passes, backward"}, {"start": 1414.2399999999998, "end": 1419.6799999999998, "text": " passes, dense, and then immediately you can already start pruning while training. So there is"}, {"start": 1419.6799999999998, "end": 1424.8, "text": " research going in that direction. But you are right that right now, at least, right in the"}, {"start": 1425.6799999999998, "end": 1431.04, "text": " standard, if you look at what's going on there out there, standardly, you're right, we do most of the"}, {"start": 1431.04, "end": 1440.48, "text": " time take a standard model and from dense we sparse it and so on. But the thing to remember,"}, {"start": 1440.48, "end": 1443.6, "text": " and this now I'm not talking about the research, because the research is going to get there."}, {"start": 1444.1599999999999, "end": 1450.3999999999999, "text": " Janek, I don't know if to what extent we will, how fast this will happen and so on, but we will"}, {"start": 1450.3999999999999, "end": 1457.2, "text": " learn how to build sparse architectures that start sparse and continues, you know, it's really a"}, {"start": 1457.2, "end": 1462.72, "text": " matter, nature does this. And so there's no reason why we wouldn't be able to do it. But I want to"}, {"start": 1462.72, "end": 1468.32, "text": " say something about today's machine learning where you kind of start with the dense and then you have"}, {"start": 1468.32, "end": 1475.6000000000001, "text": " to sparsify. This is really not the common paradigm for most users of neural networks. For most users,"}, {"start": 1475.6000000000001, "end": 1483.3600000000001, "text": " a model is given to them that, you know, from a known architecture, right, and then they transfer"}, {"start": 1483.36, "end": 1489.36, "text": " learn onto it. And most people do that rather than train from scratch. They really use the model that"}, {"start": 1489.36, "end": 1494.3999999999999, "text": " somebody already worked very hard to build for their specific use case, and then they transfer"}, {"start": 1494.3999999999999, "end": 1500.0, "text": " learn onto it. So this is what you can do with sparsity. You can take a sparse model and sparse"}, {"start": 1500.0, "end": 1504.6399999999999, "text": " transfer learn onto it. It's extremely efficient because you're running at the speed of the sparse"}, {"start": 1504.6399999999999, "end": 1510.7199999999998, "text": " network, right? So you can sparse transfer and then you don't need all of this kind of start with"}, {"start": 1510.72, "end": 1518.88, "text": " dense. And we're seeing more and more sparse networks appear, you know, in the literature and"}, {"start": 1518.88, "end": 1526.72, "text": " in the database collections of machine learning models. And as we have more and more of these"}, {"start": 1526.72, "end": 1532.88, "text": " initial good sparse models, right, people are going to learn to start with the sparse already."}, {"start": 1532.88, "end": 1536.72, "text": " That's kind of commercially, I think that's what we're going to see more and more of."}, {"start": 1536.72, "end": 1546.08, "text": " Why, you mentioned this a bit already, but why are GPUs so unsuited for sparse models? And what makes"}, {"start": 1546.08, "end": 1552.08, "text": " CPUs in the way you do it, really suited for sparse models? Or are they even suited? Or are you"}, {"start": 1552.08, "end": 1559.84, "text": " simply, you know, seeing that they're better? Yeah, I mean, look, the the GPU architecture,"}, {"start": 1559.84, "end": 1567.12, "text": " you know, is designed for this very, you know, small cores, tiny caches. You're not going to go"}, {"start": 1567.12, "end": 1572.24, "text": " and throw all that away just because, you know, you found, you discovered sparsity. So you're"}, {"start": 1572.24, "end": 1579.12, "text": " trying to do sparsity while keeping this kind of lockstep execution structure, right? And this is"}, {"start": 1579.12, "end": 1586.08, "text": " difficult to do sparse. You need, you need, you need, you need really a different kind of setup"}, {"start": 1586.08, "end": 1593.6799999999998, "text": " to get an advantage out of sparsity. Now, now I'm not, it's not like you can't do that, right?"}, {"start": 1593.6799999999998, "end": 1601.12, "text": " It's not like you can't do that. People can design and have designed hardware that utilizes sparsity"}, {"start": 1601.12, "end": 1607.6799999999998, "text": " efficiently, okay? There is such hardware. It's just not, it's not GPU-like. It's not like the"}, {"start": 1607.6799999999998, "end": 1614.32, "text": " accelerators that we have today. But all of these, again, all of these accelerators have a different"}, {"start": 1614.32, "end": 1620.0, "text": " problem that has just to do with the memory. Because of the way they're designed, right, they"}, {"start": 1620.0, "end": 1626.48, "text": " typically have very small memories. So we're talking even, even ones that can run sparse, right,"}, {"start": 1626.48, "end": 1633.12, "text": " still have the limitation of their memory size. So the reason that CPUs are attractive is not so much"}, {"start": 1633.12, "end": 1638.3999999999999, "text": " that, you know, that they, that you have a natural way of running sparsity because you can run"}, {"start": 1638.3999999999999, "end": 1643.28, "text": " asynchronous with large cores, but rather that the large cores, you know, they're not going to"}, {"start": 1643.28, "end": 1650.8, "text": " run all the time. So the large cores enable you very easy access to very large memory pools,"}, {"start": 1650.8, "end": 1658.08, "text": " right? So the advantage of having strong, powerful cores, right, is really that I can put several"}, {"start": 1658.08, "end": 1664.56, "text": " terabytes of memory next to them, right, and run easily. And that's where the big advantage is"}, {"start": 1664.56, "end": 1670.8799999999999, "text": " going to be. As we understand more and more about how to build giant models that don't run all the"}, {"start": 1670.88, "end": 1676.64, "text": " time, right, then the compute will be less important, but actually the ability to hold that"}, {"start": 1676.64, "end": 1683.0400000000002, "text": " model in one place and run it rather than break it apart on eight or 16 GPUs, that's going to be"}, {"start": 1683.0400000000002, "end": 1687.92, "text": " your advantage. And so this is, so I'm kind of saying it's not so much that you can't build a"}, {"start": 1687.92, "end": 1694.5600000000002, "text": " hard piece of hardware to run sparsity, you can, right, but you should build it looking like a CPU"}, {"start": 1694.56, "end": 1701.84, "text": " in the sense of you can access a lot of memory because you're not doing tiny cores. That's kind of,"}, {"start": 1701.84, "end": 1708.3999999999999, "text": " that's my two cents. So the the CPUs are good because they have, you know, fast connect to"}, {"start": 1708.3999999999999, "end": 1714.48, "text": " large memory, but also over the years, we've put more and more levels of cache onto the CPU. How"}, {"start": 1714.48, "end": 1719.76, "text": " much do you have to have to take this into account when you're building? I mean, you're maybe you can"}, {"start": 1719.76, "end": 1725.84, "text": " explain a little bit what your company does in terms of software, you build compilers, or can I"}, {"start": 1725.84, "end": 1733.2, "text": " just run TensorFlow or something? Yeah, so let me explain. So first of all, the connection between"}, {"start": 1733.2, "end": 1739.36, "text": " the CPU and the memory is slow. GPU has a faster memory and faster access to it, right, smaller,"}, {"start": 1739.36, "end": 1746.0, "text": " but fast, right? CPU memory is slow, but large, very large. But CPUs have a cache hierarchy,"}, {"start": 1746.0, "end": 1751.68, "text": " as you said. And so if you know how to utilize your cache hierarchy, then, you know, if you're"}, {"start": 1751.68, "end": 1757.44, "text": " running in the L1 cache of the CPU, okay, you're running as fast as the GPU. There's nothing there"}, {"start": 1757.44, "end": 1762.48, "text": " that the GPU does that the CPU can't do once you're in cache. Okay, in fact, CPU caches are"}, {"start": 1762.48, "end": 1768.96, "text": " much faster than GPU caches, and the performance is better. So the question then, right, and this"}, {"start": 1768.96, "end": 1774.32, "text": " is what NeuralMagic does is, okay, so what we do is we sparsify the model. Now, you know, if the"}, {"start": 1774.32, "end": 1780.8799999999999, "text": " pro, you know, machine learning is about, okay, I need to meet a certain latency. And because I"}, {"start": 1780.8799999999999, "end": 1786.48, "text": " couldn't meet that latency with the CPU, then we added the GPU and boom, there's machine learning"}, {"start": 1786.48, "end": 1793.12, "text": " with GPUs. Now I can meet the latency. But there's two ways to deal with latency. One is to add more"}, {"start": 1793.12, "end": 1798.48, "text": " flops, and the other is to reduce the flops, right? And so sparsity, instead of adding more"}, {"start": 1798.48, "end": 1804.4, "text": " flops in hardware, reduces the number of flops needed in software. But now that you have this"}, {"start": 1804.4, "end": 1812.56, "text": " very sparse model, because the CPU memory is slow, okay, then what happens is you hit a bottleneck,"}, {"start": 1812.56, "end": 1817.04, "text": " and it's very hard to move. If you do this layer after layer, it's very hard to move the data in"}, {"start": 1817.04, "end": 1823.28, "text": " and out, okay? So what NeuralMagic invented is a way of running neural networks depth-wide. So we"}, {"start": 1823.28, "end": 1829.28, "text": " have this technology, which we call tensor columns, where essentially you can run, okay, you know, you"}, {"start": 1829.28, "end": 1836.0, "text": " can break the model lengthwise and run, you know, each one of these kind of columns, you know, in"}, {"start": 1836.0, "end": 1843.28, "text": " cache, okay? And because you're not leaving L2 really, or rarely leaving L2, you know, you actually"}, {"start": 1843.28, "end": 1848.96, "text": " get great performance. So in a sense, right, what we're doing is we're using the natural ability of"}, {"start": 1848.96, "end": 1854.88, "text": " CPUs to prefetch things from memory and then run in cache. And because this, you know, this cache"}, {"start": 1854.88, "end": 1862.8, "text": " hierarchy on CPUs has evolved over 70 years, or maybe I'm exaggerating, 60 years of hardware design,"}, {"start": 1862.8, "end": 1869.28, "text": " it's a very, very well understood thing where people know how to optimize it, right? Especially"}, {"start": 1869.28, "end": 1876.72, "text": " the big, you know, chip makers, they really know how to make these caches work really well. And so"}, {"start": 1876.72, "end": 1885.84, "text": " with these really good cache hierarchies, you really get great performance by running the"}, {"start": 1885.84, "end": 1892.0, "text": " model depth-wise. So that's NeuralMagic, you know, we take the model, sparsify it, now it doesn't need"}, {"start": 1892.0, "end": 1897.2, "text": " the compute, and now we run it on the CPU and get speed because we're running in cache, okay? And if"}, {"start": 1897.2, "end": 1903.44, "text": " you look at the numbers, I mean, you know, we are, you know, at the speed of, I mean, some numbers we"}, {"start": 1903.44, "end": 1908.8, "text": " have in PubG, we're at the speed of an A100, even faster, in terms of how long it takes. A four-core"}, {"start": 1908.8, "end": 1917.8400000000001, "text": " CPU can, in terms of latency, do what an A100 does on a common model like BERT, okay? So it's really"}, {"start": 1917.8400000000001, "end": 1924.0, "text": " the emphasis... Given that it's sparse or... Yes, yes, yes, by sparsifying it and running it, you can make a"}, {"start": 1924.0, "end": 1929.68, "text": " four-core do what A100 does. So it's really now a matter of throughput, and the A100 has a lot of"}, {"start": 1929.68, "end": 1935.2, "text": " throughput, okay? So now the question is, you know, how many cores do you want on your CPU to meet the"}, {"start": 1935.2, "end": 1940.72, "text": " throughput of the A100? And again, the story is that, you know, the big providers are adding more"}, {"start": 1940.72, "end": 1947.68, "text": " and more and more cores, so you're going to be able to compete better with the GPUs down the road."}, {"start": 1947.68, "end": 1955.6000000000001, "text": " So that's kind of the story of NeuralMagic. So the way I can imagine these tensor columns is"}, {"start": 1955.6, "end": 1961.76, "text": " that because I execute depth-wise, the sort of values that I need for the next step in the"}, {"start": 1961.76, "end": 1968.8799999999999, "text": " computation are the results of the very last step, therefore are already going to be in cache. And"}, {"start": 1968.8799999999999, "end": 1975.36, "text": " since everything's sparse, I don't need all of the last layer for the current step, and therefore,"}, {"start": 1975.36, "end": 1981.12, "text": " you know, I have it already. Okay. Right, and of course, you know, when you think about neural"}, {"start": 1981.12, "end": 1985.4399999999998, "text": " networks, there are overlaps between these columns. And the question is, how do you deal with the"}, {"start": 1985.44, "end": 1990.48, "text": " overlaps in a way that doesn't kill your computation? And that's the magic. That's the magic of it."}, {"start": 1990.48, "end": 1995.3600000000001, "text": " There's an algorithm that allows you to do that. And because you can do it, you manage to run this"}, {"start": 1995.3600000000001, "end": 2004.0, "text": " way, and you don't hit this memory bottleneck, and boom, you're in business. Yeah. So for GPU,"}, {"start": 2004.0, "end": 2010.72, "text": " it's almost like, you know, GPUs enable us to do dense models. But I think also models have almost"}, {"start": 2010.72, "end": 2016.8, "text": " co-evolved with the GPU. So people have started building models to fit the GPU architectures"}, {"start": 2016.8, "end": 2021.28, "text": " better, right? Especially something like a transformer is like, that's it, that's like"}, {"start": 2021.28, "end": 2030.48, "text": " made for GPUs. Is there a type of sparse model? Like if you if you could wish for the best possible"}, {"start": 2031.1200000000001, "end": 2037.28, "text": " sparse, but you know, there's different kinds of sparsity, like, what is the best type of sparsity"}, {"start": 2037.28, "end": 2043.04, "text": " to let's say, execute on a CPU, if we want to look forward, and we want to especially build"}, {"start": 2043.04, "end": 2048.88, "text": " architectures? Yeah, this goes back to your original, one of the first questions you asked,"}, {"start": 2048.88, "end": 2055.36, "text": " right? It's about a different structure for the neural network execution. So we should forget the"}, {"start": 2055.36, "end": 2061.7599999999998, "text": " synchronous layer after layer execution. And think about the fact that, you know, we can run through"}, {"start": 2061.76, "end": 2068.7200000000003, "text": " a model, right, in multiple paths with multiple computing units, use the same weight structure,"}, {"start": 2068.7200000000003, "end": 2074.6400000000003, "text": " and so on, of the model, right, but run at different speeds. And by running at different"}, {"start": 2074.6400000000003, "end": 2081.5200000000004, "text": " speeds, and going through the model in different paths, I can get from the same model, multiple"}, {"start": 2081.5200000000004, "end": 2087.6800000000003, "text": " answers to my question, which is kind of what I believe what your brain does. So what happens"}, {"start": 2087.68, "end": 2093.3599999999997, "text": " there is, you have this network, but it's not like, you know, it's all firing like this layer"}, {"start": 2093.3599999999997, "end": 2099.7599999999998, "text": " after layer, it's rather, you have these asynchronous flows going through it, right,"}, {"start": 2099.7599999999998, "end": 2105.52, "text": " even going through matching paths, and CPUs are naturally built for this thing. Now, I'm not"}, {"start": 2105.52, "end": 2111.3599999999997, "text": " saying that somebody can't build a beautiful FPGA that will perhaps have a better, closer structure"}, {"start": 2111.36, "end": 2118.7200000000003, "text": " to what a brain does. Maybe so. But, you know, but there is an advantage to being commodity,"}, {"start": 2119.28, "end": 2125.04, "text": " okay? The fact that the CPU can do other things is a big win. If I can make, if I can move"}, {"start": 2125.04, "end": 2130.7200000000003, "text": " everything to software is really, is the thing, then I can really get all the advantages of modern"}, {"start": 2130.7200000000003, "end": 2137.76, "text": " software. So I'm not poo-pooing hardware accelerators, I'm saying, great, you know, they have a role,"}, {"start": 2137.76, "end": 2142.6400000000003, "text": " and so on and so forth, but they come at a price, right? And the price for any organization is that"}, {"start": 2142.6400000000003, "end": 2147.76, "text": " you, instead of just downloading or shipping your product with the machine learning piece, you have"}, {"start": 2147.76, "end": 2152.88, "text": " to ask the client to buy a certain accelerator, or run it with a certain accelerator. And this all"}, {"start": 2152.88, "end": 2159.92, "text": " goes away if we can figure out how to make the CPUs do what the GPUs do, right? Then we have, then"}, {"start": 2159.92, "end": 2165.84, "text": " we're back into this beautiful world of containerized, movable software, and that's really"}, {"start": 2165.84, "end": 2170.32, "text": " kind of where I would love machine learning to move to, rather, right? That we would have, and"}, {"start": 2170.32, "end": 2177.44, "text": " maybe down the road, right? There is this, you know, you know, CPUs have a history of absorbing"}, {"start": 2177.44, "end": 2184.08, "text": " the key components of any new paradigm that shows up. You know, virtualization started out with"}, {"start": 2184.08, "end": 2190.6400000000003, "text": " tricks on a GPU, on a CPU, and then later on added the features. Networking had special accelerators,"}, {"start": 2190.64, "end": 2196.3199999999997, "text": " and then they moved into the CPU. And I'm expecting that whatever features are necessary for machine"}, {"start": 2196.3199999999997, "end": 2202.96, "text": " learning to run well, will move into the CPU, and we won't need an outside accelerator to make this"}, {"start": 2202.96, "end": 2211.3599999999997, "text": " thing work. If you could, so I think that's, by the way, also the story of GPUs themselves, right?"}, {"start": 2211.3599999999997, "end": 2217.92, "text": " They were already kind of consumerish available, and then they can, they absorbed machine learning."}, {"start": 2217.92, "end": 2224.2400000000002, "text": " It's not necessarily the best architecture for machine learning. Let's say there's already all"}, {"start": 2224.2400000000002, "end": 2230.88, "text": " this hardware out there, right? There is very good CPUs next to very good GPUs. How do we get"}, {"start": 2231.36, "end": 2237.04, "text": " the best out of a machine like this? Right now we've advocated for, let's move things to the CPU,"}, {"start": 2237.04, "end": 2242.16, "text": " right? We have some advantages there. But what if I have a box with both? Like currently, I just use"}, {"start": 2242.16, "end": 2248.96, "text": " my CPU to ship data to the GPU, right? That's what my CPU does. But is there a way where I could"}, {"start": 2248.96, "end": 2256.64, "text": " potentially, you know, what kind of architecture would make the best use out of a combined system"}, {"start": 2256.64, "end": 2262.64, "text": " of CPUs and GPUs? No, I think this is really the vision that Nvidia has at least today for their"}, {"start": 2262.96, "end": 2268.8799999999997, "text": " Grace Hopper architecture. It's essentially, there will be a CPU and a GPU connected to one another,"}, {"start": 2268.88, "end": 2274.0, "text": " and the CPU will do all the things that are memory intense, and the GPU will do all the data intense"}, {"start": 2274.0, "end": 2279.04, "text": " things. The thing about the problem with this kind of a model is it's a beautiful model, by the way."}, {"start": 2279.04, "end": 2284.6400000000003, "text": " I'm not saying anything bad about this. If you really want to build a GPU world, that's a great"}, {"start": 2284.6400000000003, "end": 2294.08, "text": " thing to do. But again, how much you utilize your GPU, your attached GPU, has to do with how you"}, {"start": 2294.08, "end": 2300.08, "text": " write your application. Because you need to move the data into the GPU in and out, and that's slow."}, {"start": 2300.08, "end": 2308.96, "text": " Remember, it's exactly like going to memory, right? The GPU is not sitting in your caches."}, {"start": 2308.96, "end": 2314.56, "text": " So if you're on the CPU and you're computing something on a cache and suddenly you get a page"}, {"start": 2314.56, "end": 2319.68, "text": " fault and you have to go and get something from memory, that's the latency that the GPU introduces"}, {"start": 2319.68, "end": 2326.8799999999997, "text": " here, right? And so if you're going to design it with that, you have to create really good software"}, {"start": 2326.8799999999997, "end": 2333.12, "text": " to pipeline things. And this is at the level of the application. So the application programmer"}, {"start": 2333.12, "end": 2341.12, "text": " has a big programming task. And so this is a great solution for large scale, big projects where,"}, {"start": 2341.12, "end": 2347.6, "text": " okay, Facebook is going to get a thousand of these or 10,000 of these, whatever it is."}, {"start": 2347.6, "end": 2353.68, "text": " Or Google, 10,000, 100,000 of these and put them together. Then it's worthwhile to write this kind"}, {"start": 2353.68, "end": 2359.2799999999997, "text": " of complex software. But if you're Joe company, right, and you have your little thing, I don't"}, {"start": 2359.2799999999997, "end": 2368.7999999999997, "text": " think you want to be writing that interface, right? So I'm saying it's great for large things,"}, {"start": 2368.7999999999997, "end": 2374.3199999999997, "text": " right? Data center things, big things. But I'm very doubtful if this is going to be"}, {"start": 2374.32, "end": 2382.6400000000003, "text": " effective at the edge, if you can actually utilize the CPU for it. Okay. And I will say one more"}, {"start": 2382.6400000000003, "end": 2394.0, "text": " thing and that is that, you know, that the modern way that the designers of hardware think about it"}, {"start": 2394.0, "end": 2400.88, "text": " is that it's built in modules. If you look at the AMD latest architecture, right,"}, {"start": 2400.88, "end": 2408.96, "text": " essentially you have the CCXs. So the machine, even though it has, you know, maybe 40 or 50 or 60"}, {"start": 2408.96, "end": 2414.1600000000003, "text": " cores, right, they're grouped into groups of eight, right? And each group of eight like this is a"}, {"start": 2414.1600000000003, "end": 2419.52, "text": " little piece of the die. Okay. And I think Intel is shifting in that direction too. So nothing's"}, {"start": 2419.52, "end": 2425.28, "text": " to prevent you from making pieces of that die be specialized pieces of hardware, like a GPU."}, {"start": 2425.28, "end": 2430.8, "text": " You don't have to have outside device. So if you ask me what the future is going to look like,"}, {"start": 2430.8, "end": 2438.1600000000003, "text": " it's probably going to look like, you know, these large cores, right, that have or large machines"}, {"start": 2438.1600000000003, "end": 2444.32, "text": " with multiple dies and on these dies, we might have a GPU die, we might have accelerated. And"}, {"start": 2444.32, "end": 2450.7200000000003, "text": " that's more like what I expect to happen rather than having a massive, you know, accelerator on"}, {"start": 2450.72, "end": 2458.3199999999997, "text": " the side. If we, if we hear sparsity, and things not being in layers and so on, naturally, the"}, {"start": 2458.3199999999997, "end": 2463.4399999999996, "text": " topic of I think graph neural networks is very close to that, at least in the imagination of"}, {"start": 2463.4399999999996, "end": 2469.8399999999997, "text": " people, do you have anything to say about, you know, where current graph neural networks stand"}, {"start": 2469.8399999999997, "end": 2477.9199999999996, "text": " with respect to sparsity? Yeah, I would think of graph neural networks as a, as a, you know,"}, {"start": 2477.92, "end": 2485.52, "text": " graph neural networks as a, as a, as a different kind of, okay, so, so graph neural networks, I"}, {"start": 2487.04, "end": 2493.12, "text": " use some some graph neural networks in my research and the, and the idea there, you know, is that,"}, {"start": 2493.84, "end": 2499.52, "text": " you know, we can use graph neural networks to solve graph problems that otherwise would be"}, {"start": 2499.52, "end": 2505.84, "text": " very complicated to solve if we tried to solve them brute force. Okay, now, it's not generally"}, {"start": 2505.84, "end": 2515.84, "text": " true. There are quite a few limitations. But, but as a tool, I would say that, you know, rather than"}, {"start": 2515.84, "end": 2520.88, "text": " think about the neural network itself as being looking like a graph neural network, right,"}, {"start": 2520.88, "end": 2529.04, "text": " I could use graph neural networks, right, to define what we call motifs in the neural network."}, {"start": 2529.04, "end": 2535.6000000000004, "text": " So for example, when we try to look at, at how brain structure, brains are structured, right,"}, {"start": 2535.6, "end": 2541.6, "text": " graphs of brains, and we try to understand, you know, is there a motif that is repeating itself"}, {"start": 2541.6, "end": 2547.36, "text": " in this graph, right, then using a graph neural network for that is a really nice way to try to"}, {"start": 2547.36, "end": 2554.56, "text": " find these motifs, okay, efficiently, right, because the problem itself is, is PSPACE complete,"}, {"start": 2554.56, "end": 2561.6, "text": " or we don't know, it's graph isomorphism. So, so clearly, we don't know, right, how to do the"}, {"start": 2561.6, "end": 2567.6, "text": " brute force algorithm well, but, but the graph neural network can come to our aid here. And so,"}, {"start": 2567.6, "end": 2575.7599999999998, "text": " so I would say that right now, I don't really see a real network design, neural network design,"}, {"start": 2575.7599999999998, "end": 2581.7599999999998, "text": " that is specific to that or a way that it helps, but, but in research, it definitely helps. And we"}, {"start": 2581.7599999999998, "end": 2585.2, "text": " really want to use these networks to help us in research."}, {"start": 2585.2, "end": 2593.2, "text": " This might be a bit of a tech bro question. But if I hear, you know, I can do sparse computation,"}, {"start": 2593.2, "end": 2601.2, "text": " very, I can reduce the flops and so on. Is there any intrinsic connection between the"}, {"start": 2601.2, "end": 2608.0, "text": " sparsification of neural networks, the non layer wise computation, and blockchain technology and"}, {"start": 2608.0, "end": 2611.7599999999998, "text": " smart contracts and distributed computing and things like this?"}, {"start": 2611.76, "end": 2617.1200000000003, "text": " Have you ever given this any thought or, or, yeah, is that completely off?"}, {"start": 2618.32, "end": 2625.2000000000003, "text": " Yeah, look, I think nothing is completely off with respect to machine that in the sense that I am"}, {"start": 2625.2000000000003, "end": 2632.88, "text": " sure that machine learning will find its way into into all of those areas. Right, it's a matter of"}, {"start": 2632.88, "end": 2643.44, "text": " time. And, and right now, right, the all the work there doesn't need the efficiency of right, of what"}, {"start": 2643.44, "end": 2648.2400000000002, "text": " machine learning offers, because machine learning in the end is an optimization technique. And so"}, {"start": 2648.2400000000002, "end": 2655.28, "text": " when I think when all these blockchain algorithms and all, you know, become more commonplace, and we"}, {"start": 2655.28, "end": 2660.7200000000003, "text": " need to provide them with things like security, further security or analysis, and so on, and so"}, {"start": 2660.72, "end": 2666.3199999999997, "text": " on, I think then we're going to see applications of machine learning there. And with that, I think,"}, {"start": 2666.7999999999997, "end": 2674.3199999999997, "text": " all these things of sparsity and so on are going to appear. But, you know, but, but for me, right,"}, {"start": 2674.8799999999997, "end": 2683.7599999999998, "text": " it really is the whole story of sparsity, right, is the story of a phenomenon that is very prevalent"}, {"start": 2683.76, "end": 2691.6000000000004, "text": " in nature, right, that may you can say surprisingly or not surprisingly shows up in machine learning."}, {"start": 2691.6000000000004, "end": 2698.8, "text": " And it kind of it makes me feel like it's strengthening my belief, right, that even though"}, {"start": 2698.8, "end": 2703.44, "text": " the exact computations that we're doing are not the same as spiking neural networks in brains,"}, {"start": 2703.44, "end": 2709.6000000000004, "text": " right, that there is a lot of commonality there. And the emergence of these similar phenomena,"}, {"start": 2709.6, "end": 2714.96, "text": " like sparsity, like, you know, pruning and so on, and the fact that we can get benefits from it,"}, {"start": 2714.96, "end": 2721.68, "text": " this tells me, oh, okay, these are related. I think that's a very important point to keep in mind."}, {"start": 2722.24, "end": 2729.2799999999997, "text": " With neural magic, who is your main target audience? Like who who is listening to this?"}, {"start": 2729.2799999999997, "end": 2733.44, "text": " Do you want to let know like, we are exactly for you."}, {"start": 2733.44, "end": 2740.4, "text": " So we span the gamut from the data center to the edge. I would like to say, I mean,"}, {"start": 2740.4, "end": 2747.52, "text": " we just now are moving into providing the same properties for ARM architectures. And so I would"}, {"start": 2747.52, "end": 2753.68, "text": " say the exciting new thing in neural magic is we're moving from doing this, you know, for AMD"}, {"start": 2753.68, "end": 2758.48, "text": " and Intel architectures to doing it for ARM, which means that we're going to span the gamut all the"}, {"start": 2758.48, "end": 2764.16, "text": " way to the very bottom of the of the food chain, if you will. And I think this is very exciting."}, {"start": 2764.16, "end": 2770.2400000000002, "text": " Because as you know, because because sparsity has a dual role as you go down the food chain,"}, {"start": 2770.2400000000002, "end": 2774.8, "text": " right, because for the large accelerating, you know, the fact that the memory footprint is large"}, {"start": 2774.8, "end": 2780.08, "text": " as small is not that important. But as I go down, sparsity gives me two things speed with neural"}, {"start": 2780.08, "end": 2784.96, "text": " magic gives you speed, but it also makes the model extremely small. So you're getting a small,"}, {"start": 2784.96, "end": 2790.88, "text": " accurate model, right running on a very small device. And this, you know, typically is an ARM"}, {"start": 2790.88, "end": 2796.32, "text": " device. And so that's, that's, that's the audience that I'd like to say, hey, we're coming, you know,"}, {"start": 2796.32, "end": 2800.4, "text": " we're coming in, we're going to deliver the same things that we can deliver for Intel and AMD."}, {"start": 2800.4, "end": 2807.76, "text": " We're now going to deliver it for ARM at the very end. If you say edge, do you mean smartphones?"}, {"start": 2807.76, "end": 2811.6, "text": " Do you mean security cameras? Do you mean robots? Everything? Okay,"}, {"start": 2811.6, "end": 2817.2, "text": " I mean everything I not like I'm going to do everything to start with. But but yes, yes,"}, {"start": 2817.2, "end": 2823.12, "text": " we're aiming in that direction. Yes. And with the danger that this has become going to become like"}, {"start": 2823.12, "end": 2828.24, "text": " a marketing opportunity question, but how easy is it to get started with what you're doing?"}, {"start": 2829.12, "end": 2834.56, "text": " Like, let's say I'm a I'm like, I've done, you know, my TensorFlow tutorials, I know how to build"}, {"start": 2834.56, "end": 2840.16, "text": " a model and train it and so on. Like, how much does it take for me to do that? And then you know,"}, {"start": 2840.16, "end": 2844.56, "text": " how much does it take for me to transition? Or to apply what you're doing?"}, {"start": 2845.2799999999997, "end": 2850.7999999999997, "text": " Yeah, so you just go to our website, go to get go to get download, deep sparse, or, you know,"}, {"start": 2850.7999999999997, "end": 2858.72, "text": " our engine download our ML tooling. And, you know, immediately, you just either pick a sparse model"}, {"start": 2858.72, "end": 2863.3599999999997, "text": " and transfer learn on to it with our tool. So we have recipes, you have a model, you have a recipe,"}, {"start": 2863.3599999999997, "end": 2866.8799999999997, "text": " exactly what you would do if you went to Hugging Face and downloaded a model and"}, {"start": 2866.88, "end": 2872.0, "text": " downloaded a recipe, you do the same kind of thing. And you sparse transfer learn onto it,"}, {"start": 2872.0, "end": 2877.76, "text": " and you're in business. So it's not very hard. So I think this is really and we're working on making"}, {"start": 2877.76, "end": 2882.56, "text": " it even even easier. This is one of our goals, right is to make it really, really easy to do"}, {"start": 2882.56, "end": 2889.6800000000003, "text": " this. And the advantage, of course, is that, you know, people are already busy, you know, quantizing"}, {"start": 2889.6800000000003, "end": 2894.8, "text": " their models to get more performance. So this is like quantize, in some sense, right, you're going"}, {"start": 2894.8, "end": 2901.6800000000003, "text": " to do the same kind of thing and get a lot more performance. Is there a type of model where it"}, {"start": 2901.6800000000003, "end": 2905.92, "text": " works particularly well and the type of model where it doesn't like I'm thinking, you know,"}, {"start": 2905.92, "end": 2911.36, "text": " convnets, recursive networks, autoregressive, maybe, you know, the big language models,"}, {"start": 2911.36, "end": 2920.0800000000004, "text": " like what is it best at? Yeah, so right now, you know, it's best at at BERT, YOLO models, we do,"}, {"start": 2920.08, "end": 2925.84, "text": " we do computer vision, and we do, and we do the language models, but not the large language models,"}, {"start": 2925.84, "end": 2931.2, "text": " we haven't done the large language models yet. So for those types of things, like the BERTs and the"}, {"start": 2931.2, "end": 2937.12, "text": " YOLOs and the, you know, the, whatever the variants of efficient nets and all these guys, this is,"}, {"start": 2937.12, "end": 2943.7599999999998, "text": " you know, visual transformers, these are the things that that we do right now. And every all"}, {"start": 2943.76, "end": 2950.6400000000003, "text": " our technology is right now, you know, available for those. I'd love to do the large models,"}, {"start": 2950.6400000000003, "end": 2956.48, "text": " a CPU is a natural environment for running the knowledge models. You know, these giant models,"}, {"start": 2956.48, "end": 2962.1600000000003, "text": " these trillion or whatever parameter models that people talk about splitting across 16 GPUs,"}, {"start": 2962.1600000000003, "end": 2969.0400000000004, "text": " they fit on your desktop. Okay, so clearly a CPU is a natural place to run a very large model."}, {"start": 2969.04, "end": 2977.68, "text": " Okay. And so that's that will be a target but not right now. Okay, very exciting. Is there any"}, {"start": 2977.68, "end": 2983.2799999999997, "text": " last things you want to get out maybe about neural magic or sparsity in general? Well, you know,"}, {"start": 2984.48, "end": 2989.84, "text": " our whole machine learning software stack is open source. And we'd love people to comment and help"}, {"start": 2989.84, "end": 2996.24, "text": " us build, you know, better sparsity, use sparsity in their models and, and tell us about what they're"}, {"start": 2996.24, "end": 3000.7999999999997, "text": " doing. And, you know, that it would we have a community and we'd love you to join us."}, {"start": 3002.7999999999997, "end": 3007.04, "text": " Excellent. Nir, thank you so much for being here today. So it was very pleasant."}, {"start": 3007.04, "end": 3025.2, "text": " Thank you very much. Bye bye. Bye bye."}]