Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search (Paper Explained)
Neural Architecture Search is usually prohibitively expensive in both time and resources to be useful. A search strategy has to keep evaluating new models, training them to convergence in an inner loop to find out if they are any good. This paper proposes to abstract the problem and extract the essential part of the architecture to be optimized into a smaller version and evaluates that version on specifically custom learned data points to predict its performance, which is much faster and cheaper than running the full model.

OUTLINE:
0:00 - Intro & High-Level Overview
1:00 - Neural Architecture Search
4:30 - Predicting performance via architecture encoding
7:50 - Synthetic Petri Dish
12:50 - Motivating MNIST example
18:15 - Entire Algorithm
23:00 - Producing the synthetic data
26:00 - Combination with architecture search
27:30 - PTB RNN-Cell Experiment
29:20 - Comments & Conclusion

Paper: https://arxiv.org/abs/2005.13092
Code: https://github.com/uber-research/Synthetic-Petri-Dish

Abstract:
Neural Architecture Search (NAS) explores a large space of architectural motifs -- a compute-intensive process that often involves ground-truth evaluation of each motif by instantiating it within a large network, and training and evaluating the network with thousands of domain-specific data samples. Inspired by how biological motifs such as cells are sometimes extracted from their natural environment and studied in an artificial Petri dish setting, this paper proposes the Synthetic Petri Dish model for evaluating architectural motifs. In the Synthetic Petri Dish, architectural motifs are instantiated in very small networks and evaluated using very few learned synthetic data samples (to effectively approximate performance in the full problem). The relative performance of motifs in the Synthetic Petri Dish can substitute for their ground-truth performance, thus accelerating the most expensive step of NAS. Unlike other neural network-based prediction models that parse the structure of the motif to estimate its performance, the Synthetic Petri Dish predicts motif performance by training the actual motif in an artificial setting, thus deriving predictions from its true intrinsic properties. Experiments in this paper demonstrate that the Synthetic Petri Dish can therefore predict the performance of new motifs with significantly higher accuracy, especially when insufficient ground truth data is available. Our hope is that this work can inspire a new research direction in studying the performance of extracted components of models in an alternative controlled setting.

Authors: Aditya Rawal, Joel Lehman, Felipe Petroski Such, Jeff Clune, Kenneth O. Stanley

Links:
YouTube: https://www.youtube.com/c/yannickilcher
Twitter: https://twitter.com/ykilcher
BitChute: https://www.bitchute.com/channel/yannic-kilcher
Minds: https://www.minds.com/ykilcher
 Hi there, today we're looking at Synthetic Petri Dish, a novel surrogate model for rapid architecture search by Adi Tarawal, Joel Leeman, Felipe Petroski-Suc, Jeff Klune and Kenneth O. Stanley. This paper on a high level, it basically says if you want to do neural architecture search, if you for example search for a better non-linearity, you should be able to extract that non-linearity instantiated in a very small network and then evaluate that very small network in order to predict the performance of a large network and therefore you can find a better non-linearity in much less time. Now the exact procedure how you do this in the small network is the topic of this paper. As always if you like content like this I encourage you to subscribe if you are not already and to share out the video so other people can experience the joy themselves. All right let's dive in. So they say in the abstract neural architecture search explores a large space of architectural motives which basically so it basically means you want to find a neural architecture let's say you have a multi-layer perceptron right here a couple of layers okay and they're all connected by you know feet forward weights whatnot and each of these weights basically is a multiplication so each one of these is a multiplication of x by your weight w and then there is a non-linearity so the non-linearity could be a sigmoid so the sigmoid would be something like 1 over 1 plus e to the negative x now there's a bit of an extension in a sigmoid where you can do a sigmoid that has like a temperature parameter attached or a slope parameter where you go cx so in one case you can set c such that the sigmoid has a shape like this and in then if you put c to a different value you can make this slope you can make it like a shape well this is terrible like this you know what I mean okay so this this C right here can potentially change the behavior of your network and you want to find a good parameter C and this is a hyper parameter now there are many hyper parameters like this for example how many units you have in a particular layer in a CNN it could be your filter size in a transformer could be the number of heads and so on it could actually be not only the slope of the non-linearity but the actual non-linearity itself or famously in recurrent neural networks you have these recurrent cells and they're like okay we have an input signal and a carry signal and then the in the input here is like dot multiplied here and then there is like a gate with a non-linearity and then it's kind of like multiplied by the carry but then there is also a like a forget gate and whatnot there's a minus right here it's very complicated and so people do architecture search over these kind of problems to find better architectures for particular problems now the problem is of course that how do you know if if a given architecture is good what you have to do is you'll have to go take that cell that you have dreamed up you think well I think that's a good cell and you have to train it on the full training data set this is a data set a database right this is a full training data set then you need to evaluate it on your validation data set and then you have like a number you have like okay this is eight good and then you go back and you say okay what if I change this cell here what if I change it to a plus instead of a minus and you do the entire thing again train it for I don't know how much validate it and then this is like a nine and you can say oh cool that's a nine so this is a very basic architecture search and there has been a lot of development in this space so like evolutionary search and so on but they most of the time they require pretty much evaluating the entire thing on the full data so you get a good you get a good estimate of what your final performance back here is going to be now people have come up with methods to counter that and they say well if we can sort of encode the cell structure let's go with the let's go with the RNN cell if we could encode the cell structure in in a sort of a continuous way so you know we can encode text in a continuous way let we could also encode a cell structure because the cell structure I can write it down as an equation I can say like okay it's the forget gate of the carry times the sigmoid output of X plus the so this is the plus here and plus the sigmoid output of X multiplied by the input let's call that I something like this right this is text I can like write it down and then I can encode that into a vector much I can for example build another RNN ironically or something to to encode that or I can represent it as a computation graph like it is here and use a graph neural network to encode that into a single vector and then I have sort of an embedding space where each cell that I could build is a point in that embedding space and then I can evaluate a couple of them I can for example say okay this one here this one here this one here this one here I'm going to train them these cells I'm going to do the full training eval and so on get their scores and then I can learn basically in this latent space I can learn a predictor I can say okay here I get I got an eight I got a nine I got a two and I got a four so it appears to be that in this direction that the good cells are in this direction and then I can do it again I can sample or I can do gradient descent in this space since this is now a continuous space and the gradient descent on the model that gives me this space it's right so this this method basically tries to take in the building plan of a cell and learn to predict the performance just by looking at it if if you're thinking of the touring machine right now then you like I I immediately thought of like this this halting problem because it appears to be exactly what it is so you're trying to build a machine that takes the building plan of another machine and tries to predict its performance now in a general sense we can already state that this problem is sort of the difficulty of this problem is equivalent to the difficulty of the original problem so I'm not sure but it appears to you know it appears to work if you throw lots of compute at it but of course that's a problem you need lots of compute right so either your your option one is to run all of these things and kind of iterate them in a neural sorry in an evolutionary way or your second option is to take the building plan and predict the performance from that both are not satisfactory and both use lots of compute now neural petri dish is a or synthetic petri dish is a way to combine the two together it says can we take the building plan here but actually run on the data on data to predict the performance so what they're saying is basically if I have this cell right here and usually this cell you know it deals with vectors of let's say size 512 and so on it will say it since this is let me draw it again up here so here you have the cell and here you have somehow the connections in there when you carry and the input and the input here okay and this is the output or the carry I have 512 embedding like size of this so this is a giant cell there's 512 the vector has 512 dimensions going in basically can't I take the exact same thing but and keep the connection pattern so I would keep the entire pattern of connection right here but I only do it for one or two so this is 512 and this is just two right just the I just reduce the dimensionality but I sort of keep the connection pattern alive if I only do that I have like a very small network right now and the same goes for if this is so a lot of times these RN ends they have multiple layers of these things so they have another exactly equal box up here and then another up here I can just reduce this to one layer and out of the regularity of these neural network things it is known that or one can make the assumption that the performance on this thing will sort of kind of be correlated to the performance of the entire thing and that's one of the things that this Petri dish paper does so we try to take out what we are trying to search over namely the connection pattern we keep that as it is up here but we reduce everything else we reduce the dimensionality we reduce the number of layers and so on now they don't actually reduce the number of layers here but you can reduce the number of units and so on okay so this in essence this works whenever you can do that whenever you can keep the structure you're searching over but can reduce the rest so that's one precondition doesn't work for everything the second part here is that you don't want to use this particular training data and then this particular validation data because first of all it's a lot of training data and second of all it won't give you that good of a prediction instead of what you're trying to do and this is the second part of the idea of Petri dish is you're trying to abstract the training data to get you a very small data set such that and the validation data as well as well such that if you train on this data and evaluate on this data the performance that you get will be very predictive of the performance had you trained the big model on the big data set okay in fact in this Petri dish paper these little data sets they have nothing to do with the with the original train and validation data and that's I think that's one of the cool things here these things this training data and this validation data they are optimized as well by the procedure they're optimized special data points that are trained these are trained parameters such that if you train on the training on the small training data and evaluate on the small eval data the you will be able to predict this performance back here with high accuracy okay and this I think is where previous approaches have or might have failed because it's you know the the idea of scaling down your network in order to do the architecture search is probably has it has appeared to many people before that's not you know that's not really genius idea but probably they have all found that now we can't really do it it doesn't really give us accurate enough numbers but in this case the addition of adding these synthetic data sets that are much smaller but can still if you train and evaluate on them can still predict with high accuracy the full score of the full model that I think makes this idea work all right so I guess we're already through the idea and problem setting and everything without actually reading the paper so they give this example right here at the beginning that if you have a a two layer 100 wide so 100 dimensional MNIST networks it's two layers it's I think it's an MLP two layer MLP with a non linearity that is this the sigmoid right here this okay now you can see it has this temperature parameter here it has this slope parameter and you you want to do neural architecture search to find the best slope parameter now usually you would just do a grid search but this is an this is an example because this can be of course much higher dimensional things and then you don't want to do grid search anymore okay so what do we do if you look at how the 100 white MNIST network so we can draw it right here so this is a 100 dimensional MNIST network so this is 100 and each cell each connection here first has a weight and then has the sigmoid non-linearity and the sigmoid non-linearity is parameterized by the parameter C okay and you have you have many of them right you have one here and so on and each one has a different C and each of these networks represents one blue dot here so if you let C vary so this sigmoid slope value right here that's your parameter C if you let this vary and train the big network on the entire data set to convergence and then you eval on the validation data set you get the slope like the blue curve so if you see the blue curve the blue curve is if you start over here if you reduce this slope you'll gain in performance but if you reduce it too much you drop drastically okay until it's if it's if it's zero it's basically you know the the X is not the signal doesn't propagate anymore and you you have no learning occurring okay so that's the original performance now what if I only give you training data in this range right here I only showed you this particular range I can can't actually zoom in that much but if if I give you this and I ask you to build one of these please take the architecture and predict the performance that we saw at the beginning like one of these good little machines or Turing Turing machines you would it's basically say well that looks to me like a line so I'm gonna predict the red thing here and even if you can you know evaluate a bunch of these it just looks like a line and you're going to predict it's probably a slope like this right this happens almost independently of which model you choose to predict right here the the data of training is simply doesn't give away that the fact that there is a there is this break down here which happens in the real world so if you just give this as training data there is no way so so the the criticism about these models is valid that they will only work where you give them training data they can at best interpolate their training data but they can't really extrapolate now here since the synthetic Petri dish method which is the green thing here uses the actual not the actual non-linearity that this thing characterizes so it it instantiates the sigmoid with the parameter C that you give it just not on the large network but on a small network in fact their network is just one unit sorry one unit and then another unit so it's just a two hidden layer but just with one unit instead of 100 and of course you can't feed in MNIST right here right but we said they don't feed in the data they actually feed in their synthetic data that they learn so you give them the points here and they learn the synthetic data to to evaluate to evaluate the others and then once you ask them well if if my C is right here what's the performance going to be it's going to instantiate that in its small network it is going to use the training data that it has learned from this region right here in order to train this and then it's going to evaluate this on the synthetic validation data that is also learned on the training data and it is going to come up with a performance metric it says okay this is how good it's going to be and since it is an approximation in its building plan to the entire network it will react similarly so it will get that there is this performance dip right here okay so it you can see how this sort of makes sense you are actually running an approximation to the actual program instead of just looking at the plan of the program and trying to predict it which you know halting problem says hello okay so that is the motivating example of their MNIST thing and here is the entire algorithm all right so you take MNIST training and validation data and you instantiate a bunch of really big networks this is ground truth okay you you need this you need this to learn from you instantiate a bunch of really big networks now if I draw the graph from before right we had this was the performance of the actual networks you want you this comes from here from this region right here this is the training data okay so you instantiate a bunch of these networks each one you instantiate in one of them right each one gives rise to a different non-linearity and you do the full training ground truth training and evaluation on the full training set and the full validation set and you get validation losses right for each of these and these are the points right here now you that's the training data for your neural for your neural architecture search so for your Petri dish method what the Petri dish does is it says it extracts the motive and the motive is the thing that you optimize over so as I said you want to keep that thing in its essence but you want to reduce everything else so it reduces it instead of from a two layer on hundred YML MLP it reduces that to a two layer single neuron wide MLP okay and it now this over here is the training data for the procedure that we're going to do now so what it would take is it would take it would take one of these values it would instantiate and we have that here would instantiate the neural network in the small form of that and now we know that if I train the full data and evaluate if I train on the full training data and evaluate on the full validation data I should get this accuracy all right so I will create and we're going to look at in a second I will create training and validation data such that if I train on this training data and then validate on this validation data I get the same validation loss as if I had trained the big network with the same you know the same C parameter on the full training data and evaluate on the full validation data okay so in this step I'm optimizing the data here the training and validation data all right and now in the second step once I have this training and validation data such like that I can basically reproduce this this graph right here then I can go and actually ask my model okay now please tell me what happens over here so what am I going to do I'm gonna take that I'm gonna instantiate it I'm going to use my training data that I learned to train it I'm going to use my validation data that I learned to evaluate it and it's going to give me a number and that number is going to be like close to hopefully close to do this so this is how we can extrapolate using that method okay now there are a number of assumptions right here and you can imagine this doesn't work in any situation this works if if you you know if you basically you have to get lucky in that you have to abstract the correct things right I said you need to reduce everything else so they reduce notably you see they reduce the 100 the 100 layer width to a single neuron wide MLP and they sort of guess that that doesn't change the fundamental thing but you can also see they leave the two layer right they leave the two the two layer neural network and I'm can almost guarantee you that they tried this reducing this to a one layer neural network and it did not work and so you have to be sort of very careful of what quantities you abstract and what quantities you don't because okay now you might always think oh I can reduce the you know number of dimensions or channels that's also not always the case so I think that's kind of the crux of the method you have to actually engineer this down compressing of the architecture such that it's its properties are still kept and yeah but yeah in other things how do you how do you actually produce training and validation data to match these and there are a number of ways but what comes to mind is is meta learning right so because what you're doing they initialize the training and validation data at random points so these are just random at the beginning and then they optimize the data itself using gradient descent okay now I see synthetic training data and they are randomly initialized okay and they use gradient descent they have it somewhere yes so they have this inner training loop okay which is many steps of inner training and then they have the outer loss which is the it's the validation loss after the inner training loop and the difference for that to the true validation loss and then they do gradient descent on this outer loss now this outer loss is a result of the inner loss and the inner loss is a result of the inner training procedure and the inner training procedure is n steps of feeding in the training data every step you feed in the training data so your computational graph is going to look like so here's your training data as train and here are your initial parameters you at random lies initialize them randomly in the first step use the training data to produce theta one then in the second step you use your training parameter your training data again to produce theta two and then you use it again to use theta three and so on each time you feed the training data in order to evolve your parameters to give you a better prediction right so the gradient since somewhere back here there's a loss the gradient here will have to flow back through all of these paths and through all of these connections to the training data this is kind of you back propagate through an optimization procedure and we have this a bunch of times here and I've looked at the code and the code is like really crazy and it looks like proper research code but it appears to be that that's actually what's happening they back prop through the optimization procedure to find this synthetic training and validation data now that's I mean that's crazy but it also kind of limits how far you can go with this because usually you can't back prop for more than a couple of steps doing this now that the model the fact that the model inner model is small helps but also this introduces very very much like these things are very brittle if you back prop through an optimization procedure like this these things tend to be very brittle and so I think there's another thing there where you have to pay careful attention all right that's it's basically it the last thing they say is that they can combine this with architecture search in that so not only can you predict good architectures what you can do is you can actually predict the which architectures are good and then you can use that prediction to get new to basically input this into your neural architecture search to inform it so instead of the neural architecture search having to evaluate all of the candidates that it produces it only has to now evaluate the very small subset of candidates that the synthetic petri dish training deems most worthy of being evaluated in this case here instead of evaluating all of the things here it would limit itself to whatever the synthetic petri dish says are the highest performing ones because if the synthetic petri dishes and is any good then it will you know give accurate predictions of how they're performing and then that can go in multiple rounds so the architecture search can find new come up with new things that it thinks are better through like an evolutionary mutation algorithm the petri dish can evaluate them in the synthetic way and then suggest the like ten candidates to evaluate on the full test set and you see that way you don't have to evaluate all the like thousand candidates all right all right cool they do this for this MNIST and they also do it for finding a RNN cell for the pen treebank this is a language modeling task and the this is a benchmark for neural architecture search where you're trying to find a good RNN cell to get the perplexity really low and here you can see if they give the same amount of data to all the methods then the benchmark neural architecture search is worse than the synthetic petri dish informed architecture search now one has to say on the full date I believe the NAO gets to about here but of course if you give all of them the same data the neural the petri dish beats this method and I think still this method here uses way more compute because it always has to evaluate all the candidates and that's exactly one of these where I learn an architecture to predict the other architecture by just looking at it so it works but it doesn't work as well as actually running the architecture in an abstract fashion this also shows you the importance of selecting your experimental evaluation in a smart way like they argue they argue for very long why it makes sense to evaluate everything on reduced data such that their method here can be better and they don't have to compare to the full thing it's easier for them to work on reduced data and they argue you know it's it's it's what people usually do in practice and that's the the task they focus on so you know good good good good paper writing right here yeah that's basically it to the paper there's a lot of things to be said here I think this works in very very limited settings it seems to me that it's sort of brittle with respect to how you abstract and also it it's always the case like how many how large is this synthetic training data in their case there they like abstract this to 20 or 30 data points or something like this so it seems to me since you're optimizing this training data with gradient descent what you would mainly find are adversarial sort of adversarial examples to this architecture here so I'm going to guess that the inner optimization is very noisy and that's because if you really let your optimizer run then it will abuse every single thing it can to match that validation loss and that will usually lead to an adversarial example since you're optimizing the data itself okay so I think this suffers from that and this is we had this in the in the planning you know the planning in in learned world models and reinforcement learning where if you have a really really good planner it will just abuse the mistakes that you make in approximating the true world and the same here you're going to make mistakes approximating this architecture here and the better your your optimizer is for producing this synthetic data the probably the worse the worse the result is going to match the worst that these losses are going to actually match now okay these losses will match because they're that's what you train for but the worse these two curves will match each other because now you're just finding adversarial examples for your particular training data another concern I have here is with respect to the double descent phenomenon so if you know the double descent phenomenon if here you have your number of parameters and here you have your validation loss let's say and you know that if I add parameters I can make my validation loss go down so this is assuming I have a model with p parameters and I always train it on the train data to like to convergence if I add parameters I can generalize better until a point where I add too many parameters and I start overfitting and my validation loss goes up again but the double descent phenomenon and I think I've done a video on this shows that after a certain threshold you get the interpolation threshold the validation loss goes actually down again it goes down even further here now I'm so this is a very strange phenomenon by itself but I'm sort of concerned that if you do this abstraction that this paper proposes so you read your let's say your full model is here with a large number of parameters so it is past this interpolation threshold if you now seriously reduce the number of parameters because you want to go into this petri dish you will get maybe you will cross this interpolation threshold and actually be on this side of the curve right here now of course at the same time you reduce the amount of data which would push you over here again but it is different data so I'm not sure how all of this is going to play out it appears to work in these settings right here but I I think this is it's it's sort of it's sort of applicable in some situations and it's it'd be very cool if we develop this further such that we understand when it applies and when we can use it because I feel this can be a very cool thing if we understand it better and if we can apply it throughout all right that's the end if you like this paper leave a comment if you didn't like it leave a comment and bye bye see you next time
[{"start": 0.0, "end": 4.96, "text": " Hi there, today we're looking at Synthetic Petri Dish, a novel surrogate model for"}, {"start": 4.96, "end": 11.120000000000001, "text": " rapid architecture search by Adi Tarawal, Joel Leeman, Felipe Petroski-Suc, Jeff"}, {"start": 11.120000000000001, "end": 18.080000000000002, "text": " Klune and Kenneth O. Stanley. This paper on a high level, it basically says if you"}, {"start": 18.080000000000002, "end": 22.56, "text": " want to do neural architecture search, if you for example search for a better"}, {"start": 22.56, "end": 29.04, "text": " non-linearity, you should be able to extract that non-linearity instantiated"}, {"start": 29.04, "end": 33.76, "text": " in a very small network and then evaluate that very small network in"}, {"start": 33.76, "end": 38.24, "text": " order to predict the performance of a large network and therefore you can find"}, {"start": 38.24, "end": 44.96, "text": " a better non-linearity in much less time. Now the exact procedure how you do this"}, {"start": 44.96, "end": 50.32, "text": " in the small network is the topic of this paper. As always if you like content"}, {"start": 50.32, "end": 55.28, "text": " like this I encourage you to subscribe if you are not already and to share out"}, {"start": 55.28, "end": 61.44, "text": " the video so other people can experience the joy themselves. All right let's"}, {"start": 61.44, "end": 67.68, "text": " dive in. So they say in the abstract neural architecture search explores a"}, {"start": 67.68, "end": 74.08, "text": " large space of architectural motives which basically so it basically means"}, {"start": 74.08, "end": 78.64, "text": " you want to find a neural architecture let's say you have a multi-layer"}, {"start": 78.64, "end": 86.24, "text": " perceptron right here a couple of layers okay and they're all connected by you"}, {"start": 86.24, "end": 90.96000000000001, "text": " know feet forward weights whatnot and each of these weights basically is a"}, {"start": 90.96000000000001, "end": 96.96000000000001, "text": " multiplication so each one of these is a multiplication of x by your weight w and"}, {"start": 96.96000000000001, "end": 103.56, "text": " then there is a non-linearity so the non-linearity could be a sigmoid so the"}, {"start": 103.56, "end": 110.16, "text": " sigmoid would be something like 1 over 1 plus e to the negative x now there's a"}, {"start": 110.16, "end": 114.24000000000001, "text": " bit of an extension in a sigmoid where you can do a sigmoid that has like a"}, {"start": 114.24000000000001, "end": 119.80000000000001, "text": " temperature parameter attached or a slope parameter where you go cx so in"}, {"start": 119.80000000000001, "end": 126.36, "text": " one case you can set c such that the sigmoid has a shape like this and in"}, {"start": 126.36, "end": 131.22, "text": " then if you put c to a different value you can make this slope you can make it"}, {"start": 131.22, "end": 137.28, "text": " like a shape well this is terrible like this you know what I mean okay so this"}, {"start": 137.28, "end": 143.44, "text": " this C right here can potentially change the behavior of your network and you"}, {"start": 143.44, "end": 147.48, "text": " want to find a good parameter C and this is a hyper parameter now there are many"}, {"start": 147.48, "end": 152.92, "text": " hyper parameters like this for example how many units you have in a particular"}, {"start": 152.92, "end": 157.98, "text": " layer in a CNN it could be your filter size in a transformer could be the"}, {"start": 157.98, "end": 163.35999999999999, "text": " number of heads and so on it could actually be not only the slope of the"}, {"start": 163.35999999999999, "end": 167.83999999999997, "text": " non-linearity but the actual non-linearity itself or famously in"}, {"start": 167.83999999999997, "end": 173.07999999999998, "text": " recurrent neural networks you have these recurrent cells and they're like okay we"}, {"start": 173.07999999999998, "end": 177.32, "text": " have an input signal and a carry signal and then the in the input here is like"}, {"start": 177.32, "end": 182.35999999999999, "text": " dot multiplied here and then there is like a gate with a non-linearity and"}, {"start": 182.35999999999999, "end": 186.56, "text": " then it's kind of like multiplied by the carry but then there is also a like a"}, {"start": 186.56, "end": 192.52, "text": " forget gate and whatnot there's a minus right here it's very complicated and so"}, {"start": 192.52, "end": 196.72, "text": " people do architecture search over these kind of problems to find better"}, {"start": 196.72, "end": 203.28, "text": " architectures for particular problems now the problem is of course that how do"}, {"start": 203.28, "end": 208.6, "text": " you know if if a given architecture is good what you have to do is you'll have"}, {"start": 208.6, "end": 213.92000000000002, "text": " to go take that cell that you have dreamed up you think well I think that's"}, {"start": 213.92, "end": 218.51999999999998, "text": " a good cell and you have to train it on the full training data set this is a"}, {"start": 218.51999999999998, "end": 223.76, "text": " data set a database right this is a full training data set then you need to"}, {"start": 223.76, "end": 228.92, "text": " evaluate it on your validation data set and then you have like a number you have"}, {"start": 228.92, "end": 234.07999999999998, "text": " like okay this is eight good and then you go back and you say okay what if I"}, {"start": 234.07999999999998, "end": 239.32, "text": " change this cell here what if I change it to a plus instead of a minus and you"}, {"start": 239.32, "end": 244.92, "text": " do the entire thing again train it for I don't know how much validate it and then"}, {"start": 244.92, "end": 249.51999999999998, "text": " this is like a nine and you can say oh cool that's a nine so this is a very"}, {"start": 249.51999999999998, "end": 255.64, "text": " basic architecture search and there has been a lot of development in this space"}, {"start": 255.64, "end": 260.36, "text": " so like evolutionary search and so on but they most of the time they require"}, {"start": 260.36, "end": 266.03999999999996, "text": " pretty much evaluating the entire thing on the full data so you get a good you"}, {"start": 266.04, "end": 271.20000000000005, "text": " get a good estimate of what your final performance back here is going to be now"}, {"start": 271.20000000000005, "end": 276.0, "text": " people have come up with methods to counter that and they say well if we can"}, {"start": 276.0, "end": 282.16, "text": " sort of encode the cell structure let's go with the let's go with the RNN cell"}, {"start": 282.16, "end": 288.36, "text": " if we could encode the cell structure in in a sort of a continuous way so you"}, {"start": 288.36, "end": 293.04, "text": " know we can encode text in a continuous way let we could also encode a cell"}, {"start": 293.04, "end": 298.20000000000005, "text": " structure because the cell structure I can write it down as an equation I can"}, {"start": 298.20000000000005, "end": 305.04, "text": " say like okay it's the forget gate of the carry times the sigmoid output of X"}, {"start": 305.04, "end": 314.40000000000003, "text": " plus the so this is the plus here and plus the sigmoid output of X multiplied"}, {"start": 314.40000000000003, "end": 320.48, "text": " by the input let's call that I something like this right this is text I can like"}, {"start": 320.48, "end": 326.6, "text": " write it down and then I can encode that into a vector much I can for example"}, {"start": 326.6, "end": 333.96000000000004, "text": " build another RNN ironically or something to to encode that or I can"}, {"start": 333.96000000000004, "end": 337.8, "text": " represent it as a computation graph like it is here and use a graph neural"}, {"start": 337.8, "end": 342.02000000000004, "text": " network to encode that into a single vector and then I have sort of an"}, {"start": 342.02000000000004, "end": 347.56, "text": " embedding space where each cell that I could build is a point in that embedding"}, {"start": 347.56, "end": 353.04, "text": " space and then I can evaluate a couple of them I can for example say okay this"}, {"start": 353.04, "end": 357.16, "text": " one here this one here this one here this one here I'm going to train them"}, {"start": 357.16, "end": 361.62, "text": " these cells I'm going to do the full training eval and so on get their"}, {"start": 361.62, "end": 366.98, "text": " scores and then I can learn basically in this latent space I can learn a"}, {"start": 366.98, "end": 372.8, "text": " predictor I can say okay here I get I got an eight I got a nine I got a two"}, {"start": 372.8, "end": 379.6, "text": " and I got a four so it appears to be that in this direction that the good"}, {"start": 379.6, "end": 384.2, "text": " cells are in this direction and then I can do it again I can sample or I can do"}, {"start": 384.2, "end": 389.88, "text": " gradient descent in this space since this is now a continuous space and the"}, {"start": 389.88, "end": 394.64, "text": " gradient descent on the model that gives me this space it's right so this this"}, {"start": 394.64, "end": 400.8, "text": " method basically tries to take in the building plan of a cell and learn to"}, {"start": 400.8, "end": 406.92, "text": " predict the performance just by looking at it if if you're thinking of the"}, {"start": 406.92, "end": 412.24, "text": " touring machine right now then you like I I immediately thought of like this"}, {"start": 412.24, "end": 416.28000000000003, "text": " this halting problem because it appears to be exactly what it is so you're"}, {"start": 416.28000000000003, "end": 420.24, "text": " trying to build a machine that takes the building plan of another machine and"}, {"start": 420.24, "end": 427.84000000000003, "text": " tries to predict its performance now in a general sense we can already state"}, {"start": 427.84, "end": 433.23999999999995, "text": " that this problem is sort of the difficulty of this problem is"}, {"start": 433.23999999999995, "end": 439.64, "text": " equivalent to the difficulty of the original problem so I'm not sure but it"}, {"start": 439.64, "end": 443.23999999999995, "text": " appears to you know it appears to work if you throw lots of compute at it but"}, {"start": 443.23999999999995, "end": 448.03999999999996, "text": " of course that's a problem you need lots of compute right so either your your"}, {"start": 448.03999999999996, "end": 454.52, "text": " option one is to run all of these things and kind of iterate them in a neural"}, {"start": 454.52, "end": 461.12, "text": " sorry in an evolutionary way or your second option is to take the building"}, {"start": 461.12, "end": 466.44, "text": " plan and predict the performance from that both are not satisfactory and both"}, {"start": 466.44, "end": 472.03999999999996, "text": " use lots of compute now neural petri dish is a or synthetic petri dish is a"}, {"start": 472.03999999999996, "end": 479.0, "text": " way to combine the two together it says can we take the building plan here but"}, {"start": 479.0, "end": 486.28, "text": " actually run on the data on data to predict the performance so what they're"}, {"start": 486.28, "end": 492.4, "text": " saying is basically if I have this cell right here and usually this cell you know"}, {"start": 492.4, "end": 499.12, "text": " it deals with vectors of let's say size 512 and so on it will say it since this"}, {"start": 499.12, "end": 504.84, "text": " is let me draw it again up here so here you have the cell and here you have"}, {"start": 504.84, "end": 509.08, "text": " somehow the connections in there when you carry and the input and the input"}, {"start": 509.08, "end": 518.56, "text": " here okay and this is the output or the carry I have 512 embedding like size of"}, {"start": 518.56, "end": 524.16, "text": " this so this is a giant cell there's 512 the vector has 512 dimensions going in"}, {"start": 524.16, "end": 530.4, "text": " basically can't I take the exact same thing but and keep the connection"}, {"start": 530.4, "end": 534.52, "text": " pattern so I would keep the entire pattern of connection right here but I"}, {"start": 534.52, "end": 544.28, "text": " only do it for one or two so this is 512 and this is just two right just the I"}, {"start": 544.28, "end": 549.52, "text": " just reduce the dimensionality but I sort of keep the connection pattern"}, {"start": 549.52, "end": 556.36, "text": " alive if I only do that I have like a very small network right now and the"}, {"start": 556.36, "end": 561.64, "text": " same goes for if this is so a lot of times these RN ends they have multiple"}, {"start": 561.64, "end": 566.3199999999999, "text": " layers of these things so they have another exactly equal box up here and"}, {"start": 566.3199999999999, "end": 572.6, "text": " then another up here I can just reduce this to one layer and out of the"}, {"start": 572.6, "end": 578.88, "text": " regularity of these neural network things it is known that or one can make"}, {"start": 578.88, "end": 583.48, "text": " the assumption that the performance on this thing will sort of kind of be"}, {"start": 583.48, "end": 588.72, "text": " correlated to the performance of the entire thing and that's one of the"}, {"start": 588.72, "end": 596.5600000000001, "text": " things that this Petri dish paper does so we try to take out what we are"}, {"start": 596.5600000000001, "end": 600.96, "text": " trying to search over namely the connection pattern we keep that as it"}, {"start": 600.96, "end": 607.36, "text": " is up here but we reduce everything else we reduce the dimensionality we reduce"}, {"start": 607.36, "end": 610.6, "text": " the number of layers and so on now they don't actually reduce the number of"}, {"start": 610.6, "end": 615.6, "text": " layers here but you can reduce the number of units and so on okay so this"}, {"start": 615.6, "end": 621.28, "text": " in essence this works whenever you can do that whenever you can keep the"}, {"start": 621.28, "end": 627.84, "text": " structure you're searching over but can reduce the rest so that's one"}, {"start": 627.84, "end": 632.84, "text": " precondition doesn't work for everything the second part here is that"}, {"start": 632.84, "end": 637.0, "text": " you don't want to use this particular training data and then this particular"}, {"start": 637.0, "end": 641.72, "text": " validation data because first of all it's a lot of training data and second"}, {"start": 641.72, "end": 647.84, "text": " of all it won't give you that good of a prediction instead of what you're trying"}, {"start": 647.84, "end": 652.72, "text": " to do and this is the second part of the idea of Petri dish is you're trying to"}, {"start": 652.72, "end": 661.24, "text": " abstract the training data to get you a very small data set such that and the"}, {"start": 661.24, "end": 667.12, "text": " validation data as well as well such that if you train on this data and"}, {"start": 667.12, "end": 673.48, "text": " evaluate on this data the performance that you get will be very predictive of"}, {"start": 673.48, "end": 679.88, "text": " the performance had you trained the big model on the big data set okay in fact"}, {"start": 679.88, "end": 685.64, "text": " in this Petri dish paper these little data sets they have nothing to do with"}, {"start": 685.64, "end": 690.52, "text": " the with the original train and validation data and that's I think that's"}, {"start": 690.52, "end": 695.8, "text": " one of the cool things here these things this training data and this validation"}, {"start": 695.8, "end": 702.0, "text": " data they are optimized as well by the procedure they're optimized special data"}, {"start": 702.0, "end": 707.76, "text": " points that are trained these are trained parameters such that if you"}, {"start": 707.76, "end": 711.3199999999999, "text": " train on the training on the small training data and evaluate on the small"}, {"start": 711.3199999999999, "end": 717.12, "text": " eval data the you will be able to predict this performance back here with"}, {"start": 717.12, "end": 722.9599999999999, "text": " high accuracy okay and this I think is where previous approaches have or might"}, {"start": 722.96, "end": 727.36, "text": " have failed because it's you know the the idea of scaling down your network in"}, {"start": 727.36, "end": 731.64, "text": " order to do the architecture search is probably has it has appeared to many"}, {"start": 731.64, "end": 737.7, "text": " people before that's not you know that's not really genius idea but probably they"}, {"start": 737.7, "end": 741.26, "text": " have all found that now we can't really do it it doesn't really give us accurate"}, {"start": 741.26, "end": 746.6800000000001, "text": " enough numbers but in this case the addition of adding these synthetic data"}, {"start": 746.68, "end": 753.16, "text": " sets that are much smaller but can still if you train and evaluate on them can"}, {"start": 753.16, "end": 759.8399999999999, "text": " still predict with high accuracy the full score of the full model that I"}, {"start": 759.8399999999999, "end": 765.3599999999999, "text": " think makes this idea work all right so I guess we're already through the idea"}, {"start": 765.3599999999999, "end": 772.1999999999999, "text": " and problem setting and everything without actually reading the paper so"}, {"start": 772.2, "end": 779.08, "text": " they give this example right here at the beginning that if you have a a two layer"}, {"start": 779.08, "end": 784.36, "text": " 100 wide so 100 dimensional MNIST networks it's two layers it's I think"}, {"start": 784.36, "end": 792.88, "text": " it's an MLP two layer MLP with a non linearity that is this the sigmoid right"}, {"start": 792.88, "end": 800.5200000000001, "text": " here this okay now you can see it has this temperature parameter here it has"}, {"start": 800.52, "end": 806.04, "text": " this slope parameter and you you want to do neural architecture search to find"}, {"start": 806.04, "end": 810.4399999999999, "text": " the best slope parameter now usually you would just do a grid search but this is"}, {"start": 810.4399999999999, "end": 814.84, "text": " an this is an example because this can be of course much higher dimensional"}, {"start": 814.84, "end": 821.96, "text": " things and then you don't want to do grid search anymore okay so what do we"}, {"start": 821.96, "end": 828.36, "text": " do if you look at how the 100 white MNIST network so we can draw it right"}, {"start": 828.36, "end": 836.32, "text": " here so this is a 100 dimensional MNIST network so this is 100 and each cell"}, {"start": 836.32, "end": 840.96, "text": " each connection here first has a weight and then has the sigmoid non-linearity"}, {"start": 840.96, "end": 845.84, "text": " and the sigmoid non-linearity is parameterized by the parameter C okay"}, {"start": 845.84, "end": 852.44, "text": " and you have you have many of them right you have one here and so on and each one"}, {"start": 852.44, "end": 857.8000000000001, "text": " has a different C and each of these networks represents one blue dot here"}, {"start": 857.8, "end": 862.76, "text": " so if you let C vary so this sigmoid slope value right here that's your"}, {"start": 862.76, "end": 866.88, "text": " parameter C if you let this vary and train the big network on the entire"}, {"start": 866.88, "end": 872.0799999999999, "text": " data set to convergence and then you eval on the validation data set you get"}, {"start": 872.0799999999999, "end": 876.8399999999999, "text": " the slope like the blue curve so if you see the blue curve the blue curve is if"}, {"start": 876.8399999999999, "end": 881.16, "text": " you start over here if you reduce this slope you'll gain in performance but if"}, {"start": 881.16, "end": 886.12, "text": " you reduce it too much you drop drastically okay until it's if it's if"}, {"start": 886.12, "end": 892.92, "text": " it's zero it's basically you know the the X is not the signal doesn't propagate"}, {"start": 892.92, "end": 898.88, "text": " anymore and you you have no learning occurring okay so that's the original"}, {"start": 898.88, "end": 904.88, "text": " performance now what if I only give you training data in this range right here I"}, {"start": 904.88, "end": 909.68, "text": " only showed you this particular range I can can't actually zoom in that much but"}, {"start": 909.68, "end": 915.34, "text": " if if I give you this and I ask you to build one of these please take the"}, {"start": 915.34, "end": 918.88, "text": " architecture and predict the performance that we saw at the beginning like one of"}, {"start": 918.88, "end": 927.88, "text": " these good little machines or Turing Turing machines you would it's basically"}, {"start": 927.88, "end": 933.32, "text": " say well that looks to me like a line so I'm gonna predict the red thing here and"}, {"start": 933.32, "end": 937.48, "text": " even if you can you know evaluate a bunch of these it just looks like a line"}, {"start": 937.48, "end": 943.1600000000001, "text": " and you're going to predict it's probably a slope like this right this"}, {"start": 943.16, "end": 947.8, "text": " happens almost independently of which model you choose to predict right here"}, {"start": 947.8, "end": 952.88, "text": " the the data of training is simply doesn't give away that the fact that"}, {"start": 952.88, "end": 957.8399999999999, "text": " there is a there is this break down here which happens in the real world so if you"}, {"start": 957.8399999999999, "end": 965.16, "text": " just give this as training data there is no way so so the the criticism about"}, {"start": 965.16, "end": 970.36, "text": " these models is valid that they will only work where you give them training"}, {"start": 970.36, "end": 974.12, "text": " data they can at best interpolate their training data but they can't really"}, {"start": 974.12, "end": 980.04, "text": " extrapolate now here since the synthetic Petri dish method which is the green"}, {"start": 980.04, "end": 986.92, "text": " thing here uses the actual not the actual non-linearity that this thing"}, {"start": 986.92, "end": 992.76, "text": " characterizes so it it instantiates the sigmoid with the parameter C that you"}, {"start": 992.76, "end": 996.64, "text": " give it just not on the large network but on a small network in fact their"}, {"start": 996.64, "end": 1004.04, "text": " network is just one unit sorry one unit and then another unit so it's just a two"}, {"start": 1004.04, "end": 1009.1999999999999, "text": " hidden layer but just with one unit instead of 100 and of course you can't"}, {"start": 1009.1999999999999, "end": 1013.4, "text": " feed in MNIST right here right but we said they don't feed in the data they"}, {"start": 1013.4, "end": 1019.12, "text": " actually feed in their synthetic data that they learn so you give them the"}, {"start": 1019.12, "end": 1025.56, "text": " points here and they learn the synthetic data to to evaluate to evaluate the"}, {"start": 1025.56, "end": 1031.48, "text": " others and then once you ask them well if if my C is right here what's the"}, {"start": 1031.48, "end": 1036.8799999999999, "text": " performance going to be it's going to instantiate that in its small network it"}, {"start": 1036.8799999999999, "end": 1041.48, "text": " is going to use the training data that it has learned from this region right"}, {"start": 1041.48, "end": 1045.56, "text": " here in order to train this and then it's going to evaluate this on the"}, {"start": 1045.56, "end": 1051.0, "text": " synthetic validation data that is also learned on the training data and it is"}, {"start": 1051.0, "end": 1056.16, "text": " going to come up with a performance metric it says okay this is how good"}, {"start": 1056.16, "end": 1061.4, "text": " it's going to be and since it is an approximation in its building plan to"}, {"start": 1061.4, "end": 1069.6, "text": " the entire network it will react similarly so it will get that there is"}, {"start": 1069.6, "end": 1074.08, "text": " this performance dip right here okay so it you can see how this sort of makes"}, {"start": 1074.08, "end": 1078.16, "text": " sense you are actually running an approximation to the actual program"}, {"start": 1078.16, "end": 1082.52, "text": " instead of just looking at the plan of the program and trying to predict it"}, {"start": 1082.52, "end": 1090.92, "text": " which you know halting problem says hello okay so that is the motivating"}, {"start": 1090.92, "end": 1100.24, "text": " example of their MNIST thing and here is the entire algorithm all right so you"}, {"start": 1100.24, "end": 1106.8000000000002, "text": " take MNIST training and validation data and you instantiate a bunch of really"}, {"start": 1106.8, "end": 1111.12, "text": " big networks this is ground truth okay you you need this you need this to learn"}, {"start": 1111.12, "end": 1116.44, "text": " from you instantiate a bunch of really big networks now if I draw the graph"}, {"start": 1116.44, "end": 1125.44, "text": " from before right we had this was the performance of the actual networks you"}, {"start": 1125.44, "end": 1130.6399999999999, "text": " want you this comes from here from this region right here this is the training"}, {"start": 1130.6399999999999, "end": 1135.8799999999999, "text": " data okay so you instantiate a bunch of these networks each one you instantiate"}, {"start": 1135.88, "end": 1140.8400000000001, "text": " in one of them right each one gives rise to a different non-linearity and you do"}, {"start": 1140.8400000000001, "end": 1145.0400000000002, "text": " the full training ground truth training and evaluation on the full training set"}, {"start": 1145.0400000000002, "end": 1149.6000000000001, "text": " and the full validation set and you get validation losses right for each of"}, {"start": 1149.6000000000001, "end": 1154.64, "text": " these and these are the points right here now you that's the training data"}, {"start": 1154.64, "end": 1159.3600000000001, "text": " for your neural for your neural architecture search so for your Petri"}, {"start": 1159.36, "end": 1166.36, "text": " dish method what the Petri dish does is it says it extracts the motive and the"}, {"start": 1166.36, "end": 1171.24, "text": " motive is the thing that you optimize over so as I said you want to keep that"}, {"start": 1171.24, "end": 1177.6, "text": " thing in its essence but you want to reduce everything else so it reduces it"}, {"start": 1177.6, "end": 1183.4799999999998, "text": " instead of from a two layer on hundred YML MLP it reduces that to a two layer"}, {"start": 1183.48, "end": 1193.92, "text": " single neuron wide MLP okay and it now this over here is the training data for"}, {"start": 1193.92, "end": 1197.72, "text": " the procedure that we're going to do now so what it would take is it would take"}, {"start": 1197.72, "end": 1202.68, "text": " it would take one of these values it would instantiate and we have that here"}, {"start": 1202.68, "end": 1208.84, "text": " would instantiate the neural network in the small form of that and now we know"}, {"start": 1208.84, "end": 1214.84, "text": " that if I train the full data and evaluate if I train on the full training"}, {"start": 1214.84, "end": 1221.36, "text": " data and evaluate on the full validation data I should get this accuracy all"}, {"start": 1221.36, "end": 1228.52, "text": " right so I will create and we're going to look at in a second I will create"}, {"start": 1228.52, "end": 1233.9199999999998, "text": " training and validation data such that if I train on this training data and"}, {"start": 1233.92, "end": 1241.5600000000002, "text": " then validate on this validation data I get the same validation loss as if I had"}, {"start": 1241.5600000000002, "end": 1246.78, "text": " trained the big network with the same you know the same C parameter on the"}, {"start": 1246.78, "end": 1250.6000000000001, "text": " full training data and evaluate on the full validation data okay so in this"}, {"start": 1250.6000000000001, "end": 1256.04, "text": " step I'm optimizing the data here the training and validation data all right"}, {"start": 1256.04, "end": 1261.68, "text": " and now in the second step once I have this training and validation data such"}, {"start": 1261.68, "end": 1269.3600000000001, "text": " like that I can basically reproduce this this graph right here then I can go and"}, {"start": 1269.3600000000001, "end": 1275.76, "text": " actually ask my model okay now please tell me what happens over here so what"}, {"start": 1275.76, "end": 1281.04, "text": " am I going to do I'm gonna take that I'm gonna instantiate it I'm going to use my"}, {"start": 1281.04, "end": 1285.5600000000002, "text": " training data that I learned to train it I'm going to use my validation data that"}, {"start": 1285.5600000000002, "end": 1289.16, "text": " I learned to evaluate it and it's going to give me a number and that number is"}, {"start": 1289.16, "end": 1295.76, "text": " going to be like close to hopefully close to do this so this is how we can"}, {"start": 1295.76, "end": 1301.88, "text": " extrapolate using that method okay now there are a number of assumptions right"}, {"start": 1301.88, "end": 1306.92, "text": " here and you can imagine this doesn't work in any situation this works if if"}, {"start": 1306.92, "end": 1313.3200000000002, "text": " you you know if you basically you have to get lucky in that you have to"}, {"start": 1313.3200000000002, "end": 1318.88, "text": " abstract the correct things right I said you need to reduce everything else so"}, {"start": 1318.88, "end": 1325.2800000000002, "text": " they reduce notably you see they reduce the 100 the 100 layer width to a single"}, {"start": 1325.2800000000002, "end": 1330.4, "text": " neuron wide MLP and they sort of guess that that doesn't change the fundamental"}, {"start": 1330.4, "end": 1336.72, "text": " thing but you can also see they leave the two layer right they leave the two"}, {"start": 1336.72, "end": 1342.48, "text": " the two layer neural network and I'm can almost guarantee you that they tried"}, {"start": 1342.48, "end": 1348.3200000000002, "text": " this reducing this to a one layer neural network and it did not work and so you"}, {"start": 1348.32, "end": 1353.4399999999998, "text": " have to be sort of very careful of what quantities you abstract and what"}, {"start": 1353.4399999999998, "end": 1358.08, "text": " quantities you don't because okay now you might always think oh I can reduce"}, {"start": 1358.08, "end": 1363.1599999999999, "text": " the you know number of dimensions or channels that's also not always the case"}, {"start": 1363.1599999999999, "end": 1367.04, "text": " so I think that's kind of the crux of the method you have to actually"}, {"start": 1367.04, "end": 1372.6399999999999, "text": " engineer this down compressing of the architecture such that it's its"}, {"start": 1372.64, "end": 1381.0, "text": " properties are still kept and yeah but yeah in other things how do you how do"}, {"start": 1381.0, "end": 1385.76, "text": " you actually produce training and validation data to match these and there"}, {"start": 1385.76, "end": 1390.8000000000002, "text": " are a number of ways but what comes to mind is is meta learning right so"}, {"start": 1390.8000000000002, "end": 1395.3600000000001, "text": " because what you're doing they initialize the training and validation"}, {"start": 1395.3600000000001, "end": 1400.24, "text": " data at random points so these are just random at the beginning and then they"}, {"start": 1400.24, "end": 1408.68, "text": " optimize the data itself using gradient descent okay now I see synthetic training"}, {"start": 1408.68, "end": 1415.32, "text": " data and they are randomly initialized okay and they use gradient descent they"}, {"start": 1415.32, "end": 1420.68, "text": " have it somewhere yes so they have this inner training loop okay which is many"}, {"start": 1420.68, "end": 1427.84, "text": " steps of inner training and then they have the outer loss which is the it's"}, {"start": 1427.84, "end": 1432.0, "text": " the validation loss after the inner training loop and the difference for"}, {"start": 1432.0, "end": 1436.04, "text": " that to the true validation loss and then they do gradient descent on this"}, {"start": 1436.04, "end": 1440.8, "text": " outer loss now this outer loss is a result of the inner loss and the inner"}, {"start": 1440.8, "end": 1444.9599999999998, "text": " loss is a result of the inner training procedure and the inner training"}, {"start": 1444.9599999999998, "end": 1450.3999999999999, "text": " procedure is n steps of feeding in the training data every step you feed in the"}, {"start": 1450.3999999999999, "end": 1454.52, "text": " training data so your computational graph is going to look like so here's"}, {"start": 1454.52, "end": 1460.48, "text": " your training data as train and here are your initial parameters you at random"}, {"start": 1460.48, "end": 1464.12, "text": " lies initialize them randomly in the first step use the training data to"}, {"start": 1464.12, "end": 1470.84, "text": " produce theta one then in the second step you use your training parameter"}, {"start": 1470.84, "end": 1475.48, "text": " your training data again to produce theta two and then you use it again to"}, {"start": 1475.48, "end": 1479.56, "text": " use theta three and so on each time you feed the training data in order to"}, {"start": 1479.56, "end": 1484.2, "text": " evolve your parameters to give you a better prediction right so the gradient"}, {"start": 1484.2, "end": 1490.1200000000001, "text": " since somewhere back here there's a loss the gradient here will have to flow back"}, {"start": 1490.1200000000001, "end": 1494.52, "text": " through all of these paths and through all of these connections to the training"}, {"start": 1494.52, "end": 1499.0, "text": " data this is kind of you back propagate through an optimization procedure and we"}, {"start": 1499.0, "end": 1503.32, "text": " have this a bunch of times here and I've looked at the code and the code is like"}, {"start": 1503.32, "end": 1510.2, "text": " really crazy and it looks like proper research code but it appears to be that"}, {"start": 1510.2, "end": 1513.52, "text": " that's actually what's happening they back prop through the optimization"}, {"start": 1513.52, "end": 1521.44, "text": " procedure to find this synthetic training and validation data now that's"}, {"start": 1521.44, "end": 1525.48, "text": " I mean that's crazy but it also kind of limits how far you can go with this"}, {"start": 1525.48, "end": 1530.84, "text": " because usually you can't back prop for more than a couple of steps doing this"}, {"start": 1530.84, "end": 1534.72, "text": " now that the model the fact that the model inner model is small helps but"}, {"start": 1534.72, "end": 1539.36, "text": " also this introduces very very much like these things are very brittle if you"}, {"start": 1539.36, "end": 1543.84, "text": " back prop through an optimization procedure like this these things tend to"}, {"start": 1543.84, "end": 1550.1599999999999, "text": " be very brittle and so I think there's another thing there where you have to"}, {"start": 1550.1599999999999, "end": 1558.1999999999998, "text": " pay careful attention all right that's it's basically it the last thing they"}, {"start": 1558.1999999999998, "end": 1563.8799999999999, "text": " say is that they can combine this with architecture search in that so not only"}, {"start": 1563.88, "end": 1570.0400000000002, "text": " can you predict good architectures what you can do is you can actually predict"}, {"start": 1570.0400000000002, "end": 1574.96, "text": " the which architectures are good and then you can use that prediction to get"}, {"start": 1574.96, "end": 1580.7600000000002, "text": " new to basically input this into your neural architecture search to inform it"}, {"start": 1580.7600000000002, "end": 1585.1200000000001, "text": " so instead of the neural architecture search having to evaluate all of the"}, {"start": 1585.1200000000001, "end": 1590.3200000000002, "text": " candidates that it produces it only has to now evaluate the very small subset of"}, {"start": 1590.32, "end": 1595.2, "text": " candidates that the synthetic petri dish training deems most worthy of being"}, {"start": 1595.2, "end": 1601.84, "text": " evaluated in this case here instead of evaluating all of the things here it"}, {"start": 1601.84, "end": 1606.4399999999998, "text": " would limit itself to whatever the synthetic petri dish says are the"}, {"start": 1606.4399999999998, "end": 1611.8, "text": " highest performing ones because if the synthetic petri dishes and is any good"}, {"start": 1611.8, "end": 1615.6799999999998, "text": " then it will you know give accurate predictions of how they're performing"}, {"start": 1615.68, "end": 1621.2, "text": " and then that can go in multiple rounds so the architecture search can find new"}, {"start": 1621.2, "end": 1626.0800000000002, "text": " come up with new things that it thinks are better through like an evolutionary"}, {"start": 1626.0800000000002, "end": 1631.3200000000002, "text": " mutation algorithm the petri dish can evaluate them in the synthetic way and"}, {"start": 1631.3200000000002, "end": 1638.96, "text": " then suggest the like ten candidates to evaluate on the full test set and you"}, {"start": 1638.96, "end": 1643.4, "text": " see that way you don't have to evaluate all the like thousand candidates all"}, {"start": 1643.4, "end": 1650.48, "text": " right all right cool they do this for this MNIST and they also do it for"}, {"start": 1650.48, "end": 1660.96, "text": " finding a RNN cell for the pen treebank this is a language modeling task and the"}, {"start": 1660.96, "end": 1664.44, "text": " this is a benchmark for neural architecture search where you're trying"}, {"start": 1664.44, "end": 1671.0400000000002, "text": " to find a good RNN cell to get the perplexity really low and here you can"}, {"start": 1671.04, "end": 1676.8799999999999, "text": " see if they give the same amount of data to all the methods then the benchmark"}, {"start": 1676.8799999999999, "end": 1684.52, "text": " neural architecture search is worse than the synthetic petri dish informed"}, {"start": 1684.52, "end": 1689.56, "text": " architecture search now one has to say on the full date I believe the NAO gets"}, {"start": 1689.56, "end": 1696.2, "text": " to about here but of course if you give all of them the same data the neural the"}, {"start": 1696.2, "end": 1702.76, "text": " petri dish beats this method and I think still this method here uses way more"}, {"start": 1702.76, "end": 1708.3600000000001, "text": " compute because it always has to evaluate all the candidates and that's"}, {"start": 1708.3600000000001, "end": 1713.32, "text": " exactly one of these where I learn an architecture to predict the other"}, {"start": 1713.32, "end": 1718.52, "text": " architecture by just looking at it so it works but it doesn't work as well as"}, {"start": 1718.52, "end": 1723.16, "text": " actually running the architecture in an abstract fashion this also shows you the"}, {"start": 1723.16, "end": 1728.6000000000001, "text": " importance of selecting your experimental evaluation in a smart way"}, {"start": 1728.6000000000001, "end": 1732.64, "text": " like they argue they argue for very long why it makes sense to evaluate"}, {"start": 1732.64, "end": 1739.1200000000001, "text": " everything on reduced data such that their method here can be better and they"}, {"start": 1739.1200000000001, "end": 1744.6000000000001, "text": " don't have to compare to the full thing it's easier for them to work on reduced"}, {"start": 1744.6000000000001, "end": 1749.48, "text": " data and they argue you know it's it's it's what people usually do in practice"}, {"start": 1749.48, "end": 1756.76, "text": " and that's the the task they focus on so you know good good good good paper"}, {"start": 1756.76, "end": 1763.84, "text": " writing right here yeah that's basically it to the paper there's a lot of things"}, {"start": 1763.84, "end": 1772.64, "text": " to be said here I think this works in very very limited settings it seems to"}, {"start": 1772.64, "end": 1778.32, "text": " me that it's sort of brittle with respect to how you abstract and also it"}, {"start": 1778.32, "end": 1783.3999999999999, "text": " it's always the case like how many how large is this synthetic training data in"}, {"start": 1783.3999999999999, "end": 1789.08, "text": " their case there they like abstract this to 20 or 30 data points or something like"}, {"start": 1789.08, "end": 1793.1599999999999, "text": " this so it seems to me since you're optimizing this training data with"}, {"start": 1793.1599999999999, "end": 1799.76, "text": " gradient descent what you would mainly find are adversarial sort of adversarial"}, {"start": 1799.76, "end": 1804.76, "text": " examples to this architecture here so I'm going to guess that the inner"}, {"start": 1804.76, "end": 1810.36, "text": " optimization is very noisy and that's because if you really let your"}, {"start": 1810.36, "end": 1816.56, "text": " optimizer run then it will abuse every single thing it can to match that"}, {"start": 1816.56, "end": 1820.24, "text": " validation loss and that will usually lead to an adversarial example since"}, {"start": 1820.24, "end": 1827.64, "text": " you're optimizing the data itself okay so I think this suffers from that and"}, {"start": 1827.64, "end": 1832.04, "text": " this is we had this in the in the planning you know the planning in in"}, {"start": 1832.04, "end": 1836.48, "text": " learned world models and reinforcement learning where if you have a really"}, {"start": 1836.48, "end": 1840.58, "text": " really good planner it will just abuse the mistakes that you make in"}, {"start": 1840.58, "end": 1844.32, "text": " approximating the true world and the same here you're going to make mistakes"}, {"start": 1844.32, "end": 1850.0, "text": " approximating this architecture here and the better your your optimizer is for"}, {"start": 1850.0, "end": 1856.44, "text": " producing this synthetic data the probably the worse the worse the result"}, {"start": 1856.44, "end": 1861.76, "text": " is going to match the worst that these losses are going to actually match now"}, {"start": 1861.76, "end": 1865.36, "text": " okay these losses will match because they're that's what you train for but"}, {"start": 1865.36, "end": 1869.72, "text": " the worse these two curves will match each other because now you're just"}, {"start": 1869.72, "end": 1875.72, "text": " finding adversarial examples for your particular training data another concern"}, {"start": 1875.72, "end": 1879.8799999999999, "text": " I have here is with respect to the double descent phenomenon so if you know"}, {"start": 1879.8799999999999, "end": 1884.72, "text": " the double descent phenomenon if here you have your number of parameters and"}, {"start": 1884.72, "end": 1892.48, "text": " here you have your validation loss let's say and you know that if I add parameters"}, {"start": 1892.48, "end": 1897.24, "text": " I can make my validation loss go down so this is assuming I have a model with p"}, {"start": 1897.24, "end": 1902.44, "text": " parameters and I always train it on the train data to like to convergence if I"}, {"start": 1902.44, "end": 1907.16, "text": " add parameters I can generalize better until a point where I add too many"}, {"start": 1907.16, "end": 1911.68, "text": " parameters and I start overfitting and my validation loss goes up again but the"}, {"start": 1911.68, "end": 1918.3200000000002, "text": " double descent phenomenon and I think I've done a video on this shows that"}, {"start": 1918.3200000000002, "end": 1923.44, "text": " after a certain threshold you get the interpolation threshold the validation"}, {"start": 1923.44, "end": 1929.24, "text": " loss goes actually down again it goes down even further here now I'm so this"}, {"start": 1929.24, "end": 1933.68, "text": " is a very strange phenomenon by itself but I'm sort of concerned that if you do"}, {"start": 1933.68, "end": 1938.3200000000002, "text": " this abstraction that this paper proposes so you read your let's say your"}, {"start": 1938.32, "end": 1942.4399999999998, "text": " full model is here with a large number of parameters so it is past this"}, {"start": 1942.4399999999998, "end": 1947.6399999999999, "text": " interpolation threshold if you now seriously reduce the number of"}, {"start": 1947.6399999999999, "end": 1953.9199999999998, "text": " parameters because you want to go into this petri dish you will get maybe you"}, {"start": 1953.9199999999998, "end": 1957.54, "text": " will cross this interpolation threshold and actually be on this side of the"}, {"start": 1957.54, "end": 1962.12, "text": " curve right here now of course at the same time you reduce the amount of data"}, {"start": 1962.12, "end": 1969.3999999999999, "text": " which would push you over here again but it is different data so I'm not sure how"}, {"start": 1969.3999999999999, "end": 1973.08, "text": " all of this is going to play out it appears to work in these settings right"}, {"start": 1973.08, "end": 1982.04, "text": " here but I I think this is it's it's sort of it's sort of applicable in some"}, {"start": 1982.04, "end": 1987.4799999999998, "text": " situations and it's it'd be very cool if we develop this further such that we"}, {"start": 1987.4799999999998, "end": 1992.08, "text": " understand when it applies and when we can use it because I feel this can be a"}, {"start": 1992.08, "end": 1997.76, "text": " very cool thing if we understand it better and if we can apply it throughout"}, {"start": 1997.76, "end": 2004.12, "text": " all right that's the end if you like this paper leave a comment if you didn't"}, {"start": 2004.12, "end": 2025.52, "text": " like it leave a comment and bye bye see you next time"}]