A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Preface. Part I: Overview Articles. 1. Learning to Learn: Introduction and Overview S. Thrun, L. Pratt. 2. A Survey of Connectionist Network Reuse Through Transfer L. Pratt, B. Jennings. 3. Transfer in Cognition A. Robins. Part II: Prediction. 4. Theoretical Models of Learning to Learn J. Baxter. 5. Multitask Learning R. Caruana. 6. Making a Low-Dimensional Representation Suitable for Diverse Tasks N. Intrator, S. Edelman. 7. The Canonical Distortion Measure for Vector Quantization and Function Approximation J. Baxter. 8. Lifelong Learning Algorithms S. Thrun. Part III: Relatedness. 9. The Parallel Transfer of Task Knowledge Using Dynamic Learning Rates Based on a Measure of Relatedness D.L. Silver, R.E. Mercer. 10. Clustering Learning Tasks and the Selective Cross-Task Transfer of Knowledge S. Thrun, J. O'Sullivan. Part IV: Control. 11. CHILD: A First Step Towards Continual Learning M.B. Ring. 12. Reinforcement Learning with Self-Modifying Policies J. Schmidhuber, et al. 13. Creating Advice-Taking Reinforcement Learners R. Maclin, J.W. Shavlik. Contributing Authors. Index. <s> BIB001 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Abstract: We describe a new class of learning models called memory networks. Memory networks reason with inference components combined with a long-term memory component; they learn how to use these jointly. The long-term memory can be read and written to, with the goal of using it for prediction. We investigate these models in the context of question answering (QA) where the long-term memory effectively acts as a (dynamic) knowledge base, and the output is a textual response. We evaluate them on a large-scale QA task, and a smaller, but more complex, toy task generated from a simulated world. In the latter, we show the reasoning power of such models by chaining multiple supporting sentences to answer questions that require understanding the intension of verbs. <s> BIB002 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Learning from a few examples remains a key challenge in machine learning. Despite recent advances in important domains such as vision and language, the standard supervised deep learning paradigm does not offer a satisfactory solution for learning new concepts rapidly from little data. In this work, we employ ideas from metric learning based on deep neural features and from recent advances that augment neural networks with external memories. Our framework learns a network that maps a small labelled support set and an unlabelled example to its label, obviating the need for fine-tuning to adapt to new class types. We then define one-shot learning problems on vision (using Omniglot, ImageNet) and language tasks. Our algorithm improves one-shot accuracy on ImageNet from 87.6% to 93.2% and from 88.0% to 93.8% on Omniglot compared to competing approaches. We also demonstrate the usefulness of the same model on language modeling by introducing a one-shot task on the Penn Treebank. <s> BIB003 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Despite recent breakthroughs in the applications of deep neural networks, one setting that presents a persistent challenge is that of "one-shot learning." Traditional gradient-based networks require a lot of data to learn, often through extensive iterative training. When new data is encountered, the models must inefficiently relearn their parameters to adequately incorporate the new information without catastrophic interference. Architectures with augmented memory capacities, such as Neural Turing Machines (NTMs), offer the ability to quickly encode and retrieve new information, and hence can potentially obviate the downsides of conventional models. Here, we demonstrate the ability of a memory-augmented neural network to rapidly assimilate new data, and leverage this data to make accurate predictions after only a few samples. We also introduce a new method for accessing an external memory that focuses on memory content, unlike previous methods that additionally use memory location-based focusing mechanisms. <s> BIB004 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> The move from hand-designed features to learned features in machine learning has been wildly successful. In spite of this, optimization algorithms are still designed by hand. In this paper we show how the design of an optimization algorithm can be cast as a learning problem, allowing the algorithm to learn to exploit structure in the problems of interest in an automatic way. Our learned algorithms, implemented by LSTMs, outperform generic, hand-designed competitors on the tasks for which they are trained, and also generalize well to new tasks with similar structure. We demonstrate this on a number of tasks, including simple convex problems, training neural networks, and styling images with neural art. <s> BIB005 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> We propose an algorithm for meta-learning that is model-agnostic, in the sense that it is compatible with any model trained with gradient descent and applicable to a variety of different learning problems, including classification, regression, and reinforcement learning. The goal of meta-learning is to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples. In our approach, the parameters of the model are explicitly trained such that a small number of gradient steps with a small amount of training data from a new task will produce good generalization performance on that task. In effect, our method trains the model to be easy to fine-tune. We demonstrate that this approach leads to state-of-the-art performance on two few-shot image classification benchmarks, produces good results on few-shot regression, and accelerates fine-tuning for policy gradient reinforcement learning with neural network policies. <s> BIB006 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Though deep neural networks have shown great success in the large data domain, they generally perform poorly on few-shot learning tasks, where a model has to quickly generalize after seeing very few examples from each class. The general belief is that gradient-based optimization in high capacity models requires many iterative steps over many examples to perform well. Here, we propose an LSTM-based meta-learner model to learn the exact optimization algorithm used to train another learner neural network in the few-shot regime. The parametrization of our model allows it to learn appropriate parameter updates specifically for the scenario where a set amount of updates will be made, while also learning a general initialization of the learner network that allows for quick convergence of training. We demonstrate that this meta-learning model is competitive with deep metric-learning techniques for few-shot learning. <s> BIB007 </s> A Survey of Optimization Methods from a Machine Learning Perspective <s> C. Optimization in Meta Learning <s> Meta-learning approaches have been proposed to tackle the few-shot learning problem. Typically, a meta-learner is trained on a variety of tasks in the hopes of being generalizable to new tasks. However, the generalizability on new tasks of a meta-learner could be fragile when it is over-trained on existing tasks during meta-training phase. In other words, the initial model of a meta-learner could be too biased towards existing tasks to adapt to new tasks, especially when only very few examples are available to update the model. To avoid a biased meta-learner and improve its generalizability, we propose a novel paradigm of Task-Agnostic Meta-Learning (TAML) algorithms. Specifically, we present an entropy-based approach that meta-learns an unbiased initial model with the largest uncertainty over the output labels by preventing it from over-performing in classification tasks. Alternatively, a more general inequality-minimization TAML is presented for more ubiquitous scenarios by directly minimizing the inequality of initial losses beyond the classification tasks wherever a suitable loss can be defined. Experiments on benchmarked datasets demonstrate that the proposed approaches outperform compared meta-learning algorithms in both few-shot classification and reinforcement learning tasks. <s> BIB008
Meta learning  ,  is a popular research direction in the field of machine learning. It solves the problem of learning to learn. In the past cognition, the research of machine learning is to obtain a large amount of data in a specific task firstly, and then use the data to train the model. In machine learning, adequate training data is the guarantee of achieving good performance. However, human beings can well process new tasks with only a few training samples, which are much more efficient than traditional machine learning methods. The key point could be that the human brain has learned "how to learn" and can make full use of past knowledge and experience to guide the learning of new tasks. Therefore, how to make machines have the ability to learn efficiently like human beings has become a frontier issue in machine learning. The goal of meta learning is to design a model that can training well in the new tasks using as few samples as possible without overfitting. The process of adapting to the new tasks is essentially a learning process in the meta-testing, but only with limited samples from new tasks. The application of meta learning methods in supervised learning can solve the few-shot learning problems BIB006 . As few-shot learning problems receive more and more attention, meta learning is also developing rapidly. In general, meta learning methods can be summarized into the following three types  : metric-based methods  ,  , BIB003 ,  , model-based methods BIB004 , BIB002 and optimization-based methods BIB005 , BIB007 , BIB006 . In this subsection, we focus on the optimization-based meta learning methods. In meta learning, there are usually some tasks with sufficient training samples and a new task with only a few training samples. The main idea can be described as follows: in the meta-train step, sample a task τ from the total task set T , which contains (D The process of sampling tasks and updating parameters are repeated multiple times. In the meta-test step, the trained metaoptimizer is used for learning a new task. Since the purpose of meta learning is to achieve fast learning, a key point is to make the gradient descent more accurately in the optimization. In some meta learning methods, the optimization process itself can be regarded as a learning problem to learn the prediction gradient rather than a determined gradient descent algorithm BIB001 . Neural networks with original gradient as input and prediction gradient as output is often used as a meta-optimizer BIB005 . The neural work is trained using the training and test samples from other tasks and used in the new task. The parameter update in the process of training is as follows: where θ t is the model parameter at the iteration t, and N is the meta-optimizer with parameter φ that learns how to predict the gradient. After training, the meta-optimizer N and its parameter φ are updated according to the loss value in the test samples. The experiments have confirmed that learning neural optimizers is advantageous compared to the most advanced adaptive stochastic gradient optimization methods used in deep learning BIB005 . Due to the similarity between the gradient update in backpropagation and the cell state update in the long short-term memory (LSTM), LSTM is often used as the meta-optimizer BIB005 , BIB007 . A model-agnostic meta learning algorithm (MAML) is another method for meta learning which was proposed to learn the parameters of any model subjected to gradient descent methods. It is applicable to different learning problems, including classification, regression and reinforcement learning BIB006 . The basic idea of the model-agnostic algorithm is to begin multiple tasks at the same time, and then get the synthetic gradient direction of different tasks, so as to learn a common base model. The main process can be described as follows: in the meta-train step, multiple tasks batch τ i , which contains (D : where α is the learning rate of training process and J τi (θ) is the loss function in task i with training samples D  where β is the meta learning rate of the test process and J τi (θ) is the loss function in task i with test samples D test i . The meta-train step is repeated multiple times to optimize a good initial parameter θ. In the meta-test step, the trained parameter θ is used as the initial parameter such that the model has a maximal performance on the new task. MAML does not introduce additional parameters for meta learning, nor does it require a specific learner architecture. The development of the method is of great significance to the optimization-based meta learning methods. Recently, an expanded task-agnostic meta learning algorithm is proposed to enhance the generalizability of meta-learner towards a variety of tasks, which achieves outstanding performance on few-shot classification and reinforcement learning tasks BIB008 .