We present novel techniques to accelerate the convergence of Deep Learning algorithms by conducting low overhead removal of redundant neurons -- apoptosis of neurons -- which do not contribute to model learning, during the training phase itself. We provide in-depth theoretical underpinnings of our heuristics (bounding accuracy loss and handling apoptosis of several neuron types), and present the methods to conduct adaptive neuron apoptosis. Specifically, we are able to improve the training time for several datasets by 2-3x, while reducing the number of parameters by up to 30x (4-5x on average) on datasets such as ImageNet classification. For the Higgs Boson dataset, our implementation improves the accuracy (measured by Area Under Curve (AUC)) for classification from 0.88 1 to 0.94 1, while reducing the number of parameters by 3x in comparison to existing literature. The proposed methods achieve a 2.44x speedup in comparison to the default (no apoptosis) algorithm.
In biology, apoptosis -- the death of neurons -- and its inverse neurogenesis, have been studied @cite_37 , and it has been determined that these processes can aid in learning effectively. Chambers modeled the apoptosis neurogenesis process by re-initializing the weights of randomly selected neurons periodically. Their study concluded that periodic apoptosis can improve the performance of a network. This forms the basis of our paper -- by conducting adaptive apoptosis during the training phase.
@cite_37
2047891649 2030723260 2750796620 2964345061
Characterization of neuronal death and neurogenesis in the adult brain of birds, humans, and other mammals raises the possibility that neuronal turnover represents a special form of neuroplasticity associated with stress responses, cognition, and the pathophysiology and treatment of psychiatric disorders. Multilayer neural network models capable of learning alphabetic character representations via incremental synaptic connection strength changes were used to assess additional learning and memory effects incurred by simulation of coordinated apoptotic and neurogenic events in the middle layer. Using a consistent incremental learning capability across all neurons and experimental conditions, increasing the number of middle layer neurons undergoing turnover increased network learning capacity for new information, and increased forgetting of old information. Simulations also showed that specific patterns of neural turnover based on individual neuronal connection characteristics, or the temporal-spatial pattern of neurons chosen for turnover during new learning impacts new learning performance. These simulations predict that apoptotic and neurogenic events could act together to produce specific learning and memory effects beyond those provided by ongoing mechanisms of connection plasticity in neuronal populations. Regulation of rates as well as patterns of neuronal turnover may serve an important function in tuning the informatic properties of plastic networks according to novel informational demands. Analogous regulation in the hippocampus may provide for adaptive cognitive and emotional responses to novel and stressful contexts, or operate suboptimally as a basis for psychiatric disorders. The implications of these elementary simulations for future biological and neural modeling research on apoptosis and neurogenesis are discussed. Traditional views of neurotrophic factor biology held that trophic factors are released from target cells, retrogradely transported along their axons, and rapidly degraded upon arrival in cell bodies. Increasing evidence indicates that several trophic factors such as brain-derived neurotrophic factor (BDNF), fibroblast growth factor (FGF-2), glial cell-line derived neurotrophic factor (GDNF), insulin-like growth factor (IGF-I), and neurotrophin-3 (NT-3), can move anterogradely along axons. They can escape the degradative pathway upon internalization and are recycled for future uses. Internalized ligands can move through intermediary cells by transcytosis, presumably by endocytosis via endosomes to the Golgi system, by trafficking of the factor to dendrites or by sorting into anterograde axonal transport with subsequent release from axon terminals and uptake by second- or third-order target neurons. Such data suggest the existence of multiple “trophic currencies,” which may be used over several steps in neural networks to enable nurturing relationships between connected neurons or glial cells, not unlike currency exchanges between trading partners in the world economy. Functions of multistep transfer of trophic material through neural networks may include regulation of neuronal survival, differentiation of phenotypes and dendritic morphology, synapse plasticity, as well as excitatory neurotransmission. The molecular mechanisms of sorting, trafficking, and release of trophic factors from distinct neuronal compartments are important for an understanding of neurotrophism, but they present challenging tasks owing to the low levels of the endogeneous factors. We show that deep convolutional neural networks combined with nonlinear dimension reduction enable reconstructing biological processes based on raw image data. We demonstrate this by reconstructing the cell cycle of Jurkat cells and disease progression in diabetic retinopathy. In further analysis of Jurkat cells, we detect and separate a subpopulation of dead cells in an unsupervised manner and, in classifying discrete cell cycle stages, we reach a sixfold reduction in error rate compared to a recent approach based on boosting on image features. In contrast to previous methods, deep learning based predictions are fast enough for on-the-fly analysis in an imaging flow cytometer. Dropout has been witnessed with great success in training deep neural networks by independently zeroing out the outputs of neurons at random. It has also received a surge of interest for shallow learning, e.g., logistic regression. However, the independent sampling for dropout could be suboptimal for the sake of convergence. In this paper, we propose to use multinomial sampling for dropout, i.e., sampling features or neurons according to a multinomial distribution with different probabilities for different features neurons. To exhibit the optimal dropout probabilities, we analyze the shallow learning with multinomial dropout and establish the risk bound for stochastic optimization. By minimizing a sampling dependent factor in the risk bound, we obtain a distribution-dependent dropout with sampling probabilities dependent on the second order statistics of the data distribution. To tackle the issue of evolving distribution of neurons in deep learning, we propose an efficient adaptive dropout (named ) that computes the sampling probabilities on-the-fly from a mini-batch of examples. Empirical studies on several benchmark datasets demonstrate that the proposed dropouts achieve not only much faster convergence and but also a smaller testing error than the standard dropout. For example, on the CIFAR-100 data, the evolutional dropout achieves relative improvements over 10 on the prediction performance and over 50 on the convergence speed compared to the standard dropout.