Towards deep semi supervised learning
Deep Learning is a quickly growing area of research in the field of Artificial Intelligence that has achieved impressive results in the last decade in various Machine Learning applications. Unsupervised learning for pre-training layers of neural networks was an essential part of the first wave of deep learning methods. A natural next step is to investigate ideas that could combine both unsupervised and supervised learning.  The Ladder Network is a recently proposed semi-supervised architecture that adds an unsupervised component to the supervised learning objective of a deep network. The model can be seen as part of a deep stack of denoising autoencoders or DAEs that learns to reconstruct each layer. At each layer, the reconstruction is done based on a corrupted version of the current layer, using feedback from the upper layer.  This thesis undertakes a systematic analysis and deconstruction of the Ladder Network, investigating which components lead to its excellent performance. We analyze nineteen different variants of the architecture derived by isolating one component of the model at a time.  In Chapters 1 and 2, we introduce fundamentals of artificial neural networks, the gradient-based way of training them and their application in representation learning. We also introduce deep supervised and unsupervised learning and discuss the possible ways of combining them.  In Chapters 3 and 4, we provide a thorough comparison of a large number of variants of the Ladder Network controlling both hyperparameter settings and data set selection. Through our investigation, we discover some general properties of the model that distinguish it from standard feedforward networks.  Finally, we introduce a variant of the Ladder Network that yields to state-of-the-art results for the Permutation-Invariant MNIST classification task in both semi- and fully- supervised settings.
Neural networks Machine learning Deep learning Representation learning Unsupervised learning Supervised learning Semi-supervised learning Model regularization