Difference target propagation
Backpropagation has been the workhorse of recent successes of deep learning but it relies on infinitesimal effects (partial derivatives) in order to perform credit assignment. This could become a serious issue as one considers deeper and more non-linear functions, e.g., consider the extreme case of non-linearity where the relation between parameters and cost is actually discrete.  Inspired by the biological implausibility of Backpropagation, this thesis proposes a novel approach,  Target Propagation. The main idea is to compute targets rather than gradients, at each layer in which feedforward and feedback networks form Auto-Encoders.  We show that a linear correction for the imperfectness of the Auto-Encoders, called Difference Target Propagation is very effective to make Target Propagation actually work, leading to results comparable to Backpropagation for deep networks with discrete and continuous units, Denoising Auto-Encoders and achieving state of the art for stochastic networks.  In Chapters 1, we introduce several classical learning rules in Deep Neural Networks, including Backpropagation and more biological plausible learning rules. In Chapters 2 and 3, we introduce a novel approach, Target Propagation, more biological plausible learning rule than Backpropagation. In addition, we show that Target Propagation is comparable to Backpropagation in Deep Neural Networks.
Neural networks Machine learning Deep learning Representation learning Optimization Biological plausibility, Learning rule Backpropagation Target propagation Rétropropagation