Backpropagation is driving today's artificial neural networks. However, despite extensive research, it remains unclear if the brain implements this algorithm. Among neuroscientists, reinforcement learning (RL) algorithms are often seen as a realistic alternative. However, the convergence rate of such learning scales poorly with the number of involved neurons. Here we propose a hybrid learning approach, in which each neuron uses an RL-type strategy to learn how to approximate the gradients that backpropagation would provide. We show that our approach learns to approximate the gradient, and can match the performance of gradient-based learning on fully connected and convolutional networks. Learning feedback weights provides a biologically plausible mechanism of achieving good performance, without the need for precise, pre-specified learning rules. It is unknown how the brain solves the credit assignment problem when learning: how does each neuron know its role in a positive (or negative) outcome, and thus know how to change its activity to perform better next time? Biologically plausible solutions to credit assignment include those based on reinforcement learning (RL) algorithms [4] . In these approaches a globally distributed reward signal provides feedback to all neurons in a network. However these methods have not been demonstrated to operate at scale. For instance, variance in the REINFORCE estimator scales with the number of units in the network. This drives the hypothesis that learning in the brain must rely on additional structures beyond a global reward signal. In artificial neural networks, credit assignment is performed with gradient-based methods computed through backpropagation. This is significantly more efficient than RL-based algorithms. However there are well known problems with implementing backpropagation in biologically realistic neural networks. For instance backpropagation requires a feedback structure with the same weights as the feedforward network to communicate gradients (so-called weight transport). Yet such structures are not observed in neural circuits. Despite this, backpropagation is the only method known to solve learning problems at scale. Thus modifications or approximations to backpropagation that are more plausible have been the focus of significant recent attention [8, 3] . Notably, it turns out that weight transport can be avoided by using fixed, random feedback weights, through a phenomenon called feedback alignment [8] . However feedback alignment does not work in larger, more complicated network architectures (such as convolutional networks). Here we propose to use an RL algorithm to train a feedback system to enable learning. We propose to use a REINFORCE-style perturbation approach to train a feedback signal to approximate what would have been provided by backpropagation. We demonstrate that our model learns as well as regular backpropagation in small models, overcomes the limitations of fixed random feedback weights ("feedback alignment") on more complicated feedforward networks, and can be utilized in convolutional networks. Our method illustrates a biologically realistic way the brain could perform gradient descent-like learning. Here we implement a perturbation-based synthetic gradient method to train neural networks. We show that this hybrid approach can be used in both fully connected and convolutional networks. By removing both the symmetric feedforward, feedback weight requirement imposed by backpropagation this approach is a step towards more biologically-plausible deep learning. In contrast to many perturbation-based methods, this hybrid approach can solve large-scale problems. We thus believe this approach can provide powerful and biologically plausible learning algorithms. While previous research has provided some insight and theory for how feedback alignment works [8, 3, 2] the effect remains somewhat mysterious, and not applicable in some network architectures. Recent studies have shown that some of these weaknesses can be addressed by instead imposing sign congruent feedforward and feedback matrices [10] . Yet what mechanism may produce congruence in biological networks is unknown. Here we show that the shortcomings of feedback alignment can be addressed in another way: the system can learn to adjust weights as needed to provide a useful error signal. Our work is closely related to Akrout et al 2019 [1] , which also uses perturbations to learn feedback weights. However our approach does not divide learning into two phases, and training of the feedback weights does not occur in a layer-wise fashion. Here we tested our method in an idealized setting, however it is consistent with neurobiology in two important ways. First, it involves the separate learning of feedforward and feedback weights. This is possible in cortical networks where complex feedback connections exist between layers, and where pyramidal cells have apical and basal compartments that allow for separate integration of feedback and feedforward signals [5] . Second, noisy perturbations are common in neural learning models. There are many mechanisms by which noise can be measured or approximated [4, 7] , or neurons could use a learning rule that does not require knowing the noise [6] . While our model involves the subtraction of a baseline loss to reduce the variance of the estimator, this does not affect the expected value of the estimator; technically the baseline could be removed or approximated [7] . Thus we believe our approach could be implemented in neural circuits. There is a large space of plausible learning rules that can learn feedback signals in order to more efficiently learn. These promise to inform both models of learning in the brain and learning algorithms in artificial networks. Here we take an early step in this direction. It is worth making the following points on each of the assumptions: • A1. In the paper we assume ξ is Gaussian. Here we prove the more general result of convergence for any subgaussian random variable. • A2. In practice this may be a fairly restrictive assumption, since it precludes using relu non-linearities. Other common choices, such as hyperbolic tangent and sigmoid non-linearities with an analytic cost function do satisfy this assumption, however. • A3. It is hard to establish general conditions under whichẽ n (ẽ n ) T will be full rank. While it may be a reasonable assumption in some cases. Extensions of Theorem 2 to a non-linear network may be possible. However, the method of proof used here is not immediately applicable because the continuous mapping theorem can not be applied in such a straightforward fashion as in Equation (10) . In the non-linear case the resulting sums over all observations are neither independent or identically distributed, which makes applying any law of large numbers complicated.
Perturbations can be used to learn feedback weights on large fully connected and convolutional networks.
{"src": "scitldr-AIC_train", "paper_id": "ByxfNXF8Ir"}