We report a learning rule for neural networks that computes how much each neuron should contribute to minimize a giving cost function via the estimation of its target value. By theoretical analysis, we show that this learning rule contains backpropagation, Hebian learning, and additional terms. We also give a general technique for weights initialization. Our results are at least as good as those obtained with backpropagation. The neural networks are trained and tested in three problems: MNIST, MNIST-Fashion, and CIFAR-10 datasets. The associated code is available at this https URL
One may wonder if merging a set of rules could accelerate the learning of a neural network. A specific combination of two rules: backpropagation and Hebbian learning was done in @cite_23 . While it was found that this hybrid rule is no better than backpropagation, the results do not exclude the possibility for better performance on other set of problems or that other combinations could improve performance.
@cite_23
1894248549
A novel neural network architecture is proposed that provides a unified framework for Hebbian and backpropagation-based learning. The learning rule for this architecture, called the hybrid learning rule, combines the features of the Hebbian learning rule, which is a good feature extractor, and the backpropagation algorithm, which is an excellent classifier. By combining these two learning rules into the hybrid learning rule, the hybrid learning rule should have the strengths of both without any of the weaknesses. The hybrid learning rule was applied to the problem of isolated character recognition. While the hybrid learning rule failed to perform better than the backpropagation algorithm, it did generate receptive fields similar to those found by R. Linsker (1986), T.D. Sanger (1989), and D.H. Hubel and T.N. Wiesel (1962). >