I'm writing my own training algorithm, but I don't know how to set the bias weight. Have I to set bias in any layer? Must the bias weight, be updated in every layer? These are my other doubtsThanks
How to update bias and bias's weight using backpropagation algorithm
machine learning;neural network;backpropagation
There should be a bias weight for each virtual neuron as it controls the threshold at which the neuron responds to combined input. So if your hidden layer has 100 neurons, that is 100 bias weights for that layer. Same applies to each layer.There are usually two different approaches taken when implementing bias. You can do one or the other:As a separate vector of bias weights for each layer, with different (slightly cut down) logic for calculating gradients.As an additional column in the weights matrix, with a matching column of 1's added to input data (or previous layer outputs), so that the exact same code calculates bias weight gradients and updates as for connection weights.In both cases, you only do backpropagation calculation from neuron activation deltas to the bias weight deltas, you don't need to calculate the activation delta for bias, because it is not something that can change, it is always 1.0. Also the bias does not contribute deltas back further to anything else.