I have the last couple of month worked with an regression problem, of turning a framed audio file into a set of mfcc features, for a  speech recognition applicationI tried a lot different network structures, Cnn, different normalisation techniques, different optimizer, adding more layers and so on.. but finally i've got some decent result, but i don't understand why.. What i did was i added a linear layer as output, and somehow that minimised the error tremendeosly, and bit puzzled why a linear layer would have that much effect?... I mean am still tried to fit the actual output to the desired output?..Why would the activation function matter here?... I mean the weight are being adjusted based on the error, so why is the neural network better at adjusting for the error when the output is linear rather than non-linear (such as: tanh, Relu).. ?
The effect of an linear layer?
neural network;regression;audio recognition
If you are performing regression, you would usually have a final layer as linear.Most likely in your case - although you do not say - your target variable has a range outside of (-1.0, +1.0). Many standard activation functions have restricted output values. For example a sigmoid activation can only output values in range (0.0, 1.0) and a ReLU activation can only output positive values. If the target value is outside of their range, they will never be able to match it and loss values will be high.The lesson to learn here is that it is important which activation function you use in the output layer. The function must be able to output the full range of values of the target variable.