In addition to constructing the input-output pairs in the
aforementioned manner, we choose to define our activation functions by
the linear unit, \(f=x\) . This activation allows us to effectively
propagate all of the input information through the network. This is in
contrast to the more commonly used rectified linear unit (ReLU),
\(f=\max (0,x)\) , which, depending on the values selected for the
weights, can suppress some information propagation through the network
by eliminating all negative values.[1]} The ReLU
activation is useful when the MLP is used for classification over a
discrete set of positive valued labels. However, we select the linear
activation because our output values are continuous and include values
less than zero.
