An affine transformation contains \(n\)  weights and bias \(b\)  for an input
vector of dimension \(n\) . According to the analysis in
[1]}, [2]}, weights and biases play
different roles. To determine proper weights of an affine transformation
is equivalent to finding a discriminant 1D projection for partitioning.
For example, one can use the linear discriminant analysis (LDA) to find
a hyper-plane that partitions the whole space into two halves. The
weight vector is the surface normal of the hyper-plane. In other words,
the affine transformation defines a projection onto a line through the
inner product of an input vector and the weight vector. Then bias term
defines the split point in the projected 1D space (i.e., greater, equal
and less than zero). The role of nonlinear activation was investigated
for CNNs in [3]} and multi-layer perceptrons
(MLPs) in [2]}. It is used to avoid the sign
confusion problem caused by two neurons in cascade.
<FIGURE>